Why Your RAG Pipeline
Is Confidently Wrong
The failure modes that never show up in a demo, why "it worked in testing" doesn't mean what you think, and a concrete reliability playbook from two production RAG systems.
↓ scroll to read · click a section on the left to jump
The demo that got everyone excited
The team builds a RAG chatbot over the internal knowledge base. The demo goes great: leadership asks a handful of questions, the bot answers them cleanly with citations, everyone's impressed, it ships. A week later, a user phrases a covered question slightly differently and gets a confident, wrong answer - one that cites a real document, which doesn't actually say what the bot claims it says. Nobody can explain why. There's no error, no stack trace, no alert. The system did exactly what it was built to do.
This isn't a bug you patch once. It's the default failure mode of retrieval-augmented generation, and it's invisible in a demo by construction - because a demo only ever samples the questions you already knew would work.
A precise vocabulary for how RAG systems fail silently in production, and the concrete engineering habits - observability, evaluation, operational reliability - that catch it before your users do. Grounded in what actually broke, and what actually fixed it, across two production RAG systems I've shipped.
A demo tests the questions you thought to ask
Retrieval quality isn't a single pass/fail number. It's a distribution across every possible question a user could ask, against every possible state your document corpus could be in. A demo samples a handful of points near the top of that distribution - the questions the team already knows the corpus answers well. Production samples the whole thing, including the long tail nobody rehearsed.
- Curated questions vs. real phrasing. Users don't ask questions the way the demo script does - they abbreviate, misspell, combine two questions into one, or ask about something adjacent to what's in the corpus.
- A static corpus vs. one that changes daily. The demo ran against whatever was indexed that morning. Production runs against a corpus that's being edited, added to, and occasionally deleted from, continuously.
- One tester vs. many users with different bars for "correct." What counts as a good-enough answer to the person who built the system is not always what counts as correct to the person relying on it for a real decision.
$ ask("What's our refund policy?")
"Refunds are processed within 5-7 business days..."
# production, three weeks later:
$ ask("can i get money back for the thing i returned last tuesday")
"I don't see a specific returns policy, but generally..."
# ^ same corpus. same answer, technically available.
# retrieval just didn't surface it this time.
Your retrieval system doesn't have one accuracy number. It has a distribution, and the demo only ever samples the top of it.
Four failure modes, and none of them throw an error
These aren't exotic edge cases. They're the default ways a RAG pipeline degrades once it's handling real traffic against a real, changing corpus.
The answer isn't in top-k, but the model answers anyway
Retrieval returns its best k chunks even when none of them actually contain the answer. The LLM, given a prompt that says "answer using this context," often answers anyway - filling the gap from its own parametric memory. Nothing signals that retrieval failed; the answer just reads confidently, from the wrong source.
Close in embedding space, wrong in meaning
Cosine similarity finds the chunk that's topically closest to the query, not necessarily the one that answers it. "Cancellation policy" and "refund policy" sit close together in embedding space and mean genuinely different things to the person asking.
The source changed; the index didn't hear about it
A document gets updated - a price, a policy, a deadline - but the vector index isn't re-embedded on that change. Retrieval keeps confidently returning the old version of the truth, and there's no error because, as far as the index is concerned, nothing is wrong.
It looks grounded. It isn't, quite.
The UI shows a citation next to the answer, which reads as proof the system is grounded. But the generated text is the model's paraphrase of the retrieved chunk, and paraphrasing introduces small, real drift - a citation with a plausible-sounding answer that doesn't fully match what the source actually says.
If you can't see what was retrieved, you can't debug what went wrong
Most RAG systems log the final answer. Almost none log the retrieval trace: the exact query embedding, which chunks came back, their similarity scores, and which of them the model actually leaned on to generate the response. Without that trace, a bad answer discovered six weeks later is close to undebuggable - you genuinely cannot tell whether retrieval failed, generation ignored good context, or the source document itself was wrong.
On the internal contracts RAG system I built, every response streamed back with multi-page citations tied to the exact retrieved chunk - not primarily because it looked good in the UI, but because it's the only way to answer "why did it say that" after the fact, for a system where being wrong has real consequences.
audit_log.write({
query: "can i get money back for the thing i returned",
retrieved_chunk_ids: ["doc_42#p3", "doc_08#p1", "doc_42#p1"],
similarity_scores: [0.71, 0.68, 0.64],
chunk_used_in_answer: "doc_08#p1", // NOT the top-scored chunk
answer: response.text,
model: "gpt-4o", embedding_model_version: "text-embedding-3-large"
})
Shipping a system where the only way to investigate a wrong answer is to ask the LLM to explain itself after the fact - which just generates a second, equally unverifiable answer.
Reliability isn't just retrieval quality. It's staying correct under load.
A pipeline that returns great answers at ten requests a minute can start behaving strangely at five hundred, and the cause is often operational, not semantic. The embedding API throttles under peak load, naive retries duplicate work or write out of order, and a job that partially fails leaves a document half-indexed - which then surfaces weeks later as a "why does retrieval only know about half this file" mystery that looks exactly like a retrieval-quality bug but is actually a reliability bug wearing a disguise.
batches = batch_by_tokens(texts, max_texts=100, max_tokens=15_000)
for batch in batches:
try:
embed(batch)
except RateLimitError as e:
sleep(e.retry_after) // respect the header, don't guess
embed(batch) // idempotent retry - same batch, same checksum
- Token-aware batching, not fixed-size batching. A batch capped at 100 texts can still blow past a token limit if the texts are long - cap on both dimensions.
- Retry-After, not exponential guesswork. The API is telling you exactly how long to wait. Use it instead of a fixed backoff curve that's either too slow or still too fast.
- Idempotent retries. A retry that isn't checksummed against what already succeeded can double-index a document - now retrieval returns two slightly different copies of the same source, silently.
Most RAG systems ship with zero evaluation, then get evaluated by angry users
Building an evaluation set feels like a delay when you're racing to ship. It's cheap insurance compared to the alternative, which is finding out your retrieval quality is bad from a support ticket instead of a test run.
30-50 real questions, before launch
Real questions your users would actually ask, with the correct answer and the expected source chunk written down in advance. Re-run it on every deploy, not just before launch.
Recall@k, not just "the answer sounded right"
Is the chunk that actually contains the answer even in your top-k results? This is measurable and separate from generation quality - and it's usually where the real problem lives.
Does the answer actually follow from the context?
An LLM-as-judge pass that checks whether the generated answer is actually supported by the retrieved chunks, run automatically on a sample of production traffic - not spot-checked by a human once a quarter.
A small, fixed set of canary questions re-run on a schedule against production catches something else entirely: silent regressions, when a document update or a model version bump quietly changes an answer that used to be correct. Treat that kind of drift as a signal worth investigating, not noise to average away.
The checklist I wish someone had handed me
Not just the final answer
Query, retrieved chunk IDs, similarity scores, and which chunk the model actually used - every request, not sampled.
Not on a schedule
A stale embedding is a correctness bug. Trigger re-indexing from the document-update event, not a nightly cron job that's hours behind reality.
Dense + sparse, not just cosine similarity
Pure embedding search misses exact terms, IDs, and names that a simple keyword match would have caught. Combine both; don't treat hybrid as an optimization for later.
Before launch, not after complaints
Thirty real questions with known-correct sources costs an afternoon. Finding out you needed one from a support queue costs a lot more.
Not noise to average away
If the same question returns a meaningfully different answer across runs, that's data about where your system is unstable - investigate it before a user does.
Not just in application logic
Multi-tenant retrieval that relies on an application-layer filter instead of hard isolation at the index level is a data leak waiting for the one request that skips the filter.
RAG isn't broken. It's just young infrastructure.
RAG gets criticized as duct tape - a workaround for the fact that LLMs don't reliably know your data. I think that criticism aims at the wrong target. The pattern itself is sound; retrieval-plus-generation is a reasonable way to ground a model in facts it wasn't trained on. What's actually missing, on most teams I've seen build this, is the operational discipline that every other production dependency gets by default: logging, evaluation, and a plan for what happens when the underlying data changes.
Here's where I land: the teams that treat retrieval quality as an ongoing operational concern - something you monitor and evaluate the same way you'd monitor API latency or error rates - ship RAG that holds up under real use. The teams that treat it as a one-time integration task ship a demo that happened to reach production, and then spend the next six months debugging it one angry user at a time.
None of this is a reason to avoid RAG - it's the right pattern for the problem it solves. It's a reason to budget for the unglamorous half of the work: the trace logging, the golden set, the re-embedding trigger. That half doesn't demo well, which is exactly why it's the half that gets skipped, and exactly why it's the half that actually determines whether the system survives contact with real users.
Where to go next
- RAGAS - RAG Assessment frameworkOpen-source metrics for retrieval and generation quality (faithfulness, answer relevance, context precision/recall) - a practical starting point for the evaluation section above.
- Anthropic - Introducing Contextual RetrievalA concrete technique for reducing the semantic near-miss problem: prepending chunk-specific context before embedding, so retrieval has more than an isolated fragment to match against.
Field notes: Anupam Kumar · Backend & Generative AI Engineer. v1.0