When the answer is wrong, find the stage

A RAG pipeline has several stages, and a bad answer comes from exactly one of them. The skill is locating which before you start changing things. Here are the failures you'll actually hit, and the fix for each.

Retrieval misses the right chunk

The answer exists in your data but never reached the prompt.

Fixes: revisit chunking (too big or too small?), try a better embedding model, add hybrid search, or raise top_k. Confirm by inspecting what was retrieved, not just the final answer.

The right chunk is retrieved but ignored

The context was there; the model answered from memory anyway.

Fixes: instruct it explicitly — "answer only from the context below; if it's not there, say you don't know." Delimit the context clearly and keep it from being buried in the middle.

Ungrounded or hallucinated claims

The answer adds facts that aren't in the sources.

Fixes: demand citations ("quote the sentence you used"), reduce the model's freedom, and measure faithfulness so you catch it systematically.

"I don't know" when the answer exists

Usually too-strict prompting or too-few retrieved chunks.

Fixes: raise top_k, loosen the refusal instruction slightly, and check retrieval recall.

Stale or wrong-source answers

Fixes: re-index when documents change, and use metadata filters so the model can't pull last year's policy.

Almost every "the RAG is broken" complaint is really one stage misbehaving. Inspect the retrieved chunks first — they instantly tell you whether to fix retrieval or fix the prompt.

Try this: Next time a RAG answer is wrong, print the retrieved chunks before the answer. If the right text isn't there, it's a retrieval problem; if it's there but the answer ignored it, it's a prompt problem. That one habit will halve your debugging time.