Don't ship on vibes

RAG systems fail quietly. The chatbot still answers fluently — it just retrieved the wrong chunk, or ignored the right one, or added a confident claim that isn't in the source. Because nothing crashes, these failures slip into production unless you measure for them. "It looked fine when I tried it" is not evaluation.

Measure two things separately

A RAG answer can go wrong at two distinct stages, and lumping them together hides the cause:

• Retrieval quality — did we fetch the right chunks? If the answer wasn't in what we retrieved, the LLM never had a chance. Measure with recall (did the right source appear?) and precision (how much of what we fetched was relevant?).
• Answer quality — given the retrieved chunks, is the response faithful (no claims beyond the context), relevant, and actually helpful?

Diagnosing which stage failed tells you what to fix: bad retrieval points to chunking or embeddings; bad answers despite good retrieval point to the prompt or model.

A simple, practical approach

You don't need fancy tooling to start:

1. Write 20–30 real questions with their expected answers and the source that contains each.
2. For each, check: was the correct source retrieved? Is the answer faithful to it?
3. Hand-grade pass/fail and track the score as you change things.

Tools like Ragas can automate faithfulness and relevance scoring, and an LLM can act as a judge — but even a hand-graded 30-question set catches most regressions before users do.

A RAG system without an eval set is one you can't safely improve — every change is a guess. Thirty graded questions turn "I think that helped" into "that raised faithfulness from 80% to 92%."

Try this: Before tweaking anything, write five question/answer/source triples for your use case. Run them, note what fails, and only then change a setting. You've just built the smallest useful evaluation harness — and you'll never tune blind again.