Measure Retrieval Quality in RAG

Split the problem

RAG has two failure points. Before tuning prompts, check whether the right chunk was even retrieved.

Metrics

• Recall@k — is the correct chunk in the top-k results?
• MRR — how high up is the first relevant chunk?
• Precision@k — how many of the top-k are actually relevant?

A simple harness

Build a set of questions paired with the IDs of chunks that should answer them. Run retrieval and compute recall@k.

def recall_at_k(retrieved_ids, gold_ids, k=5):
    return len(set(retrieved_ids[:k]) & set(gold_ids)) / len(gold_ids)

If recall is low, fix chunking/embeddings before touching the LLM prompt.