Retrieval is where RAG lives or dies

If the right chunk never makes it into the prompt, no model — however smart — can answer correctly. So when a RAG system underperforms, retrieval is the first place to look, and there are two high-leverage upgrades beyond plain semantic search.

Hybrid search

Pure semantic search is great at meaning but can miss exact terms — a product code, an error number, a specific name — because those carry little semantic signal. Classic keyword search (often BM25) nails exact matches but misses paraphrases. Hybrid search runs both and combines the scores, so you catch "recovering account access" and "error E-4021." For knowledge bases full of names, codes, and jargon, going hybrid is often a bigger win than a better embedding model.

Reranking

Your vector search returns, say, the top 20 candidates fast but roughly. A reranker is a second, more careful model that re-scores those candidates against the question and pushes the truly relevant ones to the top. You retrieve broadly (high recall), then rerank down to a precise top 3–5 (high precision) that go into the prompt. Cohere, Voyage, and Jina offer strong rerankers; adding one is frequently the single most effective retrieval upgrade you can make.

Other levers

• Query rewriting — have an LLM clarify or expand a vague question before searching.
• Metadata filters — narrow by product, date, or language first.
• Multi-query — search several rephrasings and merge the results.

Retrieve wide, then narrow: cast a broad net with hybrid search, then let a reranker pick the few chunks that truly answer the question. Recall first, precision second.

Try this: Take a query that returns mediocre chunks. Add a keyword that must appear (a manual stand-in for hybrid search) and watch retrieval sharpen. It's a quick preview of why combining meaning and exact-match beats either alone.