Hybrid search for RAG runs lexical and semantic retrieval over the same query, then fuses their ranked results before generation. Dense embeddings recover meaning and paraphrases, while sparse retrieval preserves exact terms such as product codes, names, error messages, and statutory references.
Why hybrid search for RAG fixes retrieval gaps
Dense retrieval maps queries and documents into an embedding space. It can connect different wording with similar meaning, which helps when users describe a concept differently from the source. Its weakness appears when a rare token carries most of the intent. A model may place two documents near each other semantically while missing the exact identifier the user supplied.
Sparse retrieval scores token overlap. BM25 is a common choice because it rewards matching terms while accounting for document length and term rarity. It can find an exact error code or model number but may miss a passage that answers the question with different vocabulary.
Dense and sparse retrieval are complementary. NVIDIA's hybrid search guidance describes the same division: sparse methods handle keyword matching, while dense representations capture semantic meaning. The combination is especially useful for technical corpora that mix ordinary language with domain-specific identifiers.
Dense and sparse retrieval need separate candidate lists
A typical pipeline sends the normalized query to both retrievers. The dense side returns its top semantic matches with vector scores. The sparse side returns its top lexical matches with BM25 or another text ranking score. Preserve ranks and document identifiers from both paths.
Do not add raw dense and BM25 scores unless the system has calibrated them. Their scales and distributions differ, and those values can change when an index or embedding model changes. Rank-based fusion avoids pretending that unlike scores are directly comparable.
- Run both searches. Retrieve more candidates than the final context will contain.
- Normalize document identity. Map chunks from each list to a stable document and chunk key.
- Fuse the rankings. Apply Reciprocal Rank Fusion or a validated weighted method.
- Apply filters and permissions. Enforce tenant, date, region, and authorization rules before text reaches the model.
- Rerank when justified. Use a cross-encoder or other reranker on a small fused set if evaluation shows a useful gain.
- Assemble context. Deduplicate overlapping chunks and retain source metadata for citations.
Reciprocal Rank Fusion provides a stable default
Reciprocal Rank Fusion, usually shortened to RRF, assigns each document a contribution based on its position in every result list. Documents that rank highly in either list receive credit, and documents that appear in both accumulate more.
The Azure AI Search RRF documentation describes the score as a sum of 1/(rank + k) contributions. The constant smooths the effect of the highest positions. RRF relies on order rather than the original score scale, which makes it practical for merging lexical and vector results.
RRF is not a guarantee that hybrid retrieval will beat a strong single retriever. Candidate depth, chunking, query mix, filters, and the embedding model still control what can be found. It is a sensible baseline because it is simple, inspectable, and does not require score calibration.
Weighted fusion can reflect known query behavior
Some search platforms combine normalized scores or apply explicit dense and sparse weights. A technical support corpus with many part numbers may favor the lexical path. A multilingual knowledge base may place more weight on semantic retrieval. Set those weights from labeled queries, not from a universal ratio.
Query-dependent routing can go further. A request containing a recognized SKU or quoted phrase can increase sparse weight, while a conceptual question can favor dense retrieval. Such rules need tests because an incorrect query classifier can make the system less predictable than a fixed baseline.
Evaluate hybrid retrieval before answer quality
Generation metrics cannot tell you why the source passage was absent. Build a retrieval test set with each query mapped to relevant chunks, then compare dense-only, sparse-only, and hybrid configurations.
- Recall at k: Did the candidate set contain at least one relevant chunk?
- Precision at k: How much of the retrieved set was relevant rather than distracting?
- Mean reciprocal rank: How early did the first relevant result appear?
- Normalized discounted cumulative gain: Did the ranking place highly relevant items above weaker ones?
- Filtered correctness: Did every result obey tenant and permission constraints?
- Latency and cost: What did parallel retrieval, fusion, and reranking add to the request?
Slice results by query type. Track identifiers, acronyms, paraphrases, misspellings, multilingual requests, and long natural-language questions separately. An average score can hide the exact group hybrid search was intended to repair.
Production details decide the outcome
Use identical source documents and stable chunk identifiers across retrievers. If the lexical and vector indexes are updated at different times, fusion may combine inconsistent versions. Monitor indexing lag, empty-result rates, retrieval overlap, filter failures, and the proportion of final answers supported by selected chunks.
Application architecture also matters. An AI API backend should own timeouts, retries, authorization, and trace IDs around both searches. If an agent chooses retrieval strategies dynamically, define that behavior within the AI agent development evaluation suite rather than allowing unmeasured routing changes.
A BM25 vector search comparison should use identical documents, filters, and relevance labels across every retrieval path. The right rollout begins with a lexical baseline, a dense baseline, and RRF hybrid search on the same labeled dataset. Ship the combined path only when it improves the query slices users actually send without violating latency, access-control, or operating-cost limits.