Hybrid Search Architecture for LLMs

BM25 (Best Matching 25) is a probabilistic ranking function that scores documents based on term frequency, inverse document frequency, and document length normalization. Despite being decades old, BM25 remains surprisingly competitive with neural retrievers for keyword-heavy queries. It excels at exact match scenarios: product SKUs, error codes, proper nouns, and technical jargon that embedding models often tokenize poorly. BM25 operates on inverted indices, enabling sub-millisecond retrieval even over billions of documents. Modern implementations like Elasticsearch's BM25 use skip lists and block-max WAND optimization to prune the search space aggressively. The key parameters — k1 (term frequency saturation) and b (length normalization) — can be tuned per-field for optimal performance on domain-specific corpora.

Dense retrieval encodes queries and documents into fixed-dimensional vectors (typically 768 or 1536 dimensions) using bi-encoder models like E5, BGE, or OpenAI's text-embedding-3-large. Approximate nearest neighbor (ANN) search via HNSW or IVF indices finds semantically similar documents regardless of lexical overlap. The key advantage is understanding paraphrases, synonyms, and intent — 'how to fix a broken pipe' matches documents about 'plumbing repair' even without shared keywords. The major weakness is that embedding models compress all semantic information into a single vector, losing fine-grained token-level matching. Production systems must handle index staleness (re-encoding when documents change), dimension reduction (Matryoshka embeddings for storage efficiency), and metadata filtering (pre-filter vs post-filter tradeoffs that affect recall).

Reciprocal Rank Fusion (RRF) solves the fundamental problem of combining rankings from heterogeneous retrieval systems whose scores are not comparable. BM25 returns log-probability scores while dense retrieval returns cosine similarities — normalizing these onto the same scale is fragile and domain-dependent. RRF sidesteps this entirely by using only rank positions: RRF_score(d) = Σ 1/(k + rank_i(d)) where k is a constant (typically 60) and rank_i(d) is the rank of document d in retriever i. Documents ranked highly by multiple retrievers get boosted scores. RRF requires no training data, no score calibration, and works out-of-the-box across any combination of retrievers. Research shows RRF consistently outperforms trained linear score combinations, likely because rank-based fusion is more robust to score distribution shifts across queries.

Cross-encoders process the query and document together through a single transformer pass, enabling deep token-level interaction that bi-encoders cannot achieve. While bi-encoders encode query and document independently (enabling fast ANN search), cross-encoders attend across both simultaneously, capturing fine-grained relevance signals. This makes them too slow for first-stage retrieval (scoring every document would take hours) but ideal for re-ranking the top 20–100 candidates from the hybrid retrieval stage. Modern cross-encoder models like ms-marco-MiniLM or Cohere Rerank v3 achieve significantly higher NDCG@10 than bi-encoders alone. The latency cost is roughly 1–2ms per document pair on GPU, so re-ranking 50 candidates adds 50–100ms. Production systems often distill large cross-encoders into smaller models for latency-sensitive applications.

Hypothetical Document Embeddings (HyDE) addresses a fundamental asymmetry in retrieval: queries are short and underspecified while documents are long and detailed. HyDE prompts an LLM to generate a hypothetical answer to the query, then embeds this synthetic document for retrieval instead of (or alongside) the original query. The intuition is that a hypothetical answer, even if factually imperfect, is closer in embedding space to relevant documents than the short query. For example, the query 'RAFT consensus' might generate a paragraph about leader election and log replication, which embeds much closer to actual Raft papers. HyDE improves recall by 10–20% on average for ambiguous queries, at the cost of one additional LLM call per query (typically 200–500ms). Multi-HyDE generates multiple hypothetical documents and averages their embeddings for even better coverage of diverse intents behind a single query.