Hybrid Search and Reranking

Updated Nov 16, 2026 · 5 min read
On this page

    A user pastes "error E4012 at checkout" into the help assistant. Vector search returns pages about checkout problems in general, because the embedding captures the topic but treats "E4012" as noise. The page titled "E4012: card issuer declined" exists, and a plain keyword search would have found it first. For other questions, like "my payment didn't go through", keyword search fails and vector search shines. Production retrieval usually runs both and then reranks the combined list with a slower, more careful model.

    How it works

    Semantic search understands the topic; keyword search matches the exact code. Neither alone is enough.
    Semantic search understands the topic; keyword search matches the exact code. Neither alone is enough.

    Two kinds of first-stage retrieval

    • Keyword (lexical) search, usually BM25: scores documents by how often the query's words appear in them, weighted by how rare each word is across the collection. Excellent for names, codes, IDs, rare terms and exact phrases; blind to synonyms.
    • Vector (semantic) search: compares embeddings (see Embeddings and Vector Search). Excellent for paraphrases and natural questions; weak on exact strings and rare tokens.

    Hybrid search runs both and merges the results.

    Merging: reciprocal rank fusion

    The two systems produce scores on different scales (BM25 scores can be 0–30; cosine similarity is between −1 and 1), so adding raw scores doesn't work well. Reciprocal rank fusion (RRF) uses only each document's rank in each list: score = sum of 1 / (k + rank), with k usually 60.

    Documents near the top of both lists win; a document that is first in only one list still makes the shortlist.
    Documents near the top of both lists win; a document that is first in only one list still makes the shortlist.
    def rrf(rankings, k=60):
        """rankings: lists of doc IDs, best first. Returns IDs sorted by fused score."""
        scores = {}
        for ranking in rankings:
            for rank, doc in enumerate(ranking, start=1):
                scores[doc] = scores.get(doc, 0.0) + 1.0 / (k + rank)
        return sorted(scores, key=scores.get, reverse=True)
    
    vector  = ["checkout", "payments", "cart", "shipping", "error-codes"]
    keyword = ["e4012", "error-codes", "checkout"]
    print(rrf([vector, keyword]))
    # ['checkout', 'error-codes', 'e4012', 'payments', 'cart', 'shipping']

    An alternative is a weighted sum of normalised scores (for example 0.7 × vector + 0.3 × keyword after scaling each to 0–1), which can be tuned per domain but needs care with normalisation.

    Reranking: a second, careful pass

    First-stage retrieval must be fast over millions of documents, so it embeds query and documents separately (a bi-encoder) or counts words. A reranker is a cross-encoder: a model that reads the query and one passage together and outputs a relevance score. It is far more accurate, because it sees how each query word relates to each passage word, but far too slow to run on the whole collection.

    Recall first (cheap, wide), then precision (expensive, narrow).
    Recall first (cheap, wide), then precision (expensive, narrow).

    So the standard pipeline is: retrieve 50–100 candidates with hybrid search, rerank them with a cross-encoder, keep the top 3–10 for the LLM. LLMs themselves can also rerank (ask a model to order the candidates), which is flexible but slower and costlier.

    Trade-offs and costs

    • Hybrid adds a second index to build, keep in sync and query; in exchange recall on exact terms improves a lot.
    • Reranking adds latency: scoring 50 passages with a small cross-encoder on a GPU typically takes tens of milliseconds; on CPU or with large rerankers it can take hundreds.
    • More candidates improve the reranker's chances but cost more time; tune the candidate count with recall measurements.
    • Weights and k in fusion can be tuned, but RRF's defaults are a strong baseline.

    In production

    • Measure each stage: recall@50 of the hybrid stage (is the right passage in the candidate set at all?) and precision@5 after reranking.
    • Keep indexes in sync: a document deleted from the vector index but not the keyword index will still be retrieved.
    • Apply access filters in both first-stage searches.
    • Failure modes: keyword search drowned by very common words; reranker trained on general web data underperforming on domain jargon (consider fine-tuning it); timeouts in the reranker causing empty results (fall back to the fused list).

    When to reach for it

    Almost any RAG over real business content: support articles with error codes, product catalogues with SKUs, legal texts with section numbers, code search with function names. Pure vector search is enough mainly for small, conversational corpora.

    Example

    A support assistant's retrieval recall@5 was 0.72 with vector search alone. Adding BM25 and RRF raised recall@50 from 0.85 to 0.95 (many error-code questions now hit), and a cross-encoder reranker over the top 50 lifted recall@5 to 0.88. Reranking added about 40 ms at p95. Example numbers; the measured gains justified one more index and one small model.

    Common mistakes

    • Adding raw BM25 and cosine scores without normalisation.
    • Reranking only the top 5, which leaves the reranker nothing to fix.
    • Skipping keyword search for content full of codes and names.
    • Not evaluating the reranker on your domain.
    • Forgetting access filters in one of the two searches.

    Say this in the interview

    "I'd run keyword search (BM25) and vector search in parallel, because one catches exact terms and the other catches meaning, and merge them with reciprocal rank fusion, which only needs ranks. Then a cross-encoder reranks the top 50 or so and the best 3–10 go to the LLM. I'd measure recall of the candidate set and precision after reranking separately, and keep both indexes in sync with the same access filters."

    Interview questions on this

    1. Why is a cross-encoder more accurate than a bi-encoder? It reads query and passage together, so it can model their interactions; a bi-encoder compresses each separately.
    2. Why not use a cross-encoder for everything? It must run once per passage per query, which is too slow for millions of documents.
    3. How does RRF work? Sum 1/(k + rank) across result lists; documents high in several lists rise to the top.
    4. When does keyword search beat vector search? Exact identifiers, names, rare terms and quoted phrases.
    5. How many candidates should you rerank? Enough that recall of the candidate set is high (often 50–100), within the latency budget.

    Used in

    • AI Engineering: Vector Search (HNSW, IVF); RAG End to End; Evaluating a RAG System; Your RAG Answers Are Wrong (practical question).
    • AI System Design: Design Perplexity; Design Enterprise RAG; Design Cursor.