Hybrid search and reranking

Vector search and keyword search fail in opposite ways:

Hybrid search runs both and merges the results, so a query gets the benefit of each.

#Keyword side: BM25

BM25 is the standard ranking function behind full-text search (inverted indexes). It scores a document higher when it contains the query terms, more so for rare terms and short documents. Elasticsearch, OpenSearch and most vector databases provide it. PostgreSQL's built-in full-text ranking (ts_rank) is a simpler relative; extensions such as ParadeDB's pg_search add true BM25.

#Merging: reciprocal rank fusion

The two systems' scores aren't on the same scale, so merge by rank instead:

RRF score(doc) = Σ over result lists   1 / (k + rank of doc in that list)     (k ≈ 60)

A document ranked well by both lists rises to the top. It's simple, needs no tuning of score weights, and is hard to beat as a default.

#Reranking

Both retrievers are bi-encoders: the query and each document are encoded separately, which is fast but shallow. A reranker, usually a cross-encoder, reads the query and a candidate together and scores how well the candidate answers the query. That's much more accurate, but too slow to run over a whole corpus. So you run it in stages:

hybrid retrieve top 50–100  →  rerank  →  keep top 5–10  →  model

Reranking often makes the largest single quality improvement in a RAG pipeline after chunking. It also lets you send fewer chunks to the model, which saves context and cost. Hosted rerank APIs and open cross-encoder models both work; an LLM can rerank too, at a higher cost.

#Habits

• Measure retrieval separately from generation: is the right chunk in the top 5 before and after the change?
• Filter by metadata first (tenant, permissions, date) so neither retriever wastes slots on documents the user can't see.
• Keep the reranker's top k small. The model answers better from 5 strong passages than from 20 mixed ones.