Retrieval: Semantic, Keyword (BM25) and Hybrid Search
How to find the right chunks: dense semantic search, sparse keyword search with BM25, and combining both with reciprocal rank fusion.
What is it?
Retrieval is the R in RAG: given the user's question, return the handful of chunks most likely to contain the answer. If the right chunk is not retrieved, the LLM cannot use it - no prompt can fix a retrieval miss. There are two fundamentally different ways to search, and the best systems use both.
Semantic search (dense retrieval) embeds the query and finds chunks with the most similar embeddings (see embeddings and vector-databases). It is called dense because embeddings are dense vectors: every one of the hundreds of numbers is used. Its strength is meaning: 'How much holiday do I get?' matches 'Employees receive 25 days of annual leave' even though they share no important words. Its weakness is exact tokens: product codes, error codes, names, ticket ids and rare jargon ('ERR_7731', 'Kubernetes 1.29', 'Smith v. Jones') are often blurred, because the embedding captures the gist, not the exact string.
Keyword search (sparse retrieval) matches the actual words. It is called sparse because a document is represented as a vector over the entire vocabulary that is almost all zeros. The standard scoring function is BM25 ('Best Matching 25'), used by search engines for decades. For each query word, BM25 rewards documents that contain it, with three refinements:
- Rare words count more - inverse document frequency (IDF): a word in 2 of 10,000 chunks ('ERR_7731') is far more informative than one in 9,000 ('the', 'policy').
- Repetition has diminishing returns - term-frequency saturation, controlled by parameter
k1(typically about 1.2-2.0): the 2nd mention of a word adds less than the 1st, the 10th adds almost nothing. - Long documents are penalised - length normalisation, controlled by
b(typically 0.75): a word appearing once in a short chunk is stronger evidence than once in a very long one.
BM25 is fast, needs no model, is easy to explain, and nails exact matches - but it fails on synonyms and paraphrases ('holiday' vs 'leave') unless you add synonym lists.
Hybrid search runs both and merges the results. The problem: their scores are on different scales (cosine similarity is around 0-1; BM25 scores are unbounded), so you cannot just add them. The simplest robust solution is reciprocal rank fusion (RRF): ignore the raw scores and use only each document's rank in each list. Each list contributes 1 / (k + rank) to a document's fused score, with k a constant (60 is the common default) that stops the very first ranks from dominating. Documents that appear high in both lists win. An alternative is a weighted sum of normalised scores, which needs tuning.
Other knobs that matter:
- top-k - how many chunks to retrieve. Too few and you miss the answer; too many and you add noise, cost and latency. Common pattern: retrieve generously (20-50) and narrow down with a reranker (see reranking) to the 3-8 you send to the LLM.
- Similarity threshold - drop results below a minimum score so a question with no good match gets no context (and the model can say 'I don't know').
- Metadata filters - restrict by source, date, product, tenant or permissions (see vector-databases).
- Diversity - top results are often near-duplicates (overlapping chunks, copied paragraphs). Maximal marginal relevance (MMR) picks results one by one, penalising similarity to results already chosen.
- Query transformation - rewriting, expanding or splitting the user's question before searching (see conversational-rag and advanced-rag-techniques).
Explain like I'm 10
Semantic search is a librarian who understands what you mean: ask for 'books about feeling homesick' and they bring novels about missing home, even if the word 'homesick' never appears. Keyword search is the index at the back of a book: perfect if you know the exact term ('ERR_7731'), useless for a paraphrase. Hybrid search asks both and trusts the books both of them recommend.
Examples
BM25 from scratch
const docs = [
"Employees receive 25 days of annual leave per year.",
"Error ERR_7731 means the payment gateway timed out. Retry after 30 seconds.",
"Unused annual leave can carry over, up to 5 days.",
"The payment page shows an error when the card is declined.",
"Holiday parties are organised by the social committee.",
];
const tokenize = (s) => s.toLowerCase().replace(/[^a-z0-9_ ]/g, " ").split(" ").filter(Boolean);
const docTokens = docs.map(tokenize);
const N = docs.length;
const avgdl = docTokens.reduce((s, t) => s + t.length, 0) / N;
const df = {}; // in how many docs does each term appear?
for (const toks of docTokens) for (const term of new Set(toks)) df[term] = (df[term] || 0) + 1;
function bm25(query, k1 = 1.2, b = 0.75) {
const q = tokenize(query);
return docTokens.map((toks, i) => {
let score = 0;
for (const term of q) {
const f = toks.filter((t) => t === term).length; // term frequency in this doc
if (f === 0) continue;
const idf = Math.log((N - df[term] + 0.5) / (df[term] + 0.5) + 1);
score += idf * (f * (k1 + 1)) / (f + k1 * (1 - b + b * toks.length / avgdl));
}
return { doc: i, score: +score.toFixed(3) };
}).filter((r) => r.score > 0).sort((x, y) => y.score - x.score);
}
for (const q of ["what does ERR_7731 mean", "annual leave days", "how much holiday do I get"]) {
console.log("Q:", q);
for (const r of bm25(q)) console.log(" " + r.score + " " + docs[r.doc]);
}BM25 finds the error code instantly because 'err_7731' is rare (high IDF). 'annual leave days' ranks the two leave chunks first. But 'how much holiday do I get' matches the holiday PARTY chunk, the only one containing 'holiday', and misses the leave policy entirely - the classic keyword-search failure on synonyms.
Hybrid search: semantic + BM25 merged with reciprocal rank fusion
const docs = [
"Employees receive 25 days of annual leave per year.",
"Error ERR_7731 means the payment gateway timed out. Retry after 30 seconds.",
"Unused annual leave can carry over, up to 5 days.",
"The payment page shows an error when the card is declined.",
"Holiday parties are organised by the social committee.",
];
const tokenize = (s) => s.toLowerCase().replace(/[^a-z0-9_ ]/g, " ").split(" ").filter(Boolean);
// Toy "semantic" embedding: words mapped to shared concepts, so synonyms match
const concepts = { holiday: "leave", vacation: "leave", leave: "leave", "time off": "leave", days: "time",
year: "time", parties: "social", committee: "social", social: "social", payment: "pay", card: "pay", error: "fail", declined: "fail", timed: "fail", err_7731: "fail" };
function embed(text) {
const v = {};
for (const w of tokenize(text)) { const c = concepts[w]; if (c) v[c] = (v[c] || 0) + 1; }
return v;
}
function cosine(a, b) {
let dot = 0, na = 0, nb = 0;
for (const k in a) { na += a[k] * a[k]; if (b[k]) dot += a[k] * b[k]; }
for (const k in b) nb += b[k] * b[k];
return na && nb ? dot / Math.sqrt(na * nb) : 0;
}
const semanticRank = (q) => docs.map((d, i) => ({ doc: i, s: cosine(embed(q), embed(d)) }))
.filter((r) => r.s > 0).sort((a, b) => b.s - a.s).map((r) => r.doc);
// Simple keyword rank: number of shared rare-ish words (BM25 from the previous demo works too)
const stop = new Set(["the", "a", "of", "do", "i", "how", "what", "much", "get", "per", "is", "when", "means", "many"]);
const keywordRank = (q) => docs.map((d, i) => ({ doc: i,
s: tokenize(q).filter((w) => !stop.has(w) && tokenize(d).includes(w)).length }))
.filter((r) => r.s > 0).sort((a, b) => b.s - a.s).map((r) => r.doc);
function rrf(rankings, k = 60) {
const scores = {};
for (const ranking of rankings) {
ranking.forEach((doc, index) => { scores[doc] = (scores[doc] || 0) + 1 / (k + index + 1); });
}
return Object.entries(scores).sort((a, b) => b[1] - a[1]).map(([doc, s]) => ({ doc: +doc, rrf: +s.toFixed(4) }));
}
for (const q of ["how many holiday days do I get", "what does ERR_7731 mean"]) {
const sem = semanticRank(q), kw = keywordRank(q);
console.log("Q:", q);
console.log(" semantic order:", sem.join(", "), "| keyword order:", kw.join(", "));
for (const r of rrf([sem, kw]).slice(0, 3)) console.log(" rrf " + r.rrf + " " + docs[r.doc]);
}For the holiday question, keyword search cannot tell the holiday-party chunk from the leave chunks (each shares one word with the query), while the semantic list knows 'holiday' means leave and ranks the party chunk last; for the error code, keyword search pins the exact chunk. RRF rewards documents ranked well by either retriever, and especially by both, so the fused list is good on both questions without any score normalisation.
Hybrid retrieval in Python: rank_bm25 + Chroma + RRF
# pip install rank_bm25 chromadb sentence-transformers
import re
import chromadb
from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")
col = chromadb.PersistentClient(path="./db").get_or_create_collection("docs")
# Build a BM25 index over the same chunks that live in Chroma
everything = col.get(include=["documents", "metadatas"]) # fine for small/medium corpora
ids, texts = everything["ids"], everything["documents"]
tokenize = lambda s: re.findall(r"[a-z0-9_]+", s.lower())
bm25 = BM25Okapi([tokenize(t) for t in texts])
def keyword_search(query, n=20):
scores = bm25.get_scores(tokenize(query))
order = sorted(range(len(ids)), key=lambda i: scores[i], reverse=True)
return [ids[i] for i in order[:n] if scores[i] > 0]
def semantic_search(query, n=20):
q = model.encode([query], normalize_embeddings=True).tolist()
return col.query(query_embeddings=q, n_results=n)["ids"][0]
def rrf(*rankings, k=60):
scores = {}
for ranking in rankings:
for rank, doc_id in enumerate(ranking, start=1):
scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
return sorted(scores, key=scores.get, reverse=True)
def hybrid_search(query, n=5):
fused = rrf(semantic_search(query), keyword_search(query))[:n]
got = col.get(ids=fused, include=["documents", "metadatas"])
by_id = {i: (d, m) for i, d, m in zip(got["ids"], got["documents"], got["metadatas"])}
return [(doc_id, *by_id[doc_id]) for doc_id in fused]
for doc_id, text, meta in hybrid_search("what does ERR_7731 mean?"):
print(doc_id, meta.get("source"), text[:80])The BM25 index is built in memory from the chunks already stored in Chroma, so both retrievers return the same ids and RRF can merge them. For large corpora keep a persistent keyword index instead (a search engine, or Postgres full-text search as in the next example). col.get(ids=...) does not guarantee order, hence the by_id lookup.
Hybrid search inside Postgres: full-text + pgvector + RRF (SQL)
-- One-time: a generated full-text column with a GIN index
ALTER TABLE chunks
ADD COLUMN tsv tsvector GENERATED ALWAYS AS (to_tsvector('english', content)) STORED;
CREATE INDEX chunks_tsv ON chunks USING gin (tsv);
-- $1 = query embedding, $2 = query text
WITH semantic AS (
SELECT id, ROW_NUMBER() OVER (ORDER BY embedding <=> $1) AS rank
FROM chunks
ORDER BY embedding <=> $1
LIMIT 50
),
keyword AS (
SELECT id, ROW_NUMBER() OVER (ORDER BY ts_rank_cd(tsv, q) DESC) AS rank
FROM chunks, plainto_tsquery('english', $2) AS q
WHERE tsv @@ q
ORDER BY ts_rank_cd(tsv, q) DESC
LIMIT 50
)
SELECT c.id, c.source, c.page, c.content,
COALESCE(1.0 / (60 + s.rank), 0) + COALESCE(1.0 / (60 + k.rank), 0) AS rrf_score
FROM semantic s
FULL OUTER JOIN keyword k ON k.id = s.id
JOIN chunks c ON c.id = COALESCE(s.id, k.id)
ORDER BY rrf_score DESC
LIMIT 10;Postgres's built-in ts_rank_cd is not BM25, but it is a solid keyword ranker, and with RRF only the ranks matter. A FULL OUTER JOIN keeps documents found by only one of the two retrievers. One query, one database, hybrid results.
How it works
Dense retrieval pipeline: embed the query with the same model used for chunks, run a nearest-neighbour search (with filters), return ids, texts, metadata and similarity scores. Some embedding models expect a prefix or instruction on queries versus documents (for example 'query:' and 'passage:'); check your model's documentation, since mismatches hurt quality.
Sparse retrieval pipeline: tokenize (lowercase, split, optionally remove stop words and apply stemming - reducing 'running' and 'runs' to 'run'), look up each query term in an inverted index (a map from each term to the list of chunks containing it), and score only those chunks with BM25. The inverted index makes this fast: you never touch chunks that contain none of the query terms.
BM25 formula for query Q and document D: sum over each query term t of IDF(t) x f(t,D) x (k1 + 1) / (f(t,D) + k1 x (1 - b + b x |D| / avgdl)), where f is the term count in D, |D| the document length and avgdl the average length. IDF(t) = ln((N - n(t) + 0.5) / (n(t) + 0.5) + 1) with N documents and n(t) of them containing t.
Reciprocal rank fusion: for every document d, RRF(d) = sum over retrievers r of 1 / (k + rank_r(d)), with rank starting at 1 and documents missing from a list contributing 0. With k = 60, rank 1 gives 1/61 and rank 10 gives 1/70: being in both lists matters more than being first in one. RRF needs no training and no score calibration, which is why it is the default fusion method.
Learned sparse retrieval is a middle ground worth knowing by name: models that output sparse word-weight vectors (expanding queries with related terms) so they work with inverted indexes but understand some synonyms.
question
/ \
embed + ANN tokenize + BM25
(semantic) (keyword)
| |
1. leave policy 1. holiday party
2. carry-over 2. leave policy
3. holiday party 3. ...
\ /
reciprocal rank fusion
score = sum 1/(60 + rank)
|
1. leave policy (high in both)
2. holiday party
3. carry-over
|
top 20-50 -> reranker -> top 5Why does it exist?
Neither retrieval method is good enough alone. Semantic search understands paraphrases but fumbles exact identifiers; keyword search nails identifiers but misses paraphrases. Real users ask both kinds of questions, often in the same sentence. Hybrid search exists to cover both failure modes cheaply, and RRF exists because merging scores from incompatible scales is otherwise fragile.
When to use it
Use semantic search as the baseline for natural-language questions. Add BM25 and fuse with RRF whenever your content has codes, names, version numbers, legal citations, product SKUs or domain jargon - which is most real corpora. Measure each retriever separately and the fused result on a test set to confirm hybrid helps for your data.
When not to use it
If users only ever search by exact identifiers, a plain database or keyword index is enough. If the corpus is tiny, retrieval strategy hardly matters - consider putting everything in the prompt. Do not add hybrid complexity before you have a baseline and an evaluation set showing what it fixes.
Common mistakes
Relying only on embeddings and then being surprised that exact error codes, SKUs or names are not found.
Adding raw BM25 scores to cosine similarities, which lets whichever scale is larger dominate.
Using a top-k that is too small to have the answer in it, then blaming the LLM.
Tokenizing queries and documents differently for BM25 (different lowercasing or punctuation rules), so terms never match.
Returning many near-duplicate chunks (from overlap or copied content), wasting the context on one fact.
Ignoring the query prefixes or instructions an embedding model expects, degrading semantic matches.
Never inspecting what was retrieved for failed questions - most 'LLM errors' in RAG are retrieval errors.
Practice exercises
- Easy:
In the BM25 demo, change b to 0 and then to 1. Which documents move up or down for 'annual leave days' and why?
- Easy:
Compute by hand the RRF score with k = 60 of a document ranked 1st by semantic search and absent from keyword search, and of one ranked 3rd in both. Which wins?
- Medium:
Replace keywordRank in the hybrid demo with the BM25 function from the first demo and check the fused results are still sensible.
- Medium:
Implement MMR in JavaScript: given a query vector and candidate vectors, repeatedly pick the candidate maximising 0.7 x sim(query, c) - 0.3 x max sim(c, already chosen). Show that it skips a near-duplicate.
- Hard:
Build a 30-question test set for your own documents (questions + id of the chunk that answers each). Measure recall@5 for semantic, BM25 and hybrid (RRF) retrieval with the Python example. Report which question types each method fails on.
Interview questions
What is the difference between dense and sparse retrieval?
Dense retrieval embeds queries and documents into dense vectors and finds nearest neighbours, capturing meaning and paraphrase. Sparse retrieval represents text by the words it contains (mostly-zero vectors over the vocabulary) and scores exact term matches, typically with BM25 via an inverted index. Dense handles synonyms; sparse handles exact identifiers and rare terms.
Explain BM25's main components.
IDF makes rare terms count more than common ones; term-frequency saturation (k1) gives diminishing returns for repeated occurrences; length normalisation (b) penalises long documents relative to the average length. The score is the sum over query terms of IDF times the saturated, length-normalised term frequency.
What is reciprocal rank fusion and why is it popular?
A method to merge ranked lists: each document scores the sum of 1/(k + rank) over the lists it appears in, typically with k = 60. It uses ranks only, so it needs no score normalisation or training, is robust across very different retrievers, and rewards documents that several retrievers agree on.
When does semantic search fail?
On exact tokens with little semantic content (error codes, part numbers, names, versions), on rare domain jargon the embedding model never learned, on negations and precise numeric constraints, and on very short queries. Hybrid search and metadata filters address many of these cases.
How do you choose top-k?
Retrieve a generous candidate set (for example 20-50) to maximise recall, then rerank and send a smaller set (for example 3-8) to the LLM to keep precision, cost and latency in check. Tune both numbers with recall@k on an evaluation set and by checking answer quality.
Why might you set a similarity threshold?
So that a question with no relevant content returns no context, letting the system say it does not know instead of answering from weakly related chunks. Thresholds must be calibrated per embedding model since score distributions differ, and are more reliable on reranker scores.