BM25 vs vector search: why production RAG usually needs both

Most teams building retrieval reach for embeddings first and never benchmark keyword search on their own data. That ordering is worth questioning, because the two methods do not merely differ in quality. They fail in opposite directions, which is what makes the combination worth more than either alone.

Is BM25 better than vector search?

BM25 is better for exact lexical evidence. Vector search is better when the relevant document uses different words from the query. Neither replaces the other.

BEIR evaluated 18 publicly available retrieval datasets and 10 retrieval systems zero-shot, and found BM25 remained a strong baseline while dense and sparse retrieval models often underperformed other approaches out of domain (BEIR, NeurIPS 2021).

State the date plainly, because it matters: that benchmark is from 2021, and embedding models have improved substantially since. It does not establish that BM25 beats modern embedding models today. What it supports is narrower and still useful: keep a lexical baseline, and do not assume out-of-domain generalization you have not measured.

What is the difference between keyword and semantic search?

Keyword search ranks by token overlap and term statistics, with BM25 weighting informative query terms while discounting repetition and unusually long documents. Semantic search maps queries and documents into vectors and ranks by similarity in that space.

BM25 preserves literal identity. A query containing ERR_CONNECTION_RESET, SKU-4817 or a quoted configuration key retrieves that exact string. Vector search compresses text into a representation of meaning, so rare tokens contribute less reliably than their exact lexical form.

That is the failure that bites hardest in practice: a user pastes an exact error string and receives semantically related network failures instead of the page containing the error they actually have.

Vector search preserves conceptual similarity, which BM25 cannot fake. "Why does the worker keep restarting?" can retrieve a document titled "diagnosing repeated process crashes" with no shared vocabulary at all.

When should you use BM25?

When the query contains a token whose spelling is itself the evidence. Error codes, API method names, version strings, file paths, product identifiers, legal citations and quoted text all qualify, and an embedding has no obligation to preserve any of them exactly.

BM25 also makes a good first benchmark, because it tests whether your corpus is searchable through its own vocabulary at all. If a lexical baseline already retrieves the correct chunk, an embedding model is not solving a retrieval problem that exists in your data, and any improvement on paraphrased queries should be measured against that baseline rather than against nothing.

It has one more underrated property: its failures are legible. Query terms, document fields, analyzers, stemming choices and ranking contributions can all be inspected. That does not make BM25 sufficient for every corpus. It makes its behavior diagnosable, which is not true of a vector index that simply returns the wrong neighbor.

When does vector search win?

When relevance depends on meaning expressed in different surface forms. A user asks about rotating credentials while the documentation says replacing access keys; a user asks why a job restarts while the incident report describes process crashes. Dense retrieval connects those without a shared token.

How much you gain depends on the embedding model, corpus, language, chunking and query distribution, which is exactly why the 2021 benchmark cannot answer it for you. You need a held-out query set from your own workload, with judged relevant documents and labelled failure categories.

One caution: vector search should not earn authority merely because its results look plausible. Plausibility is not retrieval. A related document is still the wrong document when the query names a particular release, error, contract clause or code symbol, and a convincing near-miss hurts generation more than obvious noise.

What is hybrid search?

Running lexical and vector retrieval for the same query, then fusing the candidate lists into one ranking. BM25 contributes exact-term precision, vector search contributes semantic recall. Reciprocal Rank Fusion combines ranked lists using positions rather than scores, so the two retrievers never need a shared calibrated scale.

The fused list can go straight to the model or through a reranker that evaluates query and passage together. The BEIR authors reported reranking and late-interaction methods achieving the best average zero-shot performance, at high computational cost. Again, that is a 2021 result: treat it as evidence for measuring the latency-quality tradeoff, not as a permanent ordering of current models.

How should you choose?

By query shape first, then verify with a corpus-specific benchmark. BM25 for literal queries, vector search for paraphrase-heavy ones, hybrid when both classes occur, and a reranker when the first-stage candidate set contains the answer but orders it badly.

Label four failure classes in your evaluation set: exact-token queries, paraphrased questions, mixed queries carrying identifiers and natural language, and queries with no answer in the corpus. Report retrieval recall at the context budget your generator actually consumes, because a retriever that finds the answer at rank 40 is not equivalent to one that finds it at rank 5 when the pipeline forwards only a small candidate set.

Hold everything else constant across runs: same chunks, filters, query-rewriting policy and relevance judgments for BM25, vector and hybrid alike. Otherwise the benchmark measures pipeline changes rather than retrieval methods.

Two adjacent decisions are worth keeping separate from this one. How to read a similarity score, and why absolute thresholds do not port, is covered in cosine similarity and what it actually means. How an agent reaches a corpus in the first place belongs with MCP, APIs, and RAG. Retrieval-method selection sits between them and should follow the query distribution inside the corpus you already have.