Retrieval Is a Ranking Problem

When a retrieval-augmented system gives a weak answer, the instinct is to blame the model or the vector database. Usually the fault is upstream: the right passage was never placed in front of the model.

Similarity is not relevance

A single embedding per chunk (a bi-encoder) is a coarse instrument. It maps text to a point and measures cosine distance. That captures topical similarity well and precise relevance poorly. Two passages about the same subject sit close together even when only one actually answers the question. First-stage vector search is a recall device: its job is to not miss the answer, not to rank it first.

Two stages beat one

The reliable pattern is retrieve-then-rerank. Use cheap recall — dense embeddings, BM25, or both — to pull a few dozen candidates. Then run a cross-encoder over the query paired with each candidate. A cross-encoder reads both texts together and scores actual relevance, not geometric proximity. It is too expensive to run over the whole corpus and exactly right for re-ordering the top-k.

Lexical and dense fail differently

Dense retrieval understands paraphrase but misses exact terms — part numbers, error codes, proper nouns. Lexical search (BM25) nails exact terms but misses synonyms. Hybrid retrieval, fused with something as simple as reciprocal rank fusion, beats either alone because their failures do not overlap.

Spend effort where the leverage is

Chunking strategy, query rewriting, hybrid recall, and a reranker move quality far more than the choice of vector store. And none of it is trustworthy without measurement: track recall@k for the recall stage and nDCG for the ranked result. If you are not measuring retrieval in isolation, you are tuning the generator to compensate for a problem it cannot see.

Retrieval is a ranking problem. Treat it like one.

References

1. Lewis, P., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401. https://arxiv.org/abs/2005.11401

2. Karpukhin, V., et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering. arXiv:2004.04906. https://arxiv.org/abs/2004.04906

3. Nogueira, R., & Cho, K. (2019). Passage Re-ranking with BERT. arXiv:1901.04085. https://arxiv.org/abs/1901.04085

devinfo.dev. (2026). "Retrieval Is a Ranking Problem." devinfo.dev:2026.0066. https://devinfo.dev/d/2026.0066