You’ve built a Retrieval-Augmented Generation (RAG) system. The embedding model is state-of-the-art, your vector database is lightning-fast, and the LLM is highly capable. During the demo, it works beautifully.

Then you deploy to production.

Suddenly, the system begins to hallucinate or return confidently wrong answers. It invents error codes or provides semantically plausible responses that are operationally useless. Most developers react by tuning the prompt, lowering the temperature, or switching to a larger model. These efforts rarely solve the underlying issue. The problem is almost never the LLM; it is a failure of retrieval.

Traditional search is designed to help humans find relevant links; if a result is slightly off, the human filters it out. In RAG, retrieval is a “fact-assembly pipeline.” The LLM does not browse; it treats every retrieved chunk as absolute ground truth. If you feed it the wrong context, no amount of prompting can save the output.

1. Stop Blaming the LLM: Precision Matters More Than Recall

In traditional information retrieval, “recall” (finding as many relevant documents as possible) is often the priority. In RAG, however, the model’s “helpfulness” becomes a liability. LLMs are optimized to synthesize information, not to critically evaluate source quality. They will attempt to make sense of whatever context you provide, even when they shouldn’t.

If you miss a critical chunk (a recall failure), the model doesn’t usually say “I don’t know.” It fills the gaps with its own training data, leading to a silent, confident failure. Conversely, if you provide too much information — such as adding a 6th or 7th chunk that is irrelevant or conflicting — the noise can actively mislead the model. In production environments, providing that extra “6th chunk” often makes the answer worse by introducing conflicting or irrelevant noise that the LLM is forced to reconcile.

“The language model can only work with what you give it. Feed it the wrong context and no amount of prompting will save you.”

2. The Surprising Renaissance of BM25 (Why “Old School” Beats “State-of-the-Art”)

A common production trap is assuming that semantic “dense” retrieval (embeddings) is always superior to “old school” keyword search (BM25). In technical and financial domains, BM25 frequently outperforms embeddings.

This is due to the fundamental mechanism of Inverse Document Frequency (IDF). Dense retrieval models are trained on natural language and tend to “smooth” unique identifiers into general semantic categories. For example, a dense model might treat a specific error code like “E4392” or an identifier like “LDAP_AUTH_001” as noise, prioritizing a chunk about general “connection errors” because it feels semantically similar.

BM25, conversely, upweights these rare, specific terms. Because “E4392” appears in very few documents, its IDF score is high, ensuring the exact document is prioritized. The T2-RAGBench finding confirms this, showing that BM25 outperforms OpenAI’s text-embedding-3-large on financial text-and-table benchmarks. BM25 ensures you don’t “drop facts on the floor” simply because they lack semantic richness.

3. “Situating” Your Data: The Magic of Contextual Retrieval

While BM25 fixes keyword gaps, it does not solve the “fragmentation” problem. Traditional RAG splits documents into isolated chunks, causing them to lose their relationship to the parent document. A chunk stating “revenue increased by 15%” is useless if the retriever loses the context of which company or which quarter it refers to.

Anthropic’s Contextual Retrieval addresses this by prepending a short, explanatory summary to each chunk before it is indexed. This “situates” the chunk within its broader context.

  • Contextual embeddings alone can reduce top-20 retrieval failure rates by 35%.
  • Contextual BM25 can reduce top-20 retrieval failure rates by 49%.

Crucially, Contextual BM25 is implemented by indexing the LLM-generated context summary alongside the raw text in the BM25 index. This allows keywords to be matched against both the specific chunk and the global document context. While this sounds expensive, prompt caching makes this process cost-effective during ingestion by offering discounts of up to 90% on repeated document tokens.

4. Hybrid Search and Reranking: The Non-Negotiable Two-Stage Pipeline

Choosing between BM25 and dense retrieval is a false dichotomy. The gold standard for production RAG is a two-stage pipeline: hybrid search (fusing results via Reciprocal Rank Fusion) followed by a cross-encoder reranker.

Hybrid search ensures you don’t miss intent (dense) or specific facts (BM25). However, initial retrievers are often bi-encoders that score queries and documents separately for speed. This leads to the “Position 8” problem — also known as the “lost in the middle” phenomenon — where the correct answer is in the candidates but ranked too low for the LLM’s context window.

A cross-encoder reranker fixes this by processing the query and document together using “cross-attention.” It is more computationally expensive but significantly more precise. In a study of ~1,200 manually evaluated query-answer pairs in customer support scenarios, adding a reranker reduced hallucination rates from ~23% to ~11%.

5. The Engineering Reality Check: CPU vs. GPU Costs

As a senior architect, I advise a pragmatic approach to infrastructure, particularly in the Indian market where GPU costs are a significant overhead. GPU infrastructure is a luxury you buy when you have revenue, not when you are experimenting.

  • CPU-only setup: An AWS t3.xlarge costs approximately ₹5,400/month. This can handle embedding, BM25, and reranking for up to 5,000 queries per day with latencies of 150-250ms.
  • GPU-accelerated setup: An AWS g4dn.xlarge (T4 GPU) costs approximately ₹34,558/month.
  • The spot instance reality: For cost-conscious engineering, GPU spot instances can bring the cost down to ₹10,000–₹18,000/month, though they require graceful failure handling for interruptions.

CPU-first is a valid architectural choice. Start with CPU-based retrieval and only scale to GPUs if your latency requirements (sub-100ms) or massive revenue demand it.

Conclusion: From Search Engines to Fact Pipelines

Building a production-ready RAG system is not about chasing the “fanciest” model; it is about an engineering-first understanding of how retrieval fails in the real world. Success requires moving away from a search-engine mindset and toward a high-precision fact-assembly mindset.

As you evaluate your system, ask yourself: are you treating your RAG system as a search engine that shows links, or as a fact-assembly pipeline that demands absolute precision?