RAG in production: what actually matters

A RAG demo takes an afternoon. A RAG system you can trust in production takes real engineering — because the failure modes only show up at scale, with real documents, and with users who ask questions you didn't anticipate.

Retrieval-augmented generation is now the default architecture for any AI system that needs to answer questions grounded in a specific corpus: internal documentation, customer records, product knowledge bases, legal contracts, support tickets. The idea is simple: instead of relying on what the language model memorized during training, you retrieve relevant chunks from your own data at query time and include them in the prompt. The model reasons over your data rather than its priors.

That simplicity is a trap. The gap between 'it works in my notebook' and 'it works reliably in production' is where most AI projects stall.

Retrieval quality is everything

The model is only as good as what you put in front of it. If your retrieval step returns the wrong chunks — chunks that are topically related but not actually relevant to the specific question — the model will hallucinate confidently, because it's trying to generate a coherent answer from bad evidence. Garbage in, plausible-sounding garbage out.

Most teams underinvest in retrieval. They use basic cosine similarity over OpenAI embeddings, get decent results on their test queries, and call it done. Then a user asks a slightly different phrasing and the system returns irrelevant results with 0.78 similarity scores.

The things that meaningfully improve retrieval quality: chunking strategy (how you split documents matters enormously — splitting mid-sentence or mid-concept degrades embedding quality), metadata filtering (letting users or the system narrow the search space before semantic search runs), hybrid search (combining dense vector search with BM25 keyword search to handle exact-match queries that semantic search misses), and re-ranking (running a cross-encoder over your top-k retrieved chunks to re-order them before passing to the model).

None of these are exotic. All of them require deliberate implementation and ongoing measurement.

Evaluation is what makes it shippable

You cannot improve what you don't measure. The most common failure mode we see in RAG projects is shipping without a proper eval harness — which means every change to the system (different chunking, a new embedding model, a prompt tweak) is a coin flip. Did it get better? Worse? Nobody knows.

A minimal eval set for a RAG system consists of: a set of representative questions, the expected answer or expected source document for each, and a way to run them automatically. From that baseline you can calculate retrieval precision (did the right chunks come back?), answer faithfulness (is the model's answer grounded in the retrieved chunks?), and answer correctness (is it actually right?).

Building this eval harness is not glamorous work, but it's the difference between a system you can confidently iterate on and one you're afraid to touch. We treat evals as a first-class deliverable on every AI engagement — not something bolted on at the end.

Tools like RAGAS, TruLens, and Langfuse make this more tractable than it was 18 months ago. The tooling has matured. The discipline of actually using it has not caught up.

Guardrails are what make it safe to ship

A language model will, given the chance, confidently answer questions outside the scope of your corpus, make up citations, and occasionally output things that are technically coherent but factually wrong. In a demo, this is fine. In production, it's a liability.

Guardrails are the engineering layer that prevents that liability from reaching users. The three most important:

Source attribution. Every answer should cite the specific document chunks it drew from. This lets users verify claims, and it lets your team audit failures. A RAG system without source attribution is a black box that's impossible to debug.

Refusal behavior. If the retrieved chunks don't contain a confident answer to the question, the system should say so rather than speculate. A well-designed refusal is far better than a plausible-sounding hallucination. Implementing this requires threshold logic on retrieval confidence and explicit prompt instructions, and it needs to be tested against edge cases.

Human review gates for high-stakes outputs. Not every RAG deployment is equal. A system answering questions about your marketing copy has very different stakes from one summarizing medical records or generating legal contract clauses. Where stakes are high, build in a human review layer — a UI for human sign-off before outputs are acted on. This is not a concession to AI's limitations; it's good engineering practice for any automated system making consequential decisions.

The infrastructure considerations teams skip

Latency is a user experience problem. A RAG pipeline that takes 8 seconds end-to-end — document retrieval, re-ranking, prompt assembly, LLM call — will have lower adoption than one that responds in under 2 seconds. Streaming responses help with perceived latency, but the underlying retrieval and inference pipeline still needs to be optimized. Caching frequently-asked questions, running retrieval and re-ranking as parallel operations, and choosing the right embedding model for your latency budget all matter.

Cost scales with use. Embedding every query against a large vector store and passing 4,000 tokens to GPT-4 on every request has a cost profile that looks fine in testing and expensive in production. Model tier selection, prompt compression, and caching strategies are the dials you'll need to tune as usage grows.

Context window management is a real problem. As RAG systems mature, teams want to pass more context to the model — more retrieved chunks, longer documents, conversation history. This hits context limits faster than expected and degrades answer quality in subtle ways (models tend to focus on content at the beginning and end of long prompts, a phenomenon called the 'lost in the middle' problem). The fix is deliberate context management: ranking chunks by relevance, truncating aggressively, and testing what actually improves quality rather than assuming more context is always better.

What separates the demos from the deployments

The teams that ship reliable RAG systems treat it like any other production engineering problem: define what 'working' means before you build, measure it continuously, and build the guardrails that make failures recoverable. The teams that end up with expensive failed pilots treat the LLM call as the hard part and everything around it as a formality.

The LLM call is not the hard part. The pipeline around it is.