Why RAG demos impress and production RAG fails, and the engineering that closes the gap: permission-aware retrieval, hybrid search, citations, and evals.
A retrieval-augmented generation demo takes an afternoon: index a folder of PDFs, wire up a chat box, ask a question, get a plausible answer with a citation. Production RAG is a different engineering problem, and most of the difference has nothing to do with the model. This post covers where knowledge assistants break after the demo, the retrieval and evaluation work that fixes them, and when RAG is the wrong tool altogether.
Demo conditions are generous. The documents are curated, the questions come from someone who already knows the answers, and there is one user with access to everything. Production removes all three. The corpus is thousands of documents that are stale, duplicated, and permission-scoped; the questions are phrased badly by people who do not know what they are looking for; and the person asking is often not allowed to see the document that contains the answer.
The failures cluster in five places:
This is consistent with the wider pattern in enterprise AI: Anaconda and Forrester research reports that 88% of agent pilots never reach production, while Gartner's Q1 2026 data shows 80% of enterprises already have at least one production app embedding an AI agent. The gap is not a modelling problem; it is a retrieval, permissions, and evaluation problem.
Retrieval must run as the user, not as the system. That means three things:
Embeddings alone are not enough for enterprise corpora. Semantic search is good at "how do I get reimbursed for travel" and bad at exact tokens such as part numbers, error codes, invoice IDs, and people's names. Production systems combine dense retrieval with lexical (keyword) retrieval, then rerank the merged candidates, and apply metadata filters for date, product line, or region before any of that runs.
Chunking should follow document structure. Keep headings as a breadcrumb prefix on each chunk, keep tables whole, and overlap chunk boundaries so a sentence split across two chunks is still recoverable.
Every factual claim the assistant makes should link to a passage the user can open and read. If retrieval returns nothing relevant, the correct answer is a clear statement that no source was found, not a fluent paragraph from the model's own memory. Show the supporting snippet inline and log which citations users open.
The evaluation set is the single most valuable artifact in a RAG project, and it should come from real demand: support tickets, internal help-desk threads, search logs, the questions people actually ask in Slack. A practical starting point is a few hundred questions, each with the expected answer and the document that supports it, plus the deny cases above.
Measure at least four things on every change:
Run this on every change to chunking, embeddings, prompts, retrieval settings, and models. Model choice in particular should be gated by your evaluation set, not by a launch announcement. Anthropic's Claude Opus 5, released in July 2026, is described as coming close to the frontier intelligence of Claude Fable 5 at half the price; whether that trade is right for your assistant is a question your eval answers in an hour and a press release cannot answer at all.
Launch is where the interesting data starts. Track index freshness lag, the rate of empty retrievals, explicit thumbs-down, escalations to a human, latency, and cost per answered question. Review a sample of conversations weekly, and feed the failures back into the evaluation set so the next release is tested against last week's problems.
These are complementary, not competing:
Our enterprise RAG and AI knowledge assistant work starts by baselining the workflow: what people ask today, where they look, and how long it takes. We instrument the pipeline from day one, build the evaluation set from real tickets before writing retrieval code, and treat evaluation as a launch gate rather than a report. Human-in-the-loop is designed in, from citation review to escalation paths, and we operate the assistant after launch because the index, the permissions, and the questions all keep changing. Where the assistant needs to take action rather than answer, it becomes part of an AI agent with the same identity and audit controls; where the front end is a support or sales channel, it ships through AI chatbot development. If you are unsure whether RAG, fine-tuning, or long context fits your case, AI consulting is the right starting point. Contact us if your assistant is stuck between demo and production.
Answers
A few hundred real questions with expected answers and supporting documents is a practical starting point, and it should include cases where the correct response is a refusal. Grow it after launch by adding every failure you find in weekly review.
Only if the corpus is small, stable, and shareable with every user. Long context gives you neither per-document permissions nor precise citations, and it becomes expensive when documents change often. Most enterprise assistants still need retrieval for those reasons.
Index access-control metadata with every chunk, keep it synced with the source system, and apply the filter during retrieval rather than after selecting the top results. Then test deny cases explicitly in your evaluation set so regressions are caught before release.
Put it into practice
The XISLABS services closest to what this article covers.
Keep reading
Stop picking one winner. A three-tier model strategy (frontier, workhorse, self-hosted) with routing by task, data residency, and eval-driven switching.
Cost per task, model tiering, prompt caching, routing, batch processing, and hard caps: how to run LLM features cheaply without a quality regression.
What the 2026-07-28 MCP spec changes mean operationally, what the adoption numbers say, and how to expose internal systems as MCP servers safely.
We build the AI agents, automation, and software behind ideas like these — scoped to a metric, shipped in weeks, operated after launch.