A retrieval-augmented generation prototype is easy. Point a loader at some PDFs, embed the chunks, drop them in a vector store, stuff the top few hits into a prompt, and you have a demo that answers questions about your documents in an afternoon. Then it meets real users, real documents, and real questions, and it falls apart. The answers cite the wrong section, miss the one paragraph that mattered, or confidently invent a figure that is nowhere in the source. The model is rarely the problem. The pipeline around it is. This is a walkthrough of the stages that decide whether a RAG system survives contact with production, based on systems we have built and run.
Ingestion and parsing
Everything downstream inherits the quality of your parsing. If a PDF table comes out as a wall of space-separated numbers, no embedding model or reranker will rescue it. The demo version of ingestion reads plain text and stops there. The production version has to deal with PDFs that are really scanned images, HTML with navigation chrome mixed into the body, tables that carry the actual answer, and documents that update on their own schedule.
Treat parsing as its own problem with its own quality bar. Extract structure, not just text: headings, lists, tables, and the boundaries between sections. Keep the metadata that retrieval and citations will need later, including source document, page or section, and a stable identifier so you can trace an answer back to where it came from. For anything with layout that matters, a layout-aware parser earns its keep over a naive text dump.
Plan for re-ingestion
Documents change. A pipeline that can only do a full rebuild will either go stale or cost you a fortune in re-embedding. Track content hashes per document so you re-process only what actually changed, and make ingestion idempotent so a re-run does not duplicate chunks in the index. This is unglamorous plumbing that no demo needs and every production system does.
Chunking: the highest-leverage decision
Chunking is where most of the answer quality is won or lost, and it gets the least attention. A chunk is the unit you retrieve, so its boundaries decide what context the model can ever see. Chunk too small, and a single fact gets split across two pieces and neither retrieves well. Chunk too large, and the embedding blurs several topics together so it matches everything and discriminates nothing.
Fixed-size character splitting is the default in every tutorial and it is rarely the right answer for real documents. Prefer chunking that respects structure: split on headings and section boundaries, keep tables and lists intact, and avoid cutting a sentence in half. A modest overlap between adjacent chunks helps preserve context across boundaries, but overlap is a patch, not a strategy. The strategy is to align chunk boundaries with the natural units of meaning in the source.
Carry context into the chunk
A chunk pulled out of a long document loses the context that gave it meaning. A paragraph that says "this applies only to existing customers" is useless if the chunk does not record which product and which policy it belongs to. Prepending a short header to each chunk (document title, section path) or storing that context as metadata gives both retrieval and the model the anchor they need. Because chunking changes propagate through embeddings, retrieval, and answers all at once, it is the first thing to tune and the thing worth measuring against a labelled set before you touch anything else.
Embeddings and the vector store
With sensible chunks in hand, embeddings turn each chunk into a vector so you can search by meaning rather than by keyword. The model choice matters less than people expect once chunking is sound, but a few things are easy to get wrong. Use the same embedding model for documents and queries, and pin its version: silently swapping embedding models invalidates your whole index, because old and new vectors no longer live in the same space. Re-embedding the corpus is the cost of any change here, so decide deliberately.
The vector store is the part people obsess over and it is often the least interesting decision. For most workloads, the choice between a dedicated vector database and a vector extension on a database you already run comes down to operational fit, not raw recall. If you already operate Postgres, pgvector keeps your data in one place. If your retrieval is fundamentally about relationships between entities rather than passage similarity, a graph store can be the better backbone. On the AllBright Collective recommendation API we built retrieval on Neo4j precisely because the useful signal lived in connections, not in document text. The store should follow the shape of your data, as we cover in that case study.
Hybrid search beats pure vectors more often than expected
Pure semantic search struggles with exact terms: product codes, error numbers, names, acronyms. A user who searches for an exact identifier wants that exact match, and a fuzzy vector neighbour is the wrong answer. Combining keyword search with vector search, then merging the results, covers both the "find the concept" and "find this exact token" cases. It is one of the cheapest reliability wins available.
Retrieval and the role of a reranker
The initial retrieval is a fast, approximate first pass. It is tuned for speed and recall, which means it casts a wide net and returns more candidates than you want to send to the model. If you take the top few of those and stop, you are trusting a coarse similarity score to rank the most relevant passage first. It often does not.
A reranker fixes this. Retrieve a generous candidate set (twenty, forty, more), then run a cross-encoder reranker that scores each candidate against the query directly and reorders them. The reranker sees the query and the passage together, so it judges relevance far more precisely than the first-pass vector score. In practice this is the single change that most reliably lifts answer quality, and it is cheaper than upgrading the generation model. The cost is latency and a second model call, so size the candidate set to balance the two.
A retrieval pass that holds up in production
- Hybrid first pass: keyword and vector search merged for both concept and exact-term matches.
- Generous candidate set: retrieve far more than you will use, so the right passage is in the pool.
- Cross-encoder reranker: reorder candidates by direct query-passage relevance, then keep the top few.
- Metadata filters: scope retrieval by tenant, document type, or recency before ranking, not after.
Grounding and prompt assembly
Once you have the right passages, the prompt decides whether the model uses them honestly. Grounding means the answer is built from the retrieved context and nothing else. The instruction has to be explicit: answer from the provided context, and if the context does not contain the answer, say so rather than guessing. A model that says "I do not have that information" is doing its job. A model that fills the gap from its training data is a liability in any domain where being wrong has consequences.
Assemble the context with structure the model can use. Label each passage with its source so the model can cite it, and so you can show users where an answer came from. Order matters: models attend unevenly across a long context, so put the strongest passages where they are most likely to be read rather than burying them in the middle. Mind the budget too. Stuffing forty passages into the prompt raises cost and dilutes the signal; a handful of well-ranked passages usually beats a heap of mediocre ones. Deciding how retrieval, prompt assembly, and model calls fit together is the orchestration layer, which we cover in what is LLM orchestration.
Evaluation: retrieval metrics and generation metrics
You cannot improve what you do not measure, and "it looked good when I tried it" is not measurement. The mistake is to evaluate the system end to end as a single black box. When an answer is bad, you need to know whether retrieval failed to find the right context or generation failed to use the context it was given. So measure the two separately.
Retrieval metrics
Did you find the right context? Context precision and recall against a labelled set tell you whether the relevant passages were retrieved and ranked highly. Most failures live here.
Generation metrics
Given the retrieved context, was the answer faithful to it and relevant to the question? Faithfulness catches the model inventing facts; answer relevance catches it dodging the question.
Building this takes a labelled evaluation set: representative questions paired with the passages that should be retrieved and the answers that count as correct. It is tedious to assemble and it is the asset that makes every later decision data-driven instead of a guess. Without it, tuning chunk size or swapping a reranker is gambling.
Run evaluation in CI
Once you have an eval set, wire it into continuous integration so any change to chunking, prompts, models, or retrieval parameters runs against it before it ships. RAG systems are full of changes that look harmless and quietly degrade quality: a new embedding model version, a tweaked chunk size, a prompt edit. An eval gate in CI turns those into visible regressions you catch before users do. The build cost of this is the difference between a system you can change with confidence and one you are afraid to touch.
Observability and iteration
A RAG system in production is never finished, because the questions users ask and the documents they ask about both keep changing. You need to see what is actually happening: what users asked, what got retrieved, what the reranker did with it, what context reached the model, and what it answered. Log the full chain per request, not just the final answer. When someone reports a bad answer, that trace is the difference between a five-minute diagnosis and an afternoon of guessing.
Watch the queries that retrieve nothing useful, because they tell you where the corpus has gaps or where chunking is splitting answers badly. Feed real failures back into the evaluation set so the system gets measurably better at the things it was getting wrong. On the British Council AiBC work, the iteration loop, seeing real usage, turning failures into test cases, and tuning retrieval against them, mattered more than any single component choice. This loop is the core of how we build AI agents and LLM products.
What separates a demo from production
The demo and the production system share an architecture and almost nothing else. A demo retrieves; a production pipeline retrieves the right thing, reliably, and can prove it. The gap is mostly the unglamorous parts: parsing that respects structure, chunking aligned to meaning, a reranker on the retrieval path, grounding that refuses to guess, and an evaluation harness that runs in CI. Before you reach for retrieval at all, it is worth being sure RAG is the right tool for the job, which is the subject of RAG vs fine-tuning.
The production checklist
- Parsing that preserves structure and metadata, with idempotent re-ingestion driven by content hashes.
- Chunking aligned to the document's natural units of meaning, tuned first and measured against a labelled set.
- A pinned, versioned embedding model and a vector store chosen to fit the shape of the data.
- Hybrid retrieval feeding a cross-encoder reranker, so the most relevant passage reaches the model.
- Grounded prompts that cite their sources and say so when the answer is not in the context.
- Retrieval and generation evaluated separately, with the eval set wired into CI as a regression gate.
- End-to-end request tracing and a feedback loop that turns real failures into new test cases.