The question almost never resolves to one or the other. Teams ask whether they should use retrieval-augmented generation or fine-tuning as if it were a fork in the road, when in practice the two solve different problems and often sit in the same system. The decision that matters is about cost, how fresh your knowledge needs to be, and who maintains the thing once it is live. Get those three right and the technical choice mostly falls out on its own.
What each technique actually does
The two get compared as if they are interchangeable, but they operate at different layers of the system, and that is the whole point.
RAG: retrieve, then generate
Retrieval-augmented generation keeps the model fixed and changes what you put in front of it. At query time you search a knowledge source (a vector index, a keyword search, a database, or some combination), pull back the passages that look relevant, and pass them into the prompt as context. The model then answers using that context rather than relying on whatever it absorbed during training. Your knowledge lives outside the model, so you update it by changing the index, not the weights.
Fine-tuning: adjust the weights
Fine-tuning takes a base model and continues training it on your own examples, nudging the weights so the model behaves differently by default. You are not teaching it new facts so much as teaching it a way of responding: a house style, a strict output format, a classification it should reach without being walked through it each time. The knowledge and the behaviour are baked in, which is the strength and also the liability.
A short version that holds up most of the time: RAG changes what the model knows at the moment of answering, fine-tuning changes how the model behaves in general. Facts that move belong in retrieval. Behaviour that should stay constant is a candidate for fine-tuning.
A decision framework
Skip the feature table. The useful questions are about your data and your team, not about which technique sounds more advanced. Here is how the two tend to break down once you frame it that way.
Reach for RAG when
Your knowledge changes often, you need to cite sources, freshness matters, the corpus is large, and you want a low upfront cost with answers you can trace back to a document.
Reach for fine-tuning when
The behaviour is stable, you need a consistent format or tone, you want to shorten prompts to save tokens and latency at scale, or the base model keeps drifting from the style you need.
On the British Council AiBC platform, the knowledge that learners ask about shifts constantly, so the parts that touch current material lean on retrieval rather than anything baked into a model. You can read more about that work in the British Council AiBC case study. The general rule held: anything that has to stay current or be cited goes through retrieval, and the model stays replaceable.
Choose RAG when most of these are true
- The underlying knowledge changes weekly or faster.
- You need to show users where an answer came from.
- You want to start shipping without a training run.
- The corpus is too large or too varied to fit in a fixed model.
- Swapping the base model later should not mean losing your knowledge.
When to combine the two
The mature setups usually do both, and they do it cleanly. Fine-tune for the parts that should never change (output schema, tone, how the model handles an instruction it sees a thousand times a day) and use RAG for the parts that must stay current (the facts, the documents, anything a user might want a citation for). The fine-tune handles behaviour, retrieval handles knowledge, and neither is asked to do the other's job. That separation is also what keeps maintenance sane: you can refresh the index daily without touching the model, and retrain the model occasionally without rebuilding your retrieval.
The costs each side hides
Both techniques look cheap in a demo and reveal their real cost in production. The failure modes are different, and knowing them ahead of time is most of the value of choosing deliberately.
What RAG hides
The model is rarely the problem in a RAG system. Retrieval quality is, and retrieval quality is mostly decided by chunking and ranking. Chunk too coarsely and you bury the relevant sentence in noise; chunk too finely and you lose the context that made the sentence meaningful. Then there is evaluation, which teams skip and later regret: without a labelled set and a harness measuring whether you retrieved the right context, you are tuning blind and every regression is a surprise in production. The ongoing cost is not training, it is keeping retrieval honest. We pull these stages apart in how to build a production RAG pipeline, which covers the chunking, reranking, and evaluation work that demos leave out.
What fine-tuning hides
Fine-tuning's cost arrives before and after the training run, rarely during it. Before: you need a clean, representative dataset, and assembling it (deciding what good looks like, labelling consistently, catching the examples that quietly teach the wrong thing) is slow, human work that no tool removes. After: the model is now a snapshot. The day you fine-tune is the day it starts going stale, because any fact you baked in is frozen at that point. Every meaningful knowledge update means another training run, another evaluation pass, and another deployment. That is the trap teams walk into when they fine-tune knowledge that should have been retrieved: they end up retraining a model to do a job an index would have done for free.
This is also where the cost comparison gets less obvious than it first looks. RAG is usually cheaper to start because it avoids the training run and lets you update by reindexing. But at high request volume, a fine-tune can lower per-request cost by shortening prompts, since behaviour you would otherwise spell out in the prompt is now in the weights. So the honest answer to which is cheaper is: it depends on your scale, and you should run the numbers for your own traffic rather than trust a general claim.
A decision summary
If you remember nothing else, sort your problem into knowledge versus behaviour first, then let cost and maintenance settle the rest.
How to decide
- Knowledge that changes or needs citing goes in RAG. Start here by default.
- Behaviour that should stay constant (format, tone, repeated instructions) is a candidate for fine-tuning.
- Combine them for production: fine-tune behaviour, retrieve knowledge, and keep the two separate.
- Never fine-tune facts that move. You will pay for it in retraining.
- Decide cost on your own request volume, not on a rule of thumb.
Most teams we work with land on RAG first, add a light fine-tune only once the behaviour is genuinely stable and the volume justifies it, and keep knowledge in retrieval throughout. If you want a second opinion on where your own system sits, that is the kind of question we help with on AI agents and LLM product engineering. The right answer is whichever one you can still maintain in a year.