RAG vs fine-tuning
Fine-tuning on your policies makes a model sound sure about policies that have already changed.
The short answer
RAG gives a model access to your documents when it answers. Fine-tuning changes how the model behaves. One brings in knowledge. The other shapes behaviour. For most enterprise agents, retrieval comes first, because the knowledge changes and the permissions must hold.
Decision table
| Question | RAG | Fine-tuning |
|---|---|---|
| What it changes | What the model can see when it answers | How the model responds by default |
| Knowledge freshness | Updated when the source is updated | Fixed until the model is trained again |
| Access control | Can filter results by the requester's rights | Harder to restrict, since the knowledge is in the weights |
| Citation | Can point to the source passage | Cannot point to a source by itself |
| Typical use | Policies, product data, case files, records | Output format, tone, classification, narrow tasks |
| Main risk | Poor retrieval returns the wrong passage | Training data drifts and the behaviour changes after an update |
Use cases for retrieval
- A staff policy assistant. Policies change during the year. Retrieval answers from the current version and shows the section it used.
- Product support over a changing catalogue. Retrieval reads the live catalogue, so a weekly change does not need a retrained model.
- Case-file search where users see different records. Permission filters run before the search, so each user sees only their own cases.
Use cases for fine-tuning
- Turning free-text tickets into a fixed label set. A narrow classification, with thousands of labelled examples and a label list that does not change.
- A strict output format for a downstream system. The format must not vary, and prompting alone keeps breaking it.
- A house writing style in one narrow content type. The style is stable and there are many examples of it.
Trade-offs
Retrieval
Advantages. Knowledge stays current when the source changes. Answers can point to the passage they came from. Access rules can be applied to what is retrieved.
Disadvantages. The answer is only as good as the passage retrieved, so retrieval needs its own testing. There are more parts to run, and each one can fail.
Fine-tuning
Advantages. Output format and style can be held consistently across many requests. Once trained, the behaviour does not depend on a long set of instructions in every call.
Disadvantages. Facts go stale, and changing them means retraining. It is hard to restrict what the model knows once it is in the weights. Each retrain can change behaviour that was working.
How each option runs
Retrieval, step by step
- Step 1. Index the source documents, with their owner and their version.
- Step 2. Filter by the requester's rights before the search runs.
- Step 3. Test retrieval on questions with known answers, before the model is involved.
- Step 4. Check that the answer cites the passage it used.
Fine-tuning, step by step
- Step 1. Collect labelled examples and check they are correct, since errors are learned.
- Step 2. Split the examples into training and held-out sets, so results can be checked honestly.
- Step 3. Train, then compare the result with prompting on the same held-out set.
- Step 4. Schedule a retrain and a re-evaluation each time the target behaviour or the data changes.
What drives the effort
- How often the source changes, and how many people must see the change.
- How many labelled examples exist, and how much effort it takes to check them.
- How much of the answer depends on a document the user is allowed to see.
- How long the team will keep the evaluation set up to date.
This site does not publish prices. The engagements page describes what each shape includes.
A checklist before you choose
- Does the answer depend on a document that changes more than once a quarter?
- Do different users need different answers from the same question?
- Do you have a few thousand labelled examples that are already correct?
- Will someone keep the evaluation set current after launch?
If the first two are yes, start with retrieval. If the last two are yes and prompting has failed on the held-out set, fine-tuning is worth testing.
What people get wrong
- They fine-tune to teach facts. The facts go stale, and the model still states them confidently.
- They skip retrieval quality. Most RAG failures come from the wrong passage being retrieved, not from the model.
- They never test the choice. Both options need the same evaluation set before one is chosen.
A hybrid that often works
Use retrieval for knowledge and access control, and prompting for format. Fine-tune only when the evaluation shows that prompting and retrieval cannot reach the required behaviour. Keep the evaluation set fixed so the two options can be compared on the same cases.
Common questions
Does fine-tuning make a model more accurate?
Not on facts by itself. It can make the output more consistent. Accuracy on facts comes from the source the model reads.
Can RAG stop a user from seeing a document they should not see?
It can, if the retrieval filters results by the requester's rights before the search runs, not after. The control has to be designed in and tested.
How do I know which one is failing?
Log the retrieved passages and the output. If the right passage was not retrieved, the problem is retrieval. If it was retrieved and the answer is still wrong, the problem may be the model or its instructions.
Do I need both?
Sometimes. Many production systems use retrieval for knowledge and a fine-tuned model for a narrow step. Add the second only when the evaluation shows a gap the first cannot close.