RAG or fine-tuning? Which problem wants which

Full post

An internal AI project almost always starts with the same sentence: “let us teach the model our own data”. That sentence has two different technical readings, and they solve different problems. Picking the wrong one does not doom the project outright, but it makes it needlessly expensive and needlessly fragile.

What each one changes

RAG is a layer that finds the relevant document and hands it to the model before answering. It does not touch the model itself; it gives it the right context. Fine-tuning updates the model’s weights with your examples and changes how it behaves.

The distinction sharpens here: RAG changes knowledge, fine-tuning changes behaviour. “What is the company’s returns policy” is a knowledge question. “Always answer in this tone and this format” is a behaviour request.

1. How often does the knowledge change?

This one question settles most decisions on its own. Price lists, stock, policy documents, contract versions — anything that changes belongs on the RAG side. You update the document and the system answers correctly from the next question onward.

Baking the same knowledge into the model through fine-tuning means retraining on every change. For something that changes twice a week, that is an unsustainable loop.

2. Do you need to cite a source?

RAG can show which document it produced an answer from; a fine-tuned model cannot, because the knowledge is now spread across its weights. In enterprise use this is usually decisive: it is not enough for the answer to be right, it has to be verifiable.

3. Do you want format, or knowledge?

This is where fine-tuning genuinely pays: making the model’s output format, tone and command of domain language permanent. Instead of describing the format at length in every prompt, the model already knows it — which lowers both prompt length and cost.

The answer is usually “both”

Setups that work well in practice usually use both: knowledge arrives through RAG, format and tone are fixed through fine-tuning. They are not rivals but different layers.

The order matters, though. RAG comes first, gets measured, and shows where it falls short. Fine-tuning done without measurement is a cost line whose effect nobody knows.

In short

And in either case, build the evaluation set first. Without measurement, which one helped remains an opinion.

From the glossary: RAG (Retrieval-Augmented Generation) · Hallucination · Evaluation set (eval)

← All posts