Fine-tuning, RAG, or prompt engineering: which one does your problem actually need?
Teams routinely reach for fine-tuning when the real problem is retrieval, and for retrieval when the real problem is a vague prompt. Diagnosing correctly saves months, because these three options differ in cost by roughly two orders of magnitude.
By Quality AboveAll · · 8 min read
- Wrong facts is a retrieval problem. Wrong format or tone is a fine-tuning problem. Inconsistent reasoning is usually a prompt problem.
- Always exhaust prompt engineering first: it is the only option with effectively zero marginal cost and no maintenance burden.
- Fine-tuning locks knowledge in at training time, so it is the wrong tool for anything that changes weekly.
Diagnose before you choose
Start by characterising the failure precisely. If the model states things about your business that are untrue or out of date, it lacks facts, and no amount of behavioural training will supply them. If the model has the facts right but formats them inconsistently or writes in the wrong register, it lacks a behavioural pattern. If it wanders, contradicts itself, or ignores half the instruction, the prompt is underspecified.
Writing that diagnosis down before choosing a technique is the single highest-return habit in AI engineering, because each technique is nearly useless against the other two failure modes.
Prompt engineering: start here, always
A well-structured prompt with explicit role, constraints, output schema and a few worked examples resolves a surprising share of problems, and it costs a developer an afternoon rather than a training budget. Few-shot examples in particular do much of what people expect fine-tuning to do, at zero training cost.
The published prompt engineering guidance from model vendors is genuinely worth reading before assuming you need something heavier. Our own practice notes are in prompt engineering for production systems.
Most teams who tell us they need a fine-tuned model have not yet written a prompt with a defined output schema.
RAG: when the problem is knowledge
Retrieval is the answer whenever the required knowledge is specific to your organisation, changes over time, needs to be cited, or must respect per-user permissions. Those four conditions cover the large majority of business AI features, which is why RAG became the default architecture.
Its ongoing cost is an ingestion pipeline and retrieval quality monitoring rather than periodic retraining. That is real work, but it is work that keeps paying off as your content changes, whereas a fine-tune decays from the day it finishes.
Fine-tuning: when the problem is behaviour
Fine-tuning genuinely wins when you need a consistent output structure across thousands of calls, a specialised classification task where a smaller tuned model beats a larger general one on both accuracy and cost, or a domain vocabulary the base model handles clumsily.
The costs are honest and worth stating: a curated training set, a training run, evaluation to prove it improved anything, and a repeat of all three every time the base model you built on is superseded. Committing to that cycle is a product decision, not an experiment.
The combination most production systems land on
In practice mature systems use a strong prompt for structure, retrieval for facts, and occasionally a small fine-tune for a narrow, high-volume task where format consistency directly affects downstream automation. They are complementary layers, not competing options.
Sequence matters for cost control: prompt first, measure, add retrieval, measure again, and only consider fine-tuning when you have evidence that a behavioural gap persists after the first two. Our note on LLM evaluation covers how to measure well enough that these decisions rest on data.
Frequently asked questions
Is fine-tuning obsolete now that models have long context windows?
No, but its niche has narrowed. Long context reduced the need to train knowledge into a model, while fine-tuning remains valuable for consistent output structure and for smaller specialised models that are cheaper to run at volume.
How much data do I need to fine-tune?
Far more than most teams expect for a quality improvement, and the quality of the examples matters more than the count. If you cannot assemble a clean, consistent dataset, fine-tuning will amplify the inconsistency rather than fix it.
Can I use RAG and fine-tuning together?
Yes, and it is common. Retrieval supplies current facts while the fine-tune enforces the house output format. They address different layers of the problem.
Unsure which of the three your problem actually needs? Describe the failure to us in a free 30-minute consultation and we will tell you which lever to pull, including when the answer is the cheap one.
