What is RAG (retrieval-augmented generation), and when does your product need it?
An LLM knows what was in its training data and nothing about your business. RAG is the pattern that closes that gap: retrieve the relevant facts from your own systems first, then ask the model to answer using only those facts.
By Quality AboveAll · · 8 min read
- RAG retrieves relevant documents from your own data, then passes them to the model as context, so answers are grounded in your facts rather than the model's memory.
- Most RAG failures are retrieval failures, not model failures. If the right chunk never reaches the prompt, no model can save the answer.
- RAG beats fine-tuning when knowledge changes often, needs citations, or must respect per-user permissions.
What RAG actually does
Retrieval-augmented generation splits answering into two jobs. First a retrieval step searches your own content, a document store, a database, a knowledge base, and pulls back the passages most likely to contain the answer. Then a generation step hands those passages to the language model along with the user's question and asks it to answer using that material.
The idea was formalised in a 2020 research paper, and it has become the default architecture for AI features that need to be right about a specific business rather than fluent in general. Instead of hoping the model memorised your refund policy, you show it your refund policy at the moment of the question.
Why teams reach for it
The practical driver is that facts change. Pricing updates, policies get revised, new products launch. A model whose knowledge is baked in at training time is stale the moment anything moves, whereas a retrieval layer reads whatever your systems say today.
The second driver is traceability. Because you know which passages were retrieved, you can cite them in the answer. That turns an unverifiable claim into something a user can check, which matters enormously in regulated contexts and is the difference between an AI feature people trust and one they quietly stop using.
If your AI feature cannot show its source, your users are being asked to take its word for it. Most of them will not, and the ones who do are the ones you should worry about.
The part that decides whether it works
Almost every disappointing RAG system we are asked to fix has the same root cause: retrieval is returning the wrong passages, and no amount of prompt tuning fixes a prompt that does not contain the answer. Chunking strategy, how you split documents, is usually the first culprit. Split too small and you sever the context that made a passage meaningful; split too large and the genuinely relevant sentence gets diluted among paragraphs of noise.
Pure vector similarity is the second. Vector search finds semantically similar text, which is excellent for questions phrased differently from the source but weak on exact identifiers, product codes, and names. Hybrid retrieval, combining keyword and vector search and then re-ranking the merged results, consistently outperforms either alone. We cover the storage side of this in our guide to vector databases and the maths behind it in embeddings explained.
RAG, fine-tuning, or both
These solve different problems and the confusion between them wastes a lot of budget. RAG changes what the model knows. Fine-tuning changes how the model behaves: tone, output format, adherence to a specific structure. If your complaint is "it gets our facts wrong", that is retrieval. If your complaint is "it will not stop writing like a chatbot", that is fine-tuning.
Plenty of mature systems use both, with retrieval supplying the facts and a light fine-tune enforcing the house format. Our comparison of fine-tuning versus RAG versus prompt engineering works through where each one earns its cost.
What a production RAG system needs beyond the demo
A demo needs a vector store and a prompt. A production system needs permission filtering so retrieval never surfaces a document the asking user is not entitled to see, an ingestion pipeline that keeps the index current as source documents change, and evaluation that measures retrieval quality separately from answer quality so you can tell which half broke.
It also needs a defined behaviour for the case where retrieval finds nothing useful. Systems that fall back to the model's general knowledge in that situation produce exactly the confident, unsourced, wrong answers RAG was adopted to prevent. Saying "I do not have that information" is a feature. See AI guardrails for how we enforce it, and securing your RAG pipeline for the access-control side.
Frequently asked questions
Is RAG better than fine-tuning?
They solve different problems. RAG supplies current, citable facts and is the right answer when knowledge changes or needs sourcing. Fine-tuning shapes behaviour, tone and output format. Teams that need both facts and a specific voice often use them together.
How much data do you need for RAG to be useful?
Far less than fine-tuning requires. RAG works with whatever documents you already have, and it is useful from a few dozen documents upward, because it is searching rather than learning statistical patterns.
Why does my RAG system still give wrong answers?
In our experience the overwhelming majority of the time the correct passage was never retrieved, so the model was answering without it. Measure retrieval quality on its own before touching the prompt or swapping models.
Building an AI feature that has to be right about your own business? A free 30-minute consultation will get you an honest read on whether RAG is the right architecture and where the retrieval risk sits.
