RAG vs fine-tuning: how to choose for your LLM application

Use RAG (retrieval augmented generation) when the model lacks knowledge: your documents, policies, products or records, especially when they change. Use fine-tuning when the model lacks a behaviour: a format, tone or narrow task it gets wrong even with the right context. Start with prompting and RAG, measure, and fine-tune only when an evaluation set shows a gap they can't close.

#What is RAG?

Retrieval augmented generation is a pattern, not a product. When a question arrives, the application searches your content for relevant passages, adds them to the prompt, and asks the model to answer from them. The term comes from a 2020 paper by Lewis and colleagues, which combined a pretrained language model with a searchable index of documents so that knowledge could be updated without retraining.

In practice, a RAG system has an ingestion pipeline (connectors, cleaning, chunking, embeddings), a retriever (keyword search, vector search or both, often followed by a reranker) and a generation step with instructions to cite sources and admit when the sources are silent. The model's weights never change.

#What is fine-tuning?

Fine-tuning continues training a pretrained model on your own examples, usually pairs of inputs and the outputs you want. The result is a model that behaves differently by default: it follows your format without long instructions, uses your terminology, or performs one narrow task more reliably.

Full fine-tuning updates every weight and needs serious hardware. Parameter-efficient methods such as LoRA (Hu et al., 2021) train small adapter matrices instead, which makes fine-tuning open-weight models practical on modest hardware. Hosted providers also offer fine-tuning of some of their models through an API, as described in OpenAI's fine-tuning guide. Libraries such as Hugging Face PEFT implement LoRA and related methods.

#RAG vs fine-tuning at a glance

#When RAG is the right choice

• The answer lives in documents you own. Support articles, policies, contracts, product specs, tickets, CRM notes.
• The knowledge changes. Weekly price lists, new tickets, revised policies. Re-indexing a document takes minutes; retraining takes a project.
• People need to check the answer. Citations let a user click through to the source, which is how trust is earned in support, legal and internal tools.
• Different users may see different things. Retrieval can filter passages by the asker's permissions before the model ever sees them.

One caveat before building a pipeline: if all of your knowledge is small, you may not need retrieval at all. Anthropic's contextual retrieval write-up suggests that a knowledge base under about 200,000 tokens (roughly 500 pages) can simply go into the prompt, with prompt caching keeping cost and latency down.

#When fine-tuning is the right choice

• A strict output format that long instructions still miss, such as a particular JSON schema or report layout.
• A narrow, repeated task (classifying tickets into your categories, extracting fields from your documents) where a smaller tuned model can match a larger general one.
• Tone and terminology that must be consistent across thousands of outputs.
• Cost or latency at volume. Shorter prompts and a smaller model can make a high-volume feature affordable.
• Private, offline deployment. A tuned open-weight model you host yourself, when data can't leave your environment.

Fine-tuning is a poor way to teach a model facts that change. It can also overfit: a model trained on a narrow set of examples may get worse on everything else. That is why every fine-tune needs a before-and-after evaluation on cases outside the training data.

#Can you use RAG and fine-tuning together?

Yes, and mature systems often do. Retrieval supplies current facts; a lightly fine-tuned model turns them into the right shape, such as a support reply in your house style that always cites its sources. Fine-tuning can also improve the retrieval side: embedding models and rerankers can be tuned on your own question and passage pairs.

The order matters. Get retrieval right first, because a fine-tuned model fed the wrong passages will produce well-formatted wrong answers.

#How to decide: a five-step test

1. Write down 50 to 200 real questions or tasks with known good answers. Use real tickets, emails or documents, not invented examples.
2. Try a strong model with a good prompt and, if knowledge is needed, the relevant documents pasted in by hand. This is your ceiling for RAG.
3. If answers fail for lack of knowledge, build retrieval and measure whether the right passages are found (retrieval hit rate) before measuring answers.
4. If answers fail on format or behaviour with the right context, that is the signal for fine-tuning. Train on a few hundred to a few thousand curated examples and compare on the same evaluation set.
5. Keep the evaluation set in CI. Every change to prompts, retrieval or models is scored against it, so regressions are caught before users see them.

We cover building and running that evaluation set in AI features that survive production.

#Common mistakes

• Fine-tuning to add knowledge. The model learns the style of your documents, not a reliable index of their facts.
• Blaming the model for retrieval failures. When the right document exists but the answer is wrong, check whether the passage was retrieved and how it ranked before rewriting prompts.
• Vector search only. Product codes, error messages and names often need keyword search too. Hybrid search with a reranker usually beats either alone; Anthropic reported that combining contextual embeddings with contextual BM25 cut failed retrievals by 49% in its tests, and by 67% with reranking added.
• No access control. An assistant that retrieves everything for everyone becomes a data leak with a friendly interface.
• No measurement. Without an evaluation set, "it feels better" decides releases.

#What it means for cost

RAG costs more per call, because retrieved passages lengthen every prompt, and it needs infrastructure to ingest and search content. Fine-tuning costs more upfront (data curation, training runs, evaluation) and again every time you retrain, but can lower per-call cost by allowing shorter prompts or a smaller model. For most business applications the deciding cost is neither: it is the engineering time to make answers trustworthy, which is why measurement comes first.

#Frequently asked questions

If you want a retrieval system built and measured on your own documents, see RAG development. If the question is which model to use, or whether fine-tuning will pay off, start with LLM development and fine tuning, or hire LLM engineers to work inside your team.