GUIDE · AI FUNDAMENTALS

Fine-tuning or RAG: which to choose for your case

By the end of this guide you will be able to tell fine-tuning and RAG apart, two different ways of adapting a language model to your content. You will understand when each one makes sense, what trade-offs each involves, and which questions to put to a supplier before deciding, starting from a concrete test on your own case.

Updated october 2026
8 min read
Altovar Technical Editorial Team

1. Two different ways of adapting a model

When a generic model is not enough, there are two main routes. The first is RAG, short for retrieval-augmented generation: at the moment of the question, the system searches the company's documents for the relevant passages and supplies them to the model along with the request. The second is fine-tuning, meaning further training of the model on selected examples, which changes how it behaves.

They are different tools for different problems. RAG brings knowledge in at the moment of answering, whereas fine-tuning changes the way the model responds. Confusing the two is the most common error: many people ask for fine-tuning so that the model 'knows' the company's documents, and then discover that the information is updated poorly and cannot be cited.

2. The criteria that matter

To choose, a few questions are needed. Does the information change often? Is it necessary to cite the source of each answer? Does the problem concern content, tone, format or a repetitive task? Are there high-quality examples, well curated and in sufficient number? And who will maintain the system over time, and with what skills? The answers to the first two questions often suffice to point the choice in one direction.

Another criterion is traceability. With RAG it is possible to show which passages were used and check them. With fine-tuning, the knowledge is spread through the model and it is not easy to indicate where an answer comes from. In sectors where every statement must be verifiable, this difference matters greatly. Choosing solely on the name of the technique risks building the wrong solution from the start.

3. Comparing the approaches

The table compares four approaches qualitatively. Combining them is often necessary when both knowledge and style matter. The assessments change with the case, so use them as a starting point for discussion. Do not look for an absolute winner: each column should be read together with the others, thinking about the documents you have, how often they change and the skills available in-house.

A well-designed prompt is often the most sensible starting point. Before investing in one of the other routes, check how much can be achieved with clear instructions, examples in the request text and a set of test questions. If the result is already good, the added complexity is not justified. A badly written prompt can hide a problem that looks like missing knowledge, so it is worth ruling it out first.

ApproachWorks best whenWeak pointSkills required
Well-designed promptGeneric tasks with simple rulesLimited when specific knowledge is neededLow
RAGAnswers from internal documents that change over timeQuality depends on retrieving the right documentsMedium: data, permissions and document handling
Fine-tuningStyle, format or repetitive tasks with reliable examplesDoes not update knowledge on its own and requires curated dataMedium to high: preparing examples and evaluation
CombinationComplex cases with specific documents and styleMore components to maintain and verifyHigh: a team with mixed skills

4. When RAG is the most suitable choice

RAG is often the most suitable choice when the company has many documents that change: procedures, price lists, manuals, contract templates, answers to recurring requests. Updating a document is simpler than retraining a model, and the system can show which passage it used. This makes verification and the correction of errors easier. Always compare the alternatives on the same set of questions, otherwise the comparison says nothing.

RAG does have weak points. If the system retrieves the wrong passage, the answer will be wrong even if the model is sound. For this reason the quality of document search is a critical part of the project and must be tested with real questions. Permissions must also be respected: the system must retrieve only what the user can see in the original source.

5. When fine-tuning makes sense, and its limits

Fine-tuning is useful when you want the model to respond in a certain way consistently: a precise format, a coherent tone, a classification with your own categories, or a repetitive task that instructions cannot describe well. It requires a set of quality examples, chosen by competent people, and an evaluation before and after the intervention. Before starting, verify that a sufficient number of examples exists, because without them the work does not get going.

The most frequent limit is believing that fine-tuning is for teaching knowledge. A model trained on documents may recall sentences imprecisely, does not update information when the documents change and does not indicate its sources. Every change requires a new cycle of training and verification. For this reason it is often more sustainable to use it for style and format, leaving knowledge to RAG.

6. How to decide and what to ask the supplier

A practical path is this: start from the simplest solution, measure the result on real questions, and add complexity only if it is needed. Before deciding, clarify how often the documents change, how much citing sources matters and who will curate the examples or documents. These answers are worth more than any product comparison. The time you spend on these initial questions greatly reduces the risk of choosing a solution that is too costly to change later.

Be wary of anyone who proposes a solution before seeing your documents and your questions. A serious supplier asks about the use case, prepares a trial and shows how it measures the quality of the answers. The questions to ask are listed below. Also ask to see a real example of an answer with its cited source, not just a prepared demo. If the supplier refuses a trial on your case, that is a signal to weigh in the choice.

  • How linked documents are updated and who decides which ones are included.
  • How access permissions are respected during search.
  • How it is verified that the retrieved passage is the right one.
  • Who keeps the examples used for any training and where they are stored.
  • What happens to the data uploaded at the end of the contract.

7. How to get started

This week, choose a use case with answers based on existing documents, and prepare twenty real questions, with the correct answer indicated by a competent person. This set of questions will serve you to compare any approach, so that the choice rests on results and not on presentations. The result will be useful even if the final choice differs from what you expected.

Then try the simplest solution first, namely well-written instructions with the right examples. Record the errors and classify them: missing information, wrong information, inadequate style. Only if the errors concern knowledge should you evaluate RAG; if they concern style or format, evaluate fine-tuning. The type of error points the way. Also note how long it takes to correct each error, because it is a concrete indicator of the effort the system places on people.

Frequently Asked Questions

Is fine-tuning a way to get the model to learn my documents?

It is not the most suitable tool for this. Fine-tuning changes how the model behaves, but it does not guarantee that information will be recalled accurately, and it does not allow easy updates. To bring company documents into answers, RAG is usually the more controllable route, because the documents remain separate and can be corrected at any time.

Does RAG eliminate hallucinations?

No. It reduces the risk because the answers are anchored to a text, but the model can still misinterpret the passage, combine information incorrectly or answer a question the document does not cover. Verification with real questions and the ability to show the source used are necessary.

Can I start with RAG and move to fine-tuning later?

Yes, and it is a common path. RAG lets you understand which questions you receive and which errors appear. Only afterwards, if the problem concerns style, format or repetitive tasks, does it make sense to evaluate fine-tuning, using the examples gathered during the earlier phase. The reverse order is riskier.

What internal skills are needed?

You need someone who knows the documents and the process, someone who manages the data and permissions, and an owner who assesses the results. The supplier can build the system, but knowledge of the content stays in the company. Without this presence, the system becomes hard to maintain and correct.

With RAG, do the documents stay outside the model?

It depends on how the solution is built and what the supplier does with the data. Typically the documents are stored in a dedicated index and retrieved at the moment of the question, but this must be verified in the contract and the configuration. Ask in writing where the data is held, who has access to it and for how long it is retained.

About this content

General informational guide. Regulations and technical requirements change over time: always check the official sources and assess your specific case with a qualified technical or legal advisor.

GET IN TOUCH

Talk to our team.

We'll show how Altovar integrates into your infrastructure. No commitment, no sales pitch. Just a technical conversation.

BOOK A DEMO →