1. Why an update is a risk
In traditional software, an update changes the code, and the result is largely predictable. In an LLM, a change can alter behaviour in less obvious ways: an answer that used to be correct no longer is, a different tone, a citation from the wrong document. Errors do not always produce error messages, which makes them harder to spot.
The problem grows over time. Users build habits around the behaviour they know, and a sudden change can be perceived as a fault even when the new model is better on average. For this reason every update must be treated as a release, with preparation, verification and communication. A change made without notice is harder to explain and harder to recover from.
2. What makes up a version
A version of the service is not just the model name. It includes the model with its precision format, the system prompt, the generation parameters, the document retrieval configuration and the filtering rules. If any of these changes, the behaviour changes. Recording only the model name means never knowing with certainty what is really running in production.
Give each version an identifier, a date, a description of the change and the name of the person who approved it. Keep the configuration file or the reference to where the weights are stored, so the version can be rebuilt. Without this data, diagnosing a problem becomes a reconstruction exercise, and comparison with the previous version is unreliable.
3. Building a regression test set
A regression test is a fixed set of questions with answers judged correct or acceptable, run every time something changes. It does not guarantee that the system is perfect, but it shows whether things that used to work no longer do. The questions must cover real use cases, including the difficult ones and those where an error would cost the most. A set that covers only easy cases says nothing about real behaviour.
Build the set with the people who know the work: collect real questions, anonymised where necessary, and have experts decide what counts as a correct answer. A set that never changes ends up measuring only what the system already does well, so update it when new cases emerge in production. Every question needs a clear reason to be in the set, and whoever proposes it must justify it. In particular, include:
- Simple, frequent questions, which must always remain correct
- Questions involving long documents or multiple sources to combine
- Cases where the system must refuse to answer or ask for clarification
- Questions on sensitive data, to check that access controls hold
- Cases that have failed in the past, to prevent them from coming back
4. The approval process
A simple process is enough, provided it is written down and followed. The crucial point is that no change goes into production without the service owner having seen and approved it in writing. The steps below describe a generic flow. Adapt it to the size of your organisation, but do not drop the comparison between the current version and the proposed one. If the process is too long for the change you are making, simplify it, but do not skip it.
Every step must leave a trace. Whoever proposes the change, whoever carries it out, whoever verifies it and whoever approves it should be separate people where possible, and the date of each decision should be kept in the project file together with the test results. This serves both to manage incidents and to answer an internal audit. A simple register, kept tidy, is worth more than a complicated system nobody updates.
- Describe the change and the reason, stating what is expected to improve and what might get worse.
- Prepare the new version in a test environment with the same configuration planned for production.
- Run the regression test on the current version and on the proposed one, and compare the results case by case.
- Analyse the differences: are the expected improvements there, and are the regressions acceptable for the service?
- Check security aspects, such as access, logs and filters, and record the outcome.
- Obtain the service owner's written approval, with version and date.
- Schedule the release in an agreed window and tell users what changes and what to do if problems arise.
- Keep the previous version available for rollback for the agreed period.
5. Gradual rollout and rollback
A gradual rollout reduces risk. You can open the new version to a small group of users, observe the behaviour for a sufficient period, and then extend access. If something goes wrong, only some users are affected and returning to the previous version is simpler to manage. If the new version shows a problem, stopping it must be immediate, without lengthy discussion.
Rollback must be tested beforehand, not improvised. Verify that the previous version can be reactivated quickly, that data produced by the new version is handled consistently, and that the team can do it without the help of whoever prepared the change. After the release, compare performance with the regression test and with user reports, to confirm that the change has produced the expected effect.
6. Common mistakes
The first mistake is to update under pressure, because a new version has been announced or because the old model is no longer supported. Every change should be assessed against its own criteria, even when the pressure is strong. The second is to evaluate the new version only with a few informal trials, which do not cover difficult cases and leave no record to compare against.
The third mistake is to change several things together: model, prompt and parameters in the same release. When something goes wrong, it is unclear what caused it. Change one thing at a time, or at least document the combination so the effects can be separated. The fourth is never updating the test set, which ends up obsolete and gives a false sense of security.
7. How to start
The first useful step this week is to give a name to the current version and note its components: model, precision format, prompt, parameters and document retrieval configuration. If any element is unknown, ask the provider and flag it as an open point. Without this snapshot, no future comparison is reliable. Once written, the snapshot should be updated at every release, not just the first time.
Then collect a first group of real questions with correct answers, and define who will decide whether a new answer is acceptable. With this tool you can assess the next change in a repeatable way, instead of relying on the impression of whoever tried it first. Also ask a colleague to review the questions and point out what is missing, because a set written by one person reflects only their point of view.
Frequently Asked Questions
Should I update the model as soon as a new version comes out?
No, not as a requirement. A newer version may be better at some tasks and worse at others. Assess the change on your own test set, with your own questions, and decide based on the result. A justified delay is a legitimate choice, provided it is documented and does not become a habit of using components that are no longer supported.
How large should the test set be?
Large enough to cover the main and critical use cases, and small enough to run every time without effort. There is no standard size. The quality of the questions and clarity about what counts as a correct answer matter more. Update the set regularly, adding the cases that emerge in production.
How do I compare two versions fairly?
Use the same questions, the same configuration conditions and the same evaluation criteria. Do not change the prompt together with the model in the comparison, otherwise you do not know what produced the difference. Record the results case by case, not just the average, because an average can hide a deterioration on an important task.
Does rollback erase the work done with the new version?
It depends on what was stored. If the new version produced data, indexed documents or logs, these must be handled explicitly. Before the release, define what happens to these items if you roll back, and verify the procedure with a trial run.
Who should approve an update?
The service owner, with input from those who know the domain and from security and data protection specialists when the change touches those areas. The provider prepares and documents the change, but approval remains an internal decision. Keep the written approval together with the test results.