1. Why the pilot is not enough
A pilot usually demonstrates one thing only: the model gives useful answers to the questions chosen by the people who tried it. In production almost everything changes. There are more users, the questions are unpredictable, the documents contain personal data and the service must keep working even when the person in charge is on holiday. The pilot measures none of this, so its success does not prove that the service is ready.
The transition requires treating the model as a real IT system, with owners, procedures and written limits. The useful question is no longer whether the model is good, but whether you can say who runs it, who decides when to change it, what happens if it stops and how you would demonstrate that the data stays where it should. If these answers are missing, the pilot is not yet complete, even if users are enthusiastic.
2. What to decide before installing
Before touching the infrastructure, settle three decisions. The first concerns the use case: what task the system must perform, for which people and with which data. An assistant that helps administrative staff search internal documents requires different controls from a tool that handles health or customer data. The second concerns the perimeter: which users, which applications and which data sources will connect to the model, because every connection is a surface to protect.
The third decision concerns responsibility. You need an internal person who answers for the service, a supplier that manages the infrastructure, and a written rule on who approves a change of model. Without an owner, every problem ends up in a discussion between departments and nobody decides. Ask the supplier to describe in writing how activities are divided between your organisation and them, and check that the division is clear to anyone who will receive user reports.
3. The steps from pilot to service
The steps follow a precise order. Skipping one usually saves no time: it simply means you find the problem later, when fixing it costs more and users have already built habits around an unstable system. The order also matters for internal or external audits, because it shows that decisions were taken before release rather than after an incident.
Each step must produce a verifiable result. If you cannot say what the step proves, it is not yet closed and it is best to return to it before moving on. The list below is a generic sequence: adapt it to your case, but do not drop the steps covering testing, access and the rollback plan. Keep it in a shared document and update it at every release.
- Define the use case in one sentence and collect real examples of the questions the system will have to handle.
- Build a test set of genuine questions, anonymised where necessary, and note the answer judged correct.
- Choose the model and its version, and verify the terms of use in writing before using it for work.
- Size the infrastructure for the expected steady-state load, not for the number of people who took part in the pilot.
- Install the service in a test environment separate from production, with the configuration planned for production.
- Run quality and security tests and record the results in the project file.
- Open access to a limited group, with a channel for reports and a period of controlled use.
- Activate monitoring, emergency procedures and the rollback plan, then extend access to the intended user base.
4. Capacity, memory and concurrency
The capacity of an LLM server depends on three factors that influence one another: the memory occupied by the model, the length of the texts the system must handle and the number of requests arriving at the same time. A large model may not fit in the memory of a single card, a long context consumes memory for every open conversation, and several simultaneous requests slow responses down. That is why knowing which model to use is not enough: you also need to know how many users will use it together, and with which documents.
When comparing offers, ask the supplier to measure the system with your use cases rather than with examples they have chosen. A result declared without conditions cannot be compared with another. Also leave room for growth and peak workloads, and define in writing what happens when the system is saturated: requests are queued, rejected or everything slows down together.
| Factor | What it affects | Question to ask the supplier |
|---|---|---|
| Model memory | Whether the model fits into available memory, and at what precision | How much memory does the chosen model need, and in which format? |
| Context length | How much memory remains for each open conversation | How many parallel conversations can we support with documents of our size? |
| Concurrency | How many requests can run together without slowing down too much | What measurements have you made with several simultaneous users? |
| Response time | How long a user waits before seeing the first word | How is it measured, and under what load conditions? |
5. Security, access and data
The model does not know who is using it and does not apply document permissions on its own. That task belongs to the application that connects it to users. If the system retrieves documents from an internal archive, it must show only those the person can already see in the source archive. Avoid shared accounts, log who uses the service and separate roles, so that an administrator is not also the only person reading logs that contain user data.
Also decide what gets recorded. Conversations may contain personal data, and technical logs may copy it without anyone noticing. Establish how long conversations are kept, who can consult them and how they are deleted. If the processing presents high risks for individuals, the data protection impact assessment required by the GDPR must be carried out before release: check with an adviser whether and how it applies to your case.
6. Versions, testing and updates
Every change to the system alters its behaviour: a new model, a new system prompt, a generation parameter or a document added to the search archive. Treat each of these changes as a release, with a version, a date and a person who approved it. If you do not know which version gave a wrong answer, you cannot understand why it happened or roll back safely.
Regression tests are the most practical way to avoid surprises. Keep the set of test questions and the answers judged correct, run it before every change and compare the results. A test does not guarantee that the system is perfect, but it tells you whether something has got worse. Also prepare a rollback plan: the previous version must remain available so that you can return to it quickly.
7. Observability and operational responsibility
A production service must be monitored continuously. Measure availability, errors, response times, queue lengths, memory and graphics card usage, and server temperature. Add an indicator of perceived quality, for example user reports of wrong answers. Technical data tells you whether the system is alive; quality data tells you whether it is doing its job.
You also need people who respond when something goes wrong. Write a runbook, meaning a document with the steps to restart the service, return to the previous version and notify users. Define on-call cover and check with the supplier which response times are contracted, rather than assuming them. An overnight failure with no one on call becomes a problem lasting days.
8. The questions the in-depth guides answer
This guide is an overview of the cluster. The questions it contains have more detailed answers in other guides, which you can read when you reach the relevant point. You do not need them all at once: each closes a risk that, if ignored, surfaces precisely when people are already using the service. Each topic covers a concrete decision the owner will have to take sooner or later, so it is worth knowing its scope now:
- What model quantisation is, and how much precision is lost with a more compact version.
- How context length consumes graphics card memory.
- What throughput, latency and tokens per second mean, and how they are measured.
- When to choose consumer or data-centre graphics cards for business use.
- What power and cooling requirements a server with graphics cards needs.
- What to observe in production and how to set up monitoring.
- How to update models without surprises, using versioning and regression tests.
- Which licences for open-weight models allow business use.
- Which security controls to apply to an internal LLM server.
9. How to start
The first useful step this week is to write a one-page sheet for the service. State the use case, the name of the responsible person, the intended user group, the types of data the system will see and the criterion by which you will judge the answers, even if it is only a qualitative assessment made by two colleagues. If you cannot fill in a line, that is the first thing to clarify.
Then take the sheet to the supplier and ask three things: how they measure quality with your cases, how they manage model versions and what happens when the system is saturated. The answers will tell you whether the supplier works with a process or with infrastructure alone. Only then can you compare offers usefully, because you will have a common yardstick for reading them.
Frequently Asked Questions
Is a good pilot enough to go into production?
No. A pilot shows that the model can be useful on a few chosen cases. Going into production requires written owners, a repeatable test set, access control, monitoring and a plan to roll back if something goes wrong. The pilot is a learning phase: moving to a service means closing these gaps, one by one, before opening the system to new users.
Who should be the service owner?
An internal person with authority over the use case, not just over the infrastructure. The owner decides when a user may have access, approves model changes and answers for reports. The supplier manages the infrastructure and its activities should be described in writing. When the two roles are confused, nobody decides at critical moments, and that is exactly where the service stops.
Do I need a data protection impact assessment?
It depends on the processing. The GDPR requires an impact assessment for processing that presents high risks to individuals' rights. Using personal data with a system that processes it at scale, or for decisions that affect people, may fall into this category. Do not take the answer for granted: check with your data protection officer or an adviser.
Can I update the model when a better version comes out?
You can, but not automatically. A new version may improve some tasks and worsen others, and it changes the behaviour users have built their habits around. Run regression tests on your set of questions, compare the results, approve the change in writing and keep the previous version ready for rollback.
How long does it take to move from pilot to service?
There is no standard duration. It depends on the maturity of the use case, the availability of test data, the infrastructure already in place and the number of controls your organisation requires. A project with clear owners and a supplier who works by process is completed much faster than one where every decision has to be reopened. Plan in phases, not fixed dates.
Does an on-premise server eliminate the risks of an external service?
No. Keeping the model locally helps you control where the data is, but it does not by itself resolve access, configuration errors, logs containing personal data or update management. Where the system is located and who can lawfully access it are two different questions, and both should be assessed.
What is the first question to ask the supplier?
Ask how they measure the quality of the service with your use cases, not with examples they have prepared. A concrete answer, with a test set and an approval process, says a great deal about the supplier's maturity. If the answer is generic or focuses only on hardware power, probe further before signing.