GUIDE · LLM ON PREM

Monitoring an LLM in production: what to observe

A large language model can be reachable and return wrong answers at the same time. This guide explains what to observe in production, how to set useful alerts without being overwhelmed, and who must respond when something goes wrong. By the end you will have a concrete list of checks to switch on over the coming weeks.

Updated october 2026
7 min read
Altovar Technical Editorial Team

1. Why monitoring an LLM is different

In a traditional web service, the most important check is whether the page responds. For an LLM, that is not enough. The system can be reachable, with regular response times, and still produce invented, incomplete or irrelevant answers. Monitoring that covers only availability sees half of the problem, and the other half emerges when a user reports that the system is no longer reliable.

Monitoring must therefore cover three layers: the infrastructure, meaning servers and GPUs; the service, meaning requests, response times and errors; and quality, meaning whether answers remain adequate over time. Each layer requires different tools and owners, and none of them replaces the others. Anyone who watches only the first layer discovers problems when someone complains.

2. Prerequisites: what must be ready

Before building the monitoring, you need to know what you are observing. You need a list of the service components: the model and its version, the system that receives requests, document retrieval if present, access controls and log storage. Without this list, metrics end up covering only what is easiest to measure.

You also need a clear rule on what gets recorded. Logs can contain questions and answers with personal data, so define which fields to keep, how long to retain them and who may consult them. Agree these rules with the data protection officer (DPO) before switching on collection, because monitoring must be designed together with data protection, not added afterwards.

3. The metrics to observe

Metrics fall into a few main families. The table lists the most useful ones to start with, without trying to cover them all. For each one, define a reference threshold chosen together with the people who use the service, not copied from a generic value. A threshold without a person who approves it soon becomes background noise. Also define who decides to change it, and on what grounds.

Choose a few metrics and read them consistently. A dashboard with dozens of charts does not help someone who must decide quickly. Better a handful of indicators that everyone knows how to interpret, with a note on what to do when they change. Each indicator should have an owner who checks it at a defined frequency. If nobody looks at it, remove it from the dashboard. The owner is also the person who proposes a new indicator when one is needed.

AreaWhat to measureWhy it matters
AvailabilityHow long the service responds correctlyShows whether users can work
Response timesWait before the first word and total timeReflects the perceived experience, especially at peaks
ErrorsFailed requests, timeouts and empty responsesSignals application or capacity faults
Load and queuesWaiting requests and concurrency over timeAnticipates saturation before response times blow up
Physical resourcesGPU memory, temperature and compute usageLinks the service to the underlying infrastructure
QualityUser reports and results of periodic testsShows whether answers remain adequate

4. Setting up alerts

An alert is only useful if someone knows what to do when it fires. For this reason every alert must have an explained threshold, a recipient and an expected action. Alerts that do not lead to an action should be removed or turned into indicators to read periodically, otherwise they create the habit of ignoring them. An alert that fires every night for no reason will be ignored even when it is important.

Distinguish between urgent alerts, which require immediate action, and signals to review within a short time. The following rules are a starting point to adapt. Revise them after the first weeks of real use, when you have seen which ones fire by mistake and which are missing. Record each change, with date and reason, so the team knows why a threshold changed.

  1. Alert the on-call engineer immediately when the service does not respond or returns only errors continuously.
  2. Alert when response times exceed the agreed threshold for longer than a normal peak.
  3. Alert when GPU memory or temperature approaches the limits set by the manufacturer.
  4. Alert when queued requests grow steadily, before the queue becomes visible to users.
  5. Notify the service owner when the share of answers judged inadequate by users changes.
  6. Record every model version change as an event, so it can be linked to any changes in the data.
  7. Periodically check that each alert is still useful and that its recipient is still in post.

5. Measuring quality, not just availability

Quality is the hardest part to monitor, because it cannot be measured with a simple counter. Two approaches are practical: collect user reports, with a simple way to flag a wrong answer, and periodically rerun the regression test on a fixed set of questions. The second is more reliable, because it always compares the same conditions. Read the result in comparison with the previous one, never on its own.

Run the periodic test even when the system seems to be working well. A silent change to a dependency, an update to document retrieval or a configuration change can alter answers without generating any error. Record the results over time and ask someone who knows the domain to review them. If results worsen without a declared change, look for the cause in the service's dependencies.

6. Common mistakes and how to verify

A frequent mistake is to measure only the infrastructure and consider the service healthy because the servers are on. Another is to have many alerts and no owner, or owners who do not know what to do. A third is to collect logs containing personal data without a retention rule, discovering the problem only during an audit. Ask the data protection officer to review the rule before switching on collection.

To verify that monitoring works, trigger a controlled failure in a test environment: stop a component or simulate a slowdown. Check that the alert reaches the right person, that the person knows what to do, and that the diagnosis is possible with the data collected. If any of these three things is missing, the monitoring is not ready yet. Record the outcome in one line, with date and owner.

7. How to start

This week, write the list of service components and assign each one a monitoring owner. Then choose five indicators from the table and define for each who checks it and how often. This step takes a few hours and immediately improves how the service is managed. You do not need to wait for a perfect tool to start. A simple first version, used every day, is worth more than a perfect project still waiting to be built.

The following day, ask your provider what data they already collect, where they store it, for how long and who has access. The answers say a lot about the maturity of the offering and help you see which indicators are available straight away and which must be built. Also note anything that is unclear, since it will be the first point to close. A provider unable to answer these points should be assessed with more care.

Frequently Asked Questions

Is a service that responds a healthy service?

Not necessarily. A system can respond on time and still produce wrong or incomplete answers, without any counter noticing. Availability is a necessary check but not a sufficient one. Alongside the technical metrics, add a periodic quality check, with a fixed set of questions and answers judged correct.

How many metrics should I monitor?

Few, and well chosen. A small number of indicators read regularly is more useful than a dashboard full of charts nobody looks at. Start with the essential areas: availability, response times, errors, load and quality. Add more only when you have a specific question they answer.

Can I use the monitoring tools already in place in my company?

Often yes, for infrastructure and availability. Model-specific indicators, such as time to first word or answer quality, may require dedicated collection. Check with your provider what is already available and what must be built, and do not assume a generic tool covers everything.

Should conversation logs be kept?

It depends on the purpose and on the rule you agree with the data protection officer. To diagnose problems it may be useful to keep part of the data for a limited period, but not necessarily the full text of conversations. Define which fields to collect, who can consult them and when they are deleted.

Who should receive the alerts?

The on-call technical engineer for faults, and the service owner for quality or usage problems. Anyone who receives an alert must know what to do and to whom to escalate if they cannot resolve it. Alerts sent to several people without a clear owner tend not to be handled by anyone.

About this content

General informational guide. Regulations and technical requirements change over time: always check the official sources and assess your specific case with a qualified technical or legal advisor.

GET IN TOUCH

Talk to our team.

We'll show how Altovar integrates into your infrastructure. No commitment, no sales pitch. Just a technical conversation.

BOOK A DEMO →