1. What an embedding model does
An embedding model turns a text into a list of numbers, called a vector, built so that texts with similar meaning have nearby vectors. In a knowledge base this makes it possible to find a passage even when the question uses different words from the document. Someone asking how to request holiday can find the paragraph that discusses leave and absences, without using the same word, and this is the practical value of the technique.
An embedding model does not answer questions or write text. Its only task is to make passages and questions comparable. For this reason its quality directly affects which documents reach the generating model, and therefore the final answer. Choosing badly means retrieving the wrong passages even when the rest of the system is well built, and the fault is hard to notice because the answer still reads fluently.
2. Criteria that really matter
The selection criteria are more numerous than they seem. The first is quality on the real Italian in your documents, which includes internal jargon, acronyms and technical terminology. A model evaluated mainly on English texts can perform worse on Italian company documents, even if it scores well on generic tests. For this reason the figure published by the manufacturer should always be reread in light of your own texts.
The other criteria concern where the model runs, how much it costs to keep the index up to date, how much space the vectors take in storage, how easy it is to replace the model in future and what guarantees the supplier gives on the handling of texts. These criteria are at least as non-technical as the technical ones, because they affect confidentiality and service continuity. None of these points should be assessed in the abstract, but always on your own case.
3. Three families of options compared
The options fall into families. The names of individual models change quickly, while the characteristics of the families remain more stable over time. Always verify the characteristics of the specific version the supplier proposes, not the one shown in a presentation. Ask for the technical sheet, the release date and the test results published by the model's developer, and compare them with what is being offered to you.
No single family is best in absolute terms. A prudent choice starts from the perimeter of the data: if the documents cannot leave the company's infrastructure, externally hosted options should be excluded or constrained by written guarantees. If instead the data can be processed by a supplier under appropriate agreements, ease of management weighs more heavily in the final decision. Also ask which Italian-language examples are included in the published tests.
| Family | Where it runs | Main strength | Watch out for |
|---|---|---|---|
| General model hosted by the supplier | On the supplier's service, reached through a programming interface | Easy to start and to maintain | Who processes the texts and where the data is located must be verified in writing |
| Open-source model run locally | On the company's infrastructure or a dedicated supplier | The text does not leave the chosen perimeter | Requires computing capacity and skills to keep it updated |
| Model specialised for Italian or for a sector | At the supplier or the company, depending on the architecture | Can perform better on technical vocabulary | Less public documentation and harder to compare |
| General-purpose multilingual model | As above, depending on the architecture | Handles several languages in the same archive | Quality on Italian must be measured, not assumed |
4. Hidden trade-offs
The most underestimated cost is not the cost of starting but the cost of changing. If tomorrow you want to replace the model, you must recalculate the representation of all documents, because vectors produced by different models are not comparable with one another. On large archives this takes time, planning and a verification window. Ask the supplier what a change of model costs in operational terms and whether the service stays available during the transition.
Another trade-off concerns the length of text the model accepts. A model with a wider window can read longer passages in one go, but it does not follow that it reads them better, and quality sometimes falls precisely on longer texts. Check the behaviour on real documents, because greater capacity does not guarantee more precise search.
5. How to decide with a test on your documents
The most reliable way to choose is a test. Take a representative group of documents and a set of real questions, with the expected answer and the document that contains it. Compare the candidate options by measuring how often the right document appears among the top search results. Use the same questions for every option, otherwise the comparison means nothing and ends up rewarding the most persuasive presentation.
Add the difficult questions to the test: internal acronyms, terms with several meanings, colloquial phrasings. These are the ones that reveal differences between models. Also note the response time and the estimated cost of management, without trusting generic list-price figures. The winner is often the one that makes fewer mistakes on the awkward questions, not the one with the highest ranking.
6. Questions to put to the supplier
The answers to these questions are worth more than a sales presentation. If the supplier does not know them or does not put them in writing, treat this as a signal to probe further before signing. Also ask whether you can repeat the test with your documents during the evaluation phase, and to see the result for individual questions, not just the average.
- On which Italian-language documents was the model evaluated, and with what criteria?
- Where are the texts processed and who can access them, including for legal obligations?
- How is a change of model handled without interrupting the service?
- Are the texts sent used to train third-party models? The answer must be given in writing.
- Which logs are kept, for how long, and who can consult them?
7. How to get started
This week, bring together a group of real questions, chosen from those colleagues ask most often, and identify the documents that answer them. Without this material no comparison between models is reliable, and with it the discussion with the supplier becomes concrete and verifiable by both sides. Use questions you have actually seen asked, without rewriting them in a more elegant form, because your way of asking is what the system will have to handle every day.
Once the test is ready, ask the supplier to run it on the options you intend to evaluate, letting the results speak for themselves. If the differences are small, choose the option that is simplest to manage and to replace, because in future it will be the one that costs less to change. If the difference between two options is minimal, the deciding factor becomes ease of management and replacement over time, not the score of a single test.
Frequently Asked Questions
Must I choose a final model straight away?
No, but you need to know how to change it. Start with a reasonable option for your data perimeter and check that the system allows it to be replaced without rewriting everything. An architecture that separates the model from the rest of the system reduces the risk of being tied to an initial choice that later stops working.
Is a larger model always better?
Not necessarily. Size and quality do not always coincide, especially on specific vocabulary such as company jargon. A smaller model, evaluated on your documents, can outperform a larger one in search. The criterion is the result measured on your archive, not the declared size.
Can we change model after a year?
It can be done, but it has an operational cost. All documents must be reprocessed with the new model and the results rechecked. Plan the change as an ordinary operation, with a check before and after and a written rollback plan to the previous version.
Must the model also handle English?
It depends on the documents. If the archive contains texts in several languages, or English technical terms, check that the model handles them well. A test with Italian questions on English documents, and vice versa, shows at once whether coverage is adequate.
Are public model leaderboards enough?
They are a starting point, not a decision. Public tests use texts and questions that can be very different from your documents. Use them to narrow the options, then decide with a test built on your archive and your questions.