TECHNICAL GUIDE · GLOBAL ARCHITECTURE

Air-Gapped RAG Pipelines: Ingestion, Chunking, and Vector Search for Private Data

Building completely isolated Retrieval-Augmented Generation workflows without leaking intellectual property to public LLMs.

Updated october 2026
8 min read
Altovar Technical Editorial Team

1. Why private retrieval needs its own design

Retrieval-augmented generation (RAG) lets a language model answer questions using your own documents. The system finds the passages most relevant to a question and places them in the prompt for the model. This is useful for contracts, technical manuals and internal procedures. The risk is that every stage of the pipeline handles sensitive content: parsing, the storage of embeddings, the prompt sent to the model and the logs that record what happened.

The term air-gapped is often used loosely. A system with no connection to the internet is one extreme. A system that uses a privately deployed model under strict contractual data terms is another, and many practical designs sit between the two. Define what isolation means for your organization: which networks can be reached, which components may send data outside the perimeter and who can access the logs. This definition drives every later decision in the design.

2. Ingestion and chunking stages

Ingestion turns files into text. Scanned documents need optical character recognition, office files need parsing, and tables or headers often lose their meaning in plain-text extraction. Validate a sample of the extracted text by hand before indexing, and flag the document types that fail. Keep a record of the source file, its version and the access rights for each document, because retrieval must respect those rights later.

Chunking splits documents into passages that can be retrieved and placed in a prompt. Chunks that are too large dilute relevance, while chunks that are too small lose context. Preserve section titles and document identifiers inside each chunk, so that every answer can cite the source passage it relies on and a reviewer can check it. Test the chunk size on real questions.

3. Comparing vector search options

Vector search finds passages whose meaning is close to the question, which is why it complements keyword search. The storage choice affects how permissions are enforced, how backups work and how the system scales. A dedicated vector database is built for large collections and for filtering. A vector extension in an existing relational database reuses the controls and skills the team already has. A hybrid search engine combines keywords and vectors, which helps with exact identifiers such as article codes or names.

The table compares the options at a general level. Choose on the basis of your access control model, your operational skills and the size of the collection, and then test retrieval quality on real questions. Include questions from the people who will use the system every day. Benchmarks published by others rarely reflect your documents, your languages or your permission rules.

OptionStrengthTrade-offBest fit
Dedicated vector database such as Milvus or QdrantBuilt for large vector collections and filteringAnother stateful service to operate and back upLarge document sets with many users
Vector extension in an existing relational databaseReuses current backup, access control and skillsPerformance may lag at very large scaleModerate collections with an existing database estate
Hybrid keyword and vector search engineStrong for exact terms, codes and namesMore tuning of ranking is neededTechnical and legal text with identifiers
Simple file-based index for a small pilotQuick to buildNot suited to growth, access control or frequent updatesProof of concept only

4. Model access, prompts and data leakage

The model is the component most likely to send data outside the perimeter. Decide whether it runs inside your environment or under a contract with a provider. In the second case, confirm what the provider retains, whether prompts may be used for training and where processing takes place. Data location and legal access are separate questions: a system hosted in one country may still be subject to legal demands from another, depending on the provider's ownership and structure.

Control what goes into the prompt. Retrieve only the passages that the user is allowed to see, and filter by identity before the model receives anything. Do not rely on the model to refuse disclosure, because a refusal is not an access control. The safety of the system comes from the retrieval layer, which must be tested with the same care as any other application that handles confidential data.

5. Checklist for a private RAG pipeline

The checklist below turns the design into testable requirements. It starts with the isolation boundary and the document classification, then covers access control, extraction quality, logging and testing. Each item should have an owner who can show the evidence, such as a configuration export, a test result or a signed data flow. Keep each piece of evidence with the document it covers.

Review the checklist with the data owners and with the security team, not only with the engineers who build the pipeline. Business owners know which documents are sensitive and which users should never see them, and that knowledge is often missing from the technical design. Ask them to confirm the classification of each collection before the first load, and record their answer.

  • Define the isolation boundary: which networks, components and data flows may leave the perimeter.
  • Classify documents and record the owner, the version and the permitted users for each collection.
  • Enforce access control at retrieval time, using the identity of the user who asks the question.
  • Validate extraction on a sample before indexing, and flag tables and scanned pages.
  • Log queries and retrieved passages with a defined retention period and restricted access.
  • Test with adversarial questions that try to obtain documents the user should not see.
  • Plan backups and re-indexing for when documents change or are deleted.

6. Common mistakes and what to ask your provider

A frequent mistake is loading everything into one collection with no permission model, and then adding filters in the user interface. Filters in the interface are not access control, because anyone who can query the system directly can bypass them. Another mistake is deleting a source document while leaving its chunks and embeddings in the index. This keeps deleted information retrievable long after the business believes it is gone.

Ask the provider or internal team where embeddings are stored, how deletions reach the indexes and backups, and who can read the logs. Ask how model updates are tested before they change answers in production. Ask how the system behaves when retrieval finds nothing relevant, and whether it says so clearly rather than inventing an answer. Embeddings and logs derived from personal data can themselves be personal data under GDPR, so they need the same classification and retention rules as the sources.

7. How to start with one document collection

Choose one document collection with clear owners, moderate sensitivity and a real question that people ask every day. A focused pilot is more useful than a general AI platform project without an owner. Define the isolation boundary in writing and agree on who approves each data flow before any document is loaded. Use a short, dated decision record for each approval.

Build a small pipeline: ingest the collection, chunk it, index it with access control and evaluate the answers against questions whose correct sources are known. The first concrete step is to write the list of document types, their owners and the users permitted to see each one, and then to ask the team which documents must never leave the environment under any circumstances.

Frequently Asked Questions

Does a private RAG system need a self-hosted model?

Not always. The requirement depends on the sensitivity of your data and on the terms you can secure from a model provider. A self-hosted model gives the most control over where processing happens, but it also needs hardware, updates and operational skills. Compare the options on data handling and on operating effort, not only on cost.

Is RAG enough to keep answers accurate?

It helps but does not guarantee accuracy. Answers can still be wrong when retrieval finds the wrong passage or when the model combines sources poorly. Measure retrieval quality with a set of test questions, and require every answer to cite the passages it used so that a person can verify it.

How do we handle documents that change or are deleted?

Track versions, re-index changed documents and remove deleted ones from the index according to your retention policy. Confirm that the removal also reaches backups and any caches. Without this discipline, users may receive outdated content or content that the business has already withdrawn.

Does GDPR apply to embeddings?

It can. Embeddings and logs derived from personal data may themselves be personal data. Treat them with the same classification, retention and access rules as the source documents, and document the legal basis with your advisor before the pipeline processes real records.

What is the first risk to address?

Access control at retrieval. A system that can show a user passages they are not entitled to see is a data breach, however isolated the model may be. Test this with deliberately difficult questions before any pilot user receives access, and fix any gap before expanding the collection.

About this content

General informational guide. Regulations and technical requirements change over time: always check the official sources and assess your specific case with a qualified technical or legal advisor.

GET IN TOUCH

Talk to our team.

We'll show how Altovar integrates into your infrastructure. No commitment, no sales pitch. Just a technical conversation.

BOOK A DEMO →