GUIDE · LOCAL AI

How to install a local AI: the practical guide

Installing a local AI means running an open-weight model on your own computer or server, with no data ever leaving your network. In this guide: the hardware you actually need, the concrete steps to be up in half an hour, and the limits to know before you start.

Updated September 2026
8 min read
Altovar Technical Editorial Team

What hardware you need, by model size

The right question is not "how powerful must my PC be?" but "which model do I want to run?". Memory — RAM and especially VRAM if you have a GPU — decides everything. Rule of thumb: a Q4-quantized model occupies in memory roughly half its parameter count in gigabytes.

ModelMemory required (Q4)Runs well on
3–4 billion params (e.g. Qwen 4B)~3 GBAny recent PC, even without a GPU
7–8 billion (e.g. Llama 8B, Qwen 8B)~5–6 GBPC with 16 GB RAM or a GPU with 8 GB VRAM
13–14 billion~9–10 GB12 GB GPU or a workstation with 32 GB RAM
30+ billion~20 GB and upDedicated GPUs (24 GB+) or a server

For office use, the 7–8 billion range is the sweet spot: useful quality for summaries, drafts and Q&A over documents, on hardware you already own. For detailed cost estimates there is a dedicated guide.

The steps: Ollama and OpenWebUI

The simplest path in 2026 uses two open-source tools: Ollama, which downloads and runs the models, and OpenWebUI, which puts a chat interface in your browser on top of it.

  1. Install Ollama from the official site (Windows, macOS and Linux).
  2. From the terminal, download a model: ollama pull qwen3:8b for the mid range, ollama pull qwen3:4b if the machine is modest.
  3. Check it answers: ollama run qwen3:8b and ask something. If it replies, the engine works.
  4. Install OpenWebUI (Docker or Python) and point it at Ollama: browser chat with history and user management, on your network.
  5. Connect it to your documents (RAG) only once the basics run: that is the step that turns the chat into a useful assistant.

No data leaves the machine: both tools run locally, and open-weight models send nothing to anyone. That is the difference from public-cloud assistants.

The mistakes we see most often

  • Underestimating memory. If the model does not fit in RAM/VRAM it slows to a crawl: a smaller model beats a limping one.
  • Expecting "the ChatGPT model". A local 8B is good at summaries, extraction and drafts; it is not the frontier model. For office use that is usually enough — better to know it upfront.
  • Forgetting maintenance. Models and tools update; without a minimum routine the installation rots. In a company, someone must own it.

When it makes sense to get help

DIY is perfect for trying things out. When the AI becomes a work tool — company documents in, reliable answers out, multiple users, permissions — the choices (which model, which hardware, how to index the archive) become a small project. In that case Altovar's Private AI plan covers assessment, implementation and operations on European infrastructure or right inside your own.

Frequently asked

Do I need internet to use a local AI?

No, after installation: model and interface run on your machine. The connection is only needed to download models and updates.

Can I run a local AI without a GPU?

Yes, with models up to 7–8 billion parameters on a recent CPU and 16 GB of RAM: slower than with a GPU, but perfectly usable for trials and light loads.

Do my documents really stay on my computer?

Yes: processing and storage live on the machine running the stack. No content is sent to external services unless you explicitly configure one.

About this content

General informational guide. Regulations and technical requirements change over time: always check the official sources and assess your specific case with a qualified technical or legal advisor.

GET IN TOUCH

Talk to our team.

We'll show how Altovar integrates into your infrastructure. No commitment, no sales pitch. Just a technical conversation.

BOOK A DEMO →