What hardware you need, by model size
The right question is not "how powerful must my PC be?" but "which model do I want to run?". Memory — RAM and especially VRAM if you have a GPU — decides everything. Rule of thumb: a Q4-quantized model occupies in memory roughly half its parameter count in gigabytes.
| Model | Memory required (Q4) | Runs well on |
|---|---|---|
| 3–4 billion params (e.g. Qwen 4B) | ~3 GB | Any recent PC, even without a GPU |
| 7–8 billion (e.g. Llama 8B, Qwen 8B) | ~5–6 GB | PC with 16 GB RAM or a GPU with 8 GB VRAM |
| 13–14 billion | ~9–10 GB | 12 GB GPU or a workstation with 32 GB RAM |
| 30+ billion | ~20 GB and up | Dedicated GPUs (24 GB+) or a server |
For office use, the 7–8 billion range is the sweet spot: useful quality for summaries, drafts and Q&A over documents, on hardware you already own. For detailed cost estimates there is a dedicated guide.
The steps: Ollama and OpenWebUI
The simplest path in 2026 uses two open-source tools: Ollama, which downloads and runs the models, and OpenWebUI, which puts a chat interface in your browser on top of it.
- Install Ollama from the official site (Windows, macOS and Linux).
- From the terminal, download a model:
ollama pull qwen3:8bfor the mid range,ollama pull qwen3:4bif the machine is modest. - Check it answers:
ollama run qwen3:8band ask something. If it replies, the engine works. - Install OpenWebUI (Docker or Python) and point it at Ollama: browser chat with history and user management, on your network.
- Connect it to your documents (RAG) only once the basics run: that is the step that turns the chat into a useful assistant.
No data leaves the machine: both tools run locally, and open-weight models send nothing to anyone. That is the difference from public-cloud assistants.
The mistakes we see most often
- Underestimating memory. If the model does not fit in RAM/VRAM it slows to a crawl: a smaller model beats a limping one.
- Expecting "the ChatGPT model". A local 8B is good at summaries, extraction and drafts; it is not the frontier model. For office use that is usually enough — better to know it upfront.
- Forgetting maintenance. Models and tools update; without a minimum routine the installation rots. In a company, someone must own it.
When it makes sense to get help
DIY is perfect for trying things out. When the AI becomes a work tool — company documents in, reliable answers out, multiple users, permissions — the choices (which model, which hardware, how to index the archive) become a small project. In that case Altovar's Private AI plan covers assessment, implementation and operations on European infrastructure or right inside your own.
Frequently asked
Do I need internet to use a local AI?
No, after installation: model and interface run on your machine. The connection is only needed to download models and updates.
Can I run a local AI without a GPU?
Yes, with models up to 7–8 billion parameters on a recent CPU and 16 GB of RAM: slower than with a GPU, but perfectly usable for trials and light loads.
Do my documents really stay on my computer?
Yes: processing and storage live on the machine running the stack. No content is sent to external services unless you explicitly configure one.