Local AI week: a shrunk 27B model for RTX 3060, an engine built for Apple Silicon

Two local-inference releases landed within a few days of each other this month: PrismML's Ternary Bonsai 2 27B, a heavily compressed model aimed at consumer GPUs, and Inco AI's Splash, an inference engine built specifically for Apple Silicon.

Bonsai 2: a 27B model that fits in 5.9GB

PrismML released Ternary Bonsai 2 27B on 17 September under an Apache 2.0 license. It uses ternary weights with FP16 group-wise scaling to shrink the model to 5.93GB, against 53.8GB for the same model in FP16, while PrismML says it retains 98.2% of the parent Qwen3.8-27B model's average score across 20 benchmarks. The company reports up to 143 tokens per second on an RTX 5090. A separate, also Apache 2.0-licensed community project tuned decode performance specifically for the RTX 3060 12GB -- described in its own benchmarking as the single most common desktop GPU on Steam's August 2026 hardware survey, at just under 4% share -- and measured decode speed improving by as much as 1.53x at short context lengths.

Splash: an engine built around one model, for Apple Silicon

Inco AI released Splash as an open-source, Apple Silicon-only inference engine specialized for two specific models, Qwen3.8-27B and Qwen3.6-35B-A3B, rather than built as a general-purpose model server. Inco reports around 144 tokens per second running Qwen3.8-27B on an M5 Max MacBook Pro, which it says is up to three times the decode speed of Ollama and twice that of MLX in its own comparisons. The engine requires an Apple M3 chip or newer and at least 36GB of unified memory, and Inco says it was integrated into LM Studio's Bionic release the same week.

What this means for MSPs

Both sets of figures come from the vendor or the project's own benchmarking, not an independent audit, and a strong demo on a single desktop GPU or MacBook is not the same problem as running tenant-isolated inference for many customers at once. What the pairing does show is that consumer hardware is closing the gap on workloads that used to require a rack -- worth tracking for pilot and edge use cases, not yet a substitute for a hosted, multi-tenant platform.

Sources: MarkTechPost, Inco AI.