SoL-Pi: fewer tokens for coding agents, but measure the savings

SoL-Pi changes how coding agents operate: in the authors' benchmark it reduces token traffic while preserving performance against the baseline. Updated on 28 October 2026, this analysis starts from the paper published on 17 September and separates its research results from the checks needed before using it on a company's code.

Four mechanisms, one harness

SoL-Pi packages four techniques that the researchers say survived testing across a growing set of development environments: Action Fusion, ObservationPack, an Evidence-Preserving Reducer, and Online Context Compact. Together they target how an agent executes actions, compacts context, handles tool output and decides what it needs to re-read, rather than changing the underlying language model doing the reasoning.

What the benchmark showed

On EdgeBench, a 51-task evaluation, the authors report SoL-Pi matches the performance of its baseline harness, Pi, across two different backing models, while cutting recorded token traffic by 44.7% to 49.0% and reducing API cost by roughly a third. Those are the paper's own reported numbers on its own benchmark — a useful signal of what's possible, not a guarantee of the same savings on a different codebase or toolchain.

What this means for engineering teams

The code is open-sourced at github.com/NVlabs/SoL-Pi, so teams running coding agents at scale can test the harness against their own repositories and tooling rather than taking the paper's benchmark numbers at face value. Early GitHub interest is not the same as production adoption, and a harness-level efficiency gain doesn't change what the underlying model can or can't do — it changes how cheaply you can run it. Anyone building on this should measure token and cost savings on their own workloads before citing the paper's figures in a customer proposal.

Why it matters: measure what coding agents finish

For a useful trial, choose tasks the team actually performs: fixing a bug, adding a small feature or updating a dependency with its tests. Repeat the comparison with the same model, repository, instructions and acceptance criteria. Count failed attempts, waiting time and manual corrections too: a session that uses fewer tokens but leaves the job unfinished is not a saving. Keep the starting state so changes can be compared and reversed without losing the original evidence.

Record input tokens, output tokens, billed cost, duration and tasks accepted after review as separate measures. Pricing depends on the model and the provider's terms: less traffic does not automatically produce the same percentage reduction on an invoice. An MSP should separate measurements by project, without copying one customer's content into another customer's logs or trials. The comparison should remain reproducible after an update, and its record should make clear which model and configuration were actually used.

Observation handling deserves its own check. Confirm that a compilation error, failed test or security warning can still be recovered after context reduction. Decide where archives are stored, who can read them and when they must be deleted. Before enabling an additional model to summarize logs, review the data path and credentials involved. A useful summary must retain the evidence a reviewer needs to judge the change, including failures that contradict an apparently successful final response.

Start with an isolated repository copy and test data, without production credentials. Set a spending limit and a stopping condition, then expand the scope only if quality and completion times remain acceptable. The decision is about the cost of a completed, verified task rather than a single counter. To place that evaluation within a broader business process, start with Altovar's artificial intelligence services and define the workload, responsible reviewer and success criteria before choosing the tooling.

Sources: arXiv, GitHub.