What a harness can — and cannot — do for a small local model

Measured forensics from Azot, an agent harness for 1–9B models on a single 12 GB GPU. Same model, six scaffold arms, pre-registered predictions. Including the finding we had to retract.

Azot's terminal UI paused at a Safety Governor approval prompt: a proposed edit to sum.py rendered as a colored diff, with allow / always / deny choices. — Azot pauses on a file write: the shell-AST **Safety Governor** surfaces the proposed edit as a diff and waits for `allow / always / deny`. On this benchmark the guards buy no extra pass@1 — on your own workstation they are the point.

The experiment

Aider Polyglot (python subset, 34 tasks) through Harbor. Every arm drives the same model — Qwen3.5-9B-Q4_K_M on the same llama.cpp server, same sampling, same step budget. The only variable is the scaffold:

arm	pass@1	wall clock
oracle (installation sanity check)	34/34	10 min
`mini-swe-agent` (~100 lines of bash), thinking on	8/34	6 h 56
toza — single-file Python control agent (~600-token prompt, zero guards), thinking off	8/34	2 h 46
Azot (~8k lines of Go, governor, sandbox, six loop guards), thinking on	7/34	4 h 23
Azot, thinking off — three runs, one loop-fix apart	7 / 5 / 3	2–4 h

The instrument has ±6 pp of noise. 1σ at n=34, p≈15% is 2.08 tasks = 6.1 pp — and two runs of byte-identical code landed 5.9 pp apart, matching theory to the decimal. Read the table accordingly: 8 vs 7 is not a difference. Any single-run claim resting on a smaller gap — in either direction — is noise. We withdrew several of our own.

That is the headline, stated against our own interest: on this benchmark, eight thousand lines of Go buy no measurable pass@1 advantage over a hundred lines of bash or one file of Python. Three independent scaffolds, statistically identical scores.

Where the failures actually live

We classified every failed trajectory of both harness arms into failure modes (hallucinated tool / budget exhausted / early resignation / plan-without-execution), with per-task output for manual spot-checks:

failure mode	Azot (thinking on)	control (thinking off)
hallucinated tool name	0	0
budget exhausted	8	2
plan without execution	0	1
worked, then wrong — mutated, verified, still failed	19 (70%)	23 (88%)

The dominant failure class on both scaffolds is a model that does everything procedurally right — reads, edits, runs the tests, iterates — and produces the wrong logic anyway. No process guard fixes that. The ceiling is the model. The budget-exhaustion asymmetry (8 vs 2) is the cost of reasoning mode, not a guard effect: with thinking on, the reasoning phase eats the step budget before content is emitted.

The finding we retracted. The first version of our classifier reported five “hallucinated tool” failures for Azot. The claim carried a pre-registered escape clause — “if the schema arm still shows these calls, my understanding is wrong” — and the A/B run tripped it: the “hallucinated” tools turned out to exist; the classifier’s author had assumed the tool list instead of reading the source. All 17 such calls had executed successfully. The retraction is recorded in the repo history, next to the claim. That clause found the error in 15 minutes; without it we would have hunted a phantom bug in the backend. This is why every measurement here ships with a prediction written down before the run.

What the guards do change

Pass@1 is blind to trajectory quality. The mechanics are not — and they are the one place the measurements are consistent and directional:

mechanical effect	measured
run timeouts across three same-code arms, after one loop fix	11 → 5 → 3
peak writes to a single file (repeat-loop pathology)	46 → 6 (−69% write calls)
silent give-up finals (control has no guard for this)	1 in control, 0 in Azot
unusable-reply recovery firing on a platform it wasn’t built on	observed live on Ollama

Same number of solved tasks — but fewer pathological trajectories, shorter runs, fewer wasted tokens. That is what a harness actually buys at this model scale. That, and the part benchmarks cannot see at all: Azot’s shell-AST safety governor and namespace sandbox run disabled in benchmark containers. On your workstation they are the point.

Mirror-image failure modes

Earlier forensics on Terminal-Bench 2 (15-task subset, same harness, different models) found the failure signatures are model-specific, not scale-specific: qwen3-coder-30b “does too much and finishes wrong” (wrong-after-work, 7/15 trials); gpt-oss-20b “does too little and stops early” (never attempts, 6/15). Two models, the same scaffold, opposite pathologies — so any fix tuned on one model was null or harmful on the other. And the same harness driving a 480B model scored in line with that model’s public leaderboard number: the harness scales with model capability; it does not substitute for it.

If you run small models locally

What this measurement series means in practice:

A harness will not make your 9B model solve more tasks. Anyone claiming otherwise for this model class should show you their ablation against a hundred-line baseline.
It can make the model safer and cheaper to run: bounded pathologies, recovered malformed replies, a shell governor between the model and your filesystem.
Failure modes are yours to measure: profile your own model’s trajectories before adopting anyone’s guard — including ours.

Azot is the harness these findings built: failure-driven (nearly every guard in the loop carries the date and task of the live run that motivated it), measured against its own interest, scoped to being a coder/executor for local models. The repository ships the full methods: FINDINGS.md (the five load-bearing findings the code cites), CHANGELOG.md, the benchmark adapters, and the classifier used above. Its control group is toza.

Methods: all Polyglot runs 2026-07-21→23, Qwen3.5-9B-Q4_K_M via llama.cpp (single slot), temp 0.2, 60 steps, 900 s/task, code frozen and predictions recorded before each run. TB2 runs 2026-07-13→16. Numbers, transcripts and per-task classifications are reproducible from the repo’s bench/ directory. · pedjaurosevic