A multilingual benchmark probing semantic grounding vs. surface statistics: questions where the statistically likely answer is wrong. Six languages, open dataset, full transcripts. A methodology roadmap toward v2 is public in the repo.
Local-first TUI & web agent for any llama.cpp server — RAG, semantic memory across sessions, schema-constrained actions. v3.0 measures its grounding: a public leaderboard ranks models on citation discipline. Shipped and maintained.
A retro handheld-console terminal for AI pair programming — real PTYs, approval dialogs, git undo checkpoints, in a Game Boy shell.
How much of an agent's score is the harness? An 8k-line Go harness, a 100-line bash scaffold and a single Python file drive the same frozen 9B model on the same 34 tasks: statistically identical pass@1. What the guards do buy is measured too — and a retracted finding stays on the record.
Same agent loop, different decoding regimes (free-form, grammar, JSON Schema), across model sizes. Does constraining output cost capability — and for whom?
What survives a hard process restart: structured, typed memory or a raw transcript? Oracle-scored delayed-intention tasks around a frozen local model.
A preregistered study: does apparent continuity causally depend on stored state? Targeted lesions, placebo deletions, token-matched controls, fixed thresholds.
One fixed model ladder, one shared oracle: mapping which scaffolding interventions help at which scale — with sign, not vibes.
The model stays frozen. No fine-tuning in the loop; every effect is attributable to the scaffold.
Oracles over self-reports. Primary metrics are decided before the run, scored by code, not by the model's own claims — and thresholds don't move after seeing results.
Consumer hardware, open weights. Everything runs on hardware people actually have; protocols, seeds and raw records ship with each study.