CANYON

Primary model benchmark standard API

This is the primary result table: it includes only standard API endpoints and local models that were tested directly, without any agent wrapper or CLI framework. A flat, high row across languages is the grounding-invariance signature; gaps below about 0.05 should be treated as noise.

Experimental model benchmark agent CLIs & combined

This combined leaderboard incorporates frontier models running inside developer agent command-line interfaces (such as Anthropic's Claude Code and OpenAI's Codex). These agent frameworks wrap the models in their own framing instructions, which dramatically changes their behavior and direct comparison validity.

A note on the top rank (The Haiku anomaly): While claude-haiku-4.5 (running inside the Claude Code agent) occupies the top spot here with a perfect 1.000, the gap to the runners-up (0.967) is just a single question. Almost all larger models (Sonnet 4.6, GPT-5.4, Qwen3) fell into a single grammatical trap in German (interpreting the ambiguous "Grand Canyon flying to Chicago" literally as the object flying due to syntax rules), whereas Haiku favored the pragmatically sensible "I" (the narrator). Such minor variations are well within the run-to-run noise threshold (~0.05) and do not represent general superiority.

What the score means

Local multilingual run

The question & spirit

Do LLMs merely stitch together likely word sequences — the stochastic parrot view — or do they build internal world models, as Hinton argues? CANYON asks trick questions where the statistically obvious answer is wrong, then checks whether the model overrides the trap.

An honest note: CANYON is a small experiment by an independent researcher, not a peer-reviewed benchmark. It grew out of Hinton's idea and my own attempt to turn that idea into a simple instrument anyone can run at home. The numbers are not proof of understanding; they are a small pile of behavior that is hard to get from text statistics alone.

The most valuable thing this tool offers is its size — it is small enough to read in an afternoon and break. Please reproduce the experiment: add a language, widen the suites, run the semantic judge, or point it at a model we could not reach. The most useful outcome would not be agreement with our numbers, but someone finding where this simple instrument is wrong — and saying so.

Does the door change the room? access-path

The leaderboard above talks to plain chat endpoints. Here we asked the same six-language probes through two frontier-model command-line agents — Anthropic's claude -p and OpenAI's codex exec — each with its shell tools stripped and run from an isolated config. Same questions, different doorway. Bars are mean SPI across languages; colour marks the access path.

Codex targeted re-check en/de/es

After the first Codex access-path run, we repeated the unstable English, German and Spanish slices with the same tools-off Codex setup. This table is a narrow robustness check, not a replacement for the six-language leaderboard.

Raw targeted outputs are published in results/leaderboard_codex_rerun_targeted.json.

Latent-space semantic drift

Cosine similarity of real hidden-state activations between successive steps inside the same counterfactual world, per layer. The grounding signature is depth structure: shallow layers diverge with the changing wording, while deep layers converge on a shared representation of the one situation. The curves below show exactly this low-shallow → high-deep pattern.

Open dataset questions + conversations

The exact probes and the actual model conversations are published as plain JSON Lines under dataset/. Inspect the questions, read the full transcripts, and judge the answers yourself instead of trusting a score. Regenerate anytime with python3 scripts/build_dataset.py.

Reproduce it

pip install -r requirements.txt

# Behavioural sweep across six languages (local llama.cpp endpoint)
python3 scripts/run_benchmark.py --backend black \
    --model openai/<your-served-model>.gguf

# Real-activation drift on a CPU-friendly local HF model
python3 scripts/run_benchmark.py --backend white --wl-lang en,zh,ru

# Rebuild the whitepaper §4 and this site's data
python3 scripts/build_report.py