CTXBENCH

Every vendor selling a context layer for AI coding agents — a memory MCP, a shared-context service, a CLAUDE.md convention — claims it makes the agent better. Almost none of them have measured it, and the people with the strongest incentive to measure are the ones a reader should trust least to do it. I was building one of those products, and the question I could not answer honestly was whether it changed what an agent actually finished, as opposed to whether it felt like it did.

Two arms, everything else held constant: the agent alone, and the same agent plus one context provider, on the same hand-authored defects, at the same pinned model, graded only by whether the target repository's own test suite goes green. The property the whole design turns on is that the only difference between the arms is one file — both arms invoke the same binary with the same flags, and only the MCP config path differs. A unit test asserts exactly that, and a runtime assert re-checks the agent's own startup event on every rollout. Because the author of the benchmark also builds the product under test, the analysis was pre-registered before the harness existed: arms, metrics, statistical method, the minimum detectable effect, and the exact wording used to interpret a result, all fixed before a single number was seen.

I was building a context layer for AI coding agents and could not answer, honestly, whether it worked. Every product in that category claims the agent gets better with it. Almost none of them have measured it, and the ones with the most to gain from measuring are exactly the ones a reader should trust least. ctxbench is that measurement, built by precisely the person you should not trust to build it, and arranged so that a favourable result would have been hard to manufacture.

The design has one property that everything else serves: the only difference between the two arms is one file. Both arms shell out to the same agent binary with the same flags, and only the MCP config path differs. A unit test pins that. A runtime assert re-checks the agent's own startup event on every single rollout — expected MCP servers all connected, zero leaked skills, the pinned model — and a rollout that fails the check is thrown away and logged rather than quietly retried until it passes.

The plan was wrong about its own founding assumption. Twelve findings were probed first-hand before any code was written, and three of them killed planned approaches: the flag the entire purity design rested on pins the agent to three tools and cannot be widened, Docker was not on the machine, and both planned task sources were unusable. The largest correction came later. The experiment had been designed to control for an MCP schema tax of roughly 9,800 tokens. Measured on an identical no-op prompt, three replicates identical to the token, it is 945. The agent registers all 30 tools and never puts their schemas in context. The premise the control was built for did not exist.

Then three bugs, each of which would have published a false result. A macOS path bug meant no test file ever matched its grading entry, so every rollout in both arms graded unresolved — a perfect false null, caught only because a trajectory was read by hand. A design flaw left each defect as an uncommitted change, so the agent's first orienting command printed the exact inverse of the fix. And the reporter selected run directories lexicographically, printing superseded numbers under a correct-looking heading. Four more were caught by gates rather than by luck, including a fixture that spelled out the fix for its own task verbatim, and live bearer tokens committed under a dot-directory the analysis code skipped.

The result was a null. Aggregate paired delta +4.5pp, 95% CI [−13.6, +13.6], McNemar p = 1.000, with only 3 of 22 tasks discriminating between the arms against a pre-registered floor of six. And then the accident that is the most interesting thing in the repository: a third arm's batch re-ran the control arm with an identical flag set on the identical task set — an unplanned partial replicate, and exactly what a single-repeat design is supposed to be blind to. It disagreed with itself on 3 of 24 tasks, against a between-arm difference of 1 task net. The noise floor was at least as large as the signal. It also dismantled the one efficiency story that looked like a finding: three control rollouts that took 1128s, 551s and 1354s finished in 82s, 97s and 68s on the replicate. "The context layer makes the agent 68% faster" would have been arithmetically true and substantively false.

The README leads with the null, gives the self-disagreement more room than the headline, and states the conflict of interest down to the npm account the package ships from.

A CLI drives a batch runner — randomised interleave, resumable, with a cost guard against a locked budget ceiling — which drives one rollout at a time. Each rollout crosses two seams.

Sandbox prepares a pristine workspace: clone at the pinned commit, install from the committed lockfile, apply the defect patch, and record the exact toolchain into every artifact. Provider returns an AgentConfig, and that is the only channel a context layer has. A provider may influence the agent through AgentConfig plus the context files it declares, and never through the prompt, the sandbox or the grading. Declared files are diffed against git status and the rollout aborts if any other tracked file changed, so a provider owns its context and cannot touch the code under test. That restriction is what keeps the single-difference property true no matter who writes the provider.

The agent runs headless with streaming JSON output, and the first thing the harness does with that stream is assert on the startup event. Pass, and grading runs: resolved means every fail_to_pass test flips and every pass_to_pass test holds, in the target repository's own suite, with no model anywhere in the loop. Fail, and the rollout is marked invalid_environment, excluded from both arms symmetrically, and logged.

Everything lands on disk as a trajectory, a run record and a report per rollout, plus a run summary. The statistics layer — paired deltas, McNemar, BCa bootstrap, medians beside means — reads only those files, and the reporter injects its output into the README between markers. Providers are loaded by dynamic import only, with a test asserting on the source that no static provider import exists, so "runs with zero vendor code installed" is machine-enforced rather than intended. LocalSandbox is the v1 default because Docker was absent; DockerSandbox ships implemented, auto-selects when a daemon answers, and is labelled unexercised.

Twelve findings (F1–F12 in the build log) were measured first-hand before a line of harness code existed, and three of them killed planned approaches outright. `--bare`, the flag the entire arm-purity design was built on, turned out to pin the agent to exactly [Bash, Edit, Read] with no way to widen it — which would have floored completion in both arms and compressed the very effect under measurement. Docker was absent from the build machine. And both planned task sources were unusable: SWE-smith is Ubuntu-and-Python only, and PR-mining was impossible because the candidate repositories have zero merge commits between them.

The biggest finding arrived later and overturned the premise the experiment was designed to control for. The whole project was planned around an MCP schema tax of roughly 9,800 tokens — a byte count over the server's own tools/list independently predicts ~9,400 — and the measured figure, on an identical no-op prompt across three replicates identical to the token, is 945 (+3.1%). Claude Code registers all 30 tools and never puts their schemas in context; ~31 tokens per tool is a name and a one-line description. It is written up as an overturned premise rather than quietly corrected.

Three bugs would each have published a false result. The first was a macOS path bug — vitest reports `/private/var`, the harness stripped `/var` — so no test file ever matched its grading entry and every rollout in both arms graded unresolved. That is the most dangerous kind of wrong answer: a plausible, publishable, entirely false null. Nothing in the pipeline flagged it. It was caught because one trajectory was read by hand and the agent had visibly fixed the bug it was being marked down for. The second: defects were applied as uncommitted changes, so the agent's first orienting command — `git diff` — printed the exact inverse of the fix. Ten of twelve pilot rollouts solved, four of them in under 30 seconds. Defects are now folded into the pinned commit. The third: the reporter selected run directories lexicographically and printed superseded numbers under a correct-looking heading. It now selects by newest summary and always prints its source — and the same failure class returned during the documentation pass, when the documented report command, with no run pinned, resolved to the wrong batch and silently swapped the README's headline result. The fix was to pin the run in the script rather than let the reporter guess.

Four more were caught by gates rather than by luck. The oracle gate rejected a snapshot test embedding a relative timestamp, which could have manufactured a between-arm difference out of wall-clock drift alone. A shared-context fixture spelled out, verbatim, the fix for one task; leakage is now a mechanical symbol diff of every fixture against its own defect patch. The clean-clone gate found live room bearer tokens committed under a dot-directory that the analysis code skipped — deleted, and purged from git history. And a multi-turn ordering bug ran turn 2 before turn 1, so the treatment arm carried context forward from a task it had not done yet.

The MCP schema tax is 945 tokens (+3.1%), not the ~9,800 the project was planned around. `--setting-sources ""`, the flag that guarantees the control arm is clean, also suppresses CLAUDE.md and AGENTS.md discovery entirely — verified three ways, and the consequence is that the third arm has to differ in two flags rather than one, which is stated rather than buried. The useful side effect is that the control arm's cleanliness became verified rather than assumed: the target repo's own checked-in CLAUDE.md was not ambient context for either arm.

The most valuable measurement was an accident. The third arm's batch re-ran the control arm with an identical flag set on the identical task set — an unplanned partial replicate, and precisely the thing a single-repeat design is supposed to be unable to see. It disagreed with itself on 3 of 24 tasks against a between-arm difference of 1 task net, and it dissolved the one efficiency story that looked like a finding: three tasks that took 1128s, 551s and 1354s in the first run took 82s, 97s and 68s in the second. Separately, a third arm delivering the same content as a plain CLAUDE.md moved nothing at all — both arms solved every task, in every category.