Agent A/B Lab

One limit, stated plainly. The lab seals a copy of your acceptance cases inside the experiment directory, and your arm's command runs inside that same directory tree. The process that scores what your arm wrote holds no expected value, no tag and no state of the lab's, and it runs in a copy taken outside that tree, so it cannot forge a result and cannot walk to the sealed copy. An arm can, like any process on your machine, read a file. An arm that goes looking for the sealed copy and answers from it will score well and will not be refused, because this kit detects a compromised measurement rather than containing an agent that has shell access, and reading a file it was not stopped from reading is not something the instrument can see. If your arms are agents you do not fully trust, run them in a container or on a throwaway machine and keep the experiment directory off it. If they are your own configurations, which is what this kit is for, the thing it protects you from is the accident: the agent that edits the tests, the prediction you rewrote after the result, the retry that quietly replaced a failure. Those it catches, every time, and it refuses a verdict rather than averaging them in.

Four ways, and only one of them is detection after the fact. The acceptance cases are never copied into the work directory: lock refuses a layout that puts them inside the fixture, and refuses a symlink, a hard link or a plain copy of them in there too. No case reuses an input the visible tests show. Each case is evaluated by its own process of the submitted module, which holds no expected value, no tag and no lab state at all, and which runs in a copy of the work tree taken outside the experiment and deleted when the last case ends: that process holds nothing that would tell it the right answer and can reach nothing that would, so a correct answer is the only way to a point, and a process that answers with anything else, or that leaves anything running behind it, is named and refused rather than believed. The lab also writes nothing down that would help: a failed case records what your code returned and never what the case wanted, and verify names any run artefact that carries a majority of the held-out answers. Everything else, the cases and the scorer and their fingerprints, is checked before every arm, again before scoring and again after it, with the score computed from the copy taken at lock time, and any movement refuses a verdict. That last part is detection, not containment: an agent with shell access can reach outside its work copy, and the kit says so plainly rather than selling you a sandbox it does not have.

Partly, and the kit is precise about the limits. Claude Code reports a per-run dollar estimate and tokens in its JSON output; Codex CLI reports tokens and no dollars. The lab parses both, and refuses every cheaper or equal-cost claim when either arm reported nothing or the two arms report different shapes. The dollar figures are client side estimates, which the report says next to them every time.