Agent A/B Lab: demo-pricing

Is the careful configuration worth its extra wall time on a small bug-fix job, or does the cheap one land the same result?

Refusals and notes

None. Every check on the refusal ladder passed.

The numbers

Support is the fraction of 4000 resamples of these pairs in which the mean difference kept its sign, counted strictly: a resample whose mean came out exactly zero is evidence for neither arm and is reported separately. This experiment locked call_at 0.9 and min_pairs 6, so a direction is named only when at least that fraction of resamples landed strictly on one side, and never on fewer than that many complete pairs, nor before every pair of the sealed sample size is complete. The 90 percent interval beside it is a separate summary of how uncertain the size of the difference is; it is not the rule. A central interval that crosses zero while the signed support clears the threshold is an ordinary result, and chapter 6 works through one.

Every pair

Every run, including the failures

Demo data. Usage and dollar amounts are synthetic. The dollar column below came from a runner that reported making it up.

Token columns are whatever your runner reported, normalised into the same four names. Shapes present in this ledger: claude-json. These names do not mean the same thing across vendors: a Claude Code run counts cache reads separately from input and a Codex run reports cached input inside its own count. Both runners may report reasoning tokens. The lab preserves the fields it receives; their accounting semantics can differ across vendors, and a field a runner never sent stays absent rather than becoming a zero. Compare a column across two arms of the same shape; across different shapes, read it as a rough size, not a rate.

Time and cost

Demo data. Usage and dollar amounts are synthetic. These totals are the demo agent's own invented numbers.

Totals across every attempt of that arm, in the runner's own accounting. A total is printed only for fields the runner actually reported; a dash is a field it never sent, which is not the same as a zero.

Demo data. Usage and dollar amounts are synthetic. No money was spent to produce the dollars below.

Usage note from the runner: synthetic usage from the lab demo agent, not a real bill

These dollars are the client side estimates your runner printed, from its bundled price table. Anthropic's cost tracking documentation says not to bill anyone or make financial decisions from them, and the authoritative number is in your console. Treat them as a scale, not an invoice.

What was locked, and what stayed still

Pre-registration locked at 2026-09-13T11:29:12Z, sha256 d949a314193c72b2. Acceptance set sha256 7da1ab6c2dc37457. Scorer surface sha256 08c522baf6f3fdc6 over 2 files. Also frozen: both arm commands, seed 20260918, 8 planned pairs, the task text and the fixture tree.

The text below is the copy sealed at lock time, read back from /private/tmp/kits-ux-final-ab-20260913-r2/3.14.3/agent-ab-lab/demo-result/locked/prereg.md, not the file as it stands now. If the two differ the report says so above and refuses a direction.

# Pre-registration: demo-pricing

Written by `ablab.py demo` so the loop can be shown end to end. Your own file
starts from templates/preregistration.md and is written by you, before the run.

## The decision this is for

Whether to make the careful configuration the default for small bug-fix jobs,
given that it is slower.

## Arm A

The cheap configuration: a fast pass at the failing tests, no review step.

## Arm B

The candidate: the same job with a review step that re-reads the task rules
before finishing. One difference from arm A, not three.

## What must be identical

The fixture copy, the task text, the acceptance set, the machine. The order of
the arms alternates by pair and the lab refuses a verdict if it does not.

## The two rival predictions

- **Prediction A wins:** the cheap arm scores within 0.05 of the careful arm,
  so the review step is not worth its wall time.
- **Prediction B wins:** the careful arm scores at least 0.15 higher, driven by
  the held-out boundary cases the visible tests never mention.
- **What a null result looks like:** the mean difference sits inside plus or
  minus 0.05 and the support stays between 0.10 and 0.90.

## The scorer, fixed now

fixtures/acceptance/cases.json, 6 required cases and 5 regression cases. Score
is the fraction of all 11 that pass.

## The analysis settings, fixed now

The lab reads these three lines at lock time, validates them, and writes them
into prereg.lock.json. From then on the lock is what the analysis uses.

call_at: 0.90
min_pairs: 6
retry_policy: none

## Sample size and stopping rule

Eight pairs, decided now, run to the end regardless of how pair three looks.

## What would make you throw this out

A touched acceptance set, unequal starting trees, an unbalanced order, or any
run whose command did not start.

## Known limits, written before the result

One task, one repo, one scorer, eight pairs, one machine. Nothing here
generalises to a different codebase or a longer job.

Commands

arm A: python3 "/private/tmp/kits-ux-final-ab-20260913-r2/3.14.3/agent-ab-lab/lab/demo_agent.py" --profile mixed-cheap --work "{work}" --emit-usage

arm B: python3 "/private/tmp/kits-ux-final-ab-20260913-r2/3.14.3/agent-ab-lab/lab/demo_agent.py" --profile mixed-care --work "{work}" --emit-usage