One
The benchmark is the step already in use, written down. Ours is AP1 on the raw arm: paste the Apollo 11 slice, ask one question, take one JSON reply. One prompt, one call, no tools.
Talk four of four · Draft · [ASK: talk date]
Take an LLM step you already run and build the evaluation that tells you which rungs pay
AI in Systems Engineering · Andrew Dunn
Deck — The session as presented to the group.
Renting frontier intelligence costs more per task even as tokens get cheaper, so enterprises will mix models and send bounded jobs to small tuned open ones. Which rung pays for you is a question only your own bench answers. Half of this talk is how to build that bench; half is the ladder it reads.
Half one: evaluation design · Half two: the ladder
Scroll to run it Read on: the plates are printed at their finished state
Plate I · Half one: evaluation design
A bench starts as a record of one step as it runs today: inputs, the call, the output, who checks it, what a bad output costs
One
The benchmark is the step already in use, written down. Ours is AP1 on the raw arm: paste the Apollo 11 slice, ask one question, take one JSON reply. One prompt, one call, no tools.
Two
The card fills from the recording. Input: 17 files, 76,647 tokens billed on Sonnet 5. Call: a fixed system prompt and a schema line, one call, a 4,000-token output cap. Output: three typed fields. Cost: $0.154, and the reply was 42 tokens.
Recorded rows: lesson-representation recorded/claude-sonnet-5.json (MEASURED).
Three
A leaderboard measures someone else's step. A constant-output null model scores an 86.5% length-controlled win rate on AlpacaEval 2.0, and fresh human-written twins of GSM8k cost models up to 8% accuracy. The step we run is the one we can check.
Zheng et al., ICLR 2025 (PR): null model 86.5% LC win rate on AlpacaEval 2.0.
Zhang et al., NeurIPS 2024 (PR), GSM1k: accuracy drops of up to 8%.
Four
Two rows stay blank until a person fills them. [ASK: who checks AP1] [ASK: what a wrong landing velocity costs]
Plate II · Half one: evaluation design
Take cases from real traffic and real misses; give each an answer key; separate what a parser checks from what needs a person
One
Cases come from real traffic and real misses. Each case gets a key, written before anyone looks at an answer.
Anthropic engineering, Jan 2026 (V): "20-50 simple tasks drawn from real failures is a great start." Vendor advice; no measured size behind it.
Husain, 2024 (B): run unit-test style checks before a model judge.
Two
AP1's key has three rows: the requirement id HLR-R002 (a string), the speed 2 (a float, tolerance 0.05), and the unit m/s (a string). A parser checks all three. No model grades.
Three
The whole bench is forty tasks and 126 fields: 79 floats, 12 exact strings, 35 string-contains checks on free text. The type decides the scorer, so the type goes into the key.
deck.json answer_schema: 24 discovery plus 16 explanation tasks (MEASURED).
Four
Our forty were written against a model, so they teach the method and nothing about sampling from traffic. A bench of your own starts from misses you remember. [ASK: which three real misses from the room become the first cases]
Plate III · Half one: evaluation design
A parser is a gauge, a judge more so: know what it credits, report both error rates, calibrate against one expert
One
A parser is a gauge, and a gauge credits some things it should not. Before trusting a score, learn what this one credits.
Two
Echo. AP1 on the table arm scores 1 of 3 on both models by repeating HLR-R002 from the question. Across the bench, 12 of the 35 string-contains keys and 3 of the 12 exact keys already sit in the question text.
Three
Partial against whole. Sonnet credits 69 of 126 fields (mean 59%) and gets 20 of 40 tasks wholly right. A format failure drops its fields: 110 of 126 carry a verdict. Say which count is being reported.
Four
On the 35 string-contains keys, the shipped scorer hits 24 for Sonnet and a field-scoped reading hits 15. That is gauge against gauge, kappa 0.55 pooled (0.51 Sonnet, 0.60 mini). Neither is a person. [ASK: who reads the 35 as the human reference]
Raw agreement runs high on skewed sets. At 85% agreement kappa can sit at 0.48, and a judge that always passes scores 95% on a skewed set. Report both disagreement rates and the chance-corrected figure.
Norman et al., 2026 (PP): 85% agreement can be kappa 0.48.
Husain, 2024 (B): an always-pass judge scores 95% on skewed data.
Gauge counts: Sonnet both 15, shipped-only 9, field-scoped-only 0, neither 11; mini 18, 7, 0, 10 (MEASURED).
No second rater exists in our data; no human kappa.
Plate IV · Half one: evaluation design
Date the metric, threshold, comparison and a prediction before the first call; criteria drift once you grade
One
A bar has four parts, dated before the first call: the metric, the threshold, the comparison, and a prediction.
Two
Ours, on file: the typed view against the slice, on Sonnet 5. Supported at 20 or more tasks correct for a quarter of the bill. Refuted at 10 or fewer. Predicted 10 to 14. The result stays shut until movement VII.
GPT-5 mini gets its own line: supported at 19, refuted at 9, predicted 8 to 12. [ASK: pre-registration file and date to cite]
scripts/typed-view/prereg.json: thresholds and predictions (MEASURED).
Three
Criteria drift once grading starts, and 16.0% of 445 benchmarks report any uncertainty. Both are reasons to write the bar early. It is a proposal; we have no evidence here that it changes outcomes.
Bean et al., NeurIPS 2025 (PR): 16.0% of 445 benchmarks report uncertainty.
Criteria drift: Shankar et al., 2024 (PP).
No evidence here that pre-registering improves outcomes.
Plate V · Half one: evaluation design
Run the step as it runs today, read transcripts, turn every new failure into a case; then the bench can price a rung
One
Run the step as it runs today. Sonnet 5 gets 20 of 40 for $6.49. GPT-5 mini gets 19 of 40 for $0.525. That is $0.324 and $0.028 per correct task.
Recorded raw arm, both models, one run per task, no repeated runs (MEASURED).
Two
Forty cases give a wide interval: about 14 to 26 of 40 for Sonnet, 13 to 25 for mini. The two models differ on one task, AP3. The bench cannot rank them.
Wilson 95% intervals from bench.json (MEASURED).
Three
Every miss is a case. Of Sonnet's 20 misses, 13 had the answer absent from the input, one was cut at the cap, six are other. For mini: 13, one, seven. Five replies per model hit the 4,000-token cap before any JSON.
Presence check: 13 of 40 tasks flagged, none answered correctly.
Four
Then a person reads the transcripts. Scores do not show a model that games the check: METR saw o3 do it on one task in 21 of 21 runs.
The misses gather into a regression set, which each rung must clear.
METR, 2025 (B): per-task counts; no general rate reported.
Plate · The hinge
You hold a bench: card, key, gauge, bar, baseline row.
One
Five pieces make a bench: the card, the key, the gauge, the bar, the baseline row.
Two
Half two adds one row per rung. A rung pays when it moves the row past the bar.
All figures from census.json, written from the recorded rows (MEASURED).
Plate VI · Half two: the ladder
Price at a fixed capability falls, but what we run at the frontier costs more each year: a bench reads cost per task
One
Two prices move in two directions. The price at a fixed capability falls; what we run at the frontier costs more each year, because we spend more tokens.
Gundlach et al., arXiv 2511.23455, 2025 (PP): the price of a given benchmark level falls 5x to 10x a year; the price of running frontier models rises 3x to 18x a year, which is token growth at the lowest listed price. Epoch, 2025 (B): fixed-capability price falls 9x to 900x a year, persistence less clear. Bai et al., arXiv 2604.22750, 2026 (PP): agentic coding uses about 1000x the tokens of chat.
Two
On our bench the same AP1 text bills 76,647 tokens at one vendor and 42,849 at the other. The tokenizers differ, so the sticker price does not compare. Cost per task does.
Across the forty tasks the Sonnet raw arm runs $0.153 to $0.193 per task, median $0.154. The mini runs $0.011 to $0.019, median $0.012. One strip per model, forty dots each.
Recording: AP1 raw rows, recorded/claude-sonnet-5.json and gpt-5-mini.json (MEASURED, one run per task).
Three
List price misleads: in 32% of model pairs the cheaper-listed model cost more, by up to 28 times. The meter has to be read on the steps we run.
Chen et al., arXiv 2603.23971, 2026 (PP): cheaper-listed model cost more in 32% of pairs, up to 28x.
Plate VII · Half two: the ladder
The bottom rungs change in minutes and roll back free; a hook takes the job away from the model
One
The bottom rungs change in minutes and roll back for free, so they go first.
Vendor and survey adoption data for agent builders (LangChain, V/S) describe agent builders, not enterprises. Not used for numbers.
Two
Context rung. The typed view gets 13 of 40 for $0.28 on Sonnet, against the full slice's 20 of 40 for $6.49. Per correct task: $0.021 against $0.324.
Against the bar from IV, 13 sits between refuted (10 or fewer) and supported (20 or more). Cheaper, and not equivalent. The bar names that outcome: neither result, so the view is not the path.
Typed view and criteria: recorded/typed-view/*.json against scripts/typed-view/prereg.json (MEASURED; the mini also gets 13 of 40 for $0.052).
Three
Hook rung. A presence check, zero tokens, flags 13 of the 40 tasks whose answer is not in the input. Neither model answered one of those correctly.
Those 13 calls billed $2.22 on Sonnet, 34% of the raw bill, and $0.19 on the mini, 37%. A schema check on the reply catches the 5 truncations per model at the 4,000-token cap, $0.97 on Sonnet.
Presence needles: src/repr/answerable.ts, ported exactly (MEASURED).
Trinh et al., Nature, 2024 (PR): a symbolic engine alongside a model solved 25 of 30 olympiad geometry problems against 10 for the prior best (the symbolic engine alone solved 14), so the checker does work the model cannot.
Plate VIII · Half two: the ladder
Multi-model use is measured, routing adoption is not; the honest baseline is the best single model
One
Using several models is measured. Routing between them, as an adoption rate, is not.
a16z enterprise survey, 2025 (S, n=100): 81% use three or more model families. Use is measured; routing is not.
Two
The baseline is the best single model. The best-of-both ceiling is 20 of 40, and Sonnet alone is 20. They agree on 19 tasks; Sonnet alone gets one (AP3); the mini gets none alone.
Three
A route is a hypothesis. Escalating from the mini to Sonnet on a format failure gives 19 of 40 for $1.48, no better than the mini alone at $0.525 and about 2.8 times the bill.
A cascade with a perfect checker reaches 20 for $4.09. It needs the checker nobody has yet. A fast-model signal is another candidate: [ASK: is Jev TypeSafe AI's System One model (jev-1.13.0 in our pilots)].
LLMRouterBench, arXiv 2601.07206, 2026 (PP): routers gain up to 4% accuracy or save up to 31.7% cost against the best single model, in the settings tested; several routers fail to reliably outperform a simple baseline (Best Single, chosen in hindsight).
Chen et al., 2023 (PP), FrugalGPT: savings 59.2% to 98.3% across three datasets. Hu et al., 2024 (PP), RouterBench: KNN/MLP routers do not significantly outperform the Zero router.
Offline routes: computed from recorded raw rows, no new calls (MEASURED).
Plate IX · Half two: the ladder
On a stable, bounded, labeled job a well-trained small model can win; training moves the cost to the teacher
One
On a stable, bounded, labelled job a small tuned model can win.
Two
Read each win for its teacher and its baseline. Christou and Tsoumakas report F1 0.83 against 0.69 and 0.66 zero-shot; the 0.83 is the best of 30 configurations.
Celikik et al. get precision/recall of 82.74/85.16 against 67.81/62.87, with GPT-5.4 as teacher. The small model learned from a bigger one, so the comparison is against that teacher's family.
Continued pretraining on a domain corpus was better on 12.1% of comparisons, tied on 49.8%, worse on 38.2% (Jeong et al.). Mostly a wash.
Christou and Tsoumakas, arXiv 2606.22606, 2026 (PP): F1 0.83 vs 0.69 / 0.66, best of 30.
Celikik et al., arXiv 2610.10827, 2026 (PP): L3 P/R 82.74/85.16 vs 67.81/62.87, teacher GPT-5.4.
Jeong et al., arXiv 2411.04118, 2024 (PR): 12.1 better, 49.8 tie, 38.2 worse.
Three
Our bench has 40 keys. That is a test set; training needs more. The row stays empty until a run.
Plate · Half two: the ladder
A fixed bench shows cost per correct. It does not show that a practice improved.
One
A fixed bench shows cost per correct. It does not show that a practice improved anything. In a randomized trial developers forecast 24% faster and measured 19% longer, with 16 of them in the study.
Gains inside the frontier can hide losses outside it: 25.1% faster inside, about 19 points less often correct outside (Dell'Acqua et al.). Keep a case that sits outside.
Becker et al. (METR), arXiv 2507.09089, 2025 (PP): 16 developers, forecast 24% faster, measured 19% longer.
Dell'Acqua et al., Organization Science, 2026 (PR): 25.1% faster inside the frontier, about 19 points less often correct outside.
Two
Run the forty again on each release. Today one column is filled, the September 2026 raw recording, at $0.324 per correct on Sonnet and $0.0276 on the mini; the typed view, recorded in October 2026, is $0.021 and $0.004. [ASK: talk date]
Where I might be wrong: forty tasks and one run per task, no repeated runs, so the models differ on one task and the interval spans about 14 to 26 of 40. The string-contains scoring is checked gauge against gauge only (kappa 0.51 Sonnet, 0.60 mini), never against a person.
Measured values: data/bench.json (MEASURED).