Talk four of four DRAFT

A Bench of Your Own

Take an LLM step you already run and build the evaluation that tells you which rungs pay

AI in Systems Engineering · Andrew Dunn

Movement I · Half one: evaluation design DRAFT

The AP1 workflow card

Your benchmark is the step you already run, written down: AP1 on the raw arm, one prompt, one call.

Movement II · Half one: evaluation design DRAFT

AP1's key and the 126-field strip

Cases come from real traffic and real misses, not invented ones.

Movement III · Half one: evaluation design DRAFT

The gauge: echo cells and dropped fields

A parser is a gauge: know what it credits.

Movement IV · Half one: evaluation design DRAFT

The bar on file, result held

A bar has four parts, dated before the first call: metric, threshold, comparison, prediction.

Movement V · Half one: evaluation design DRAFT

The baseline row with Wilson brackets

Run the step as it runs today: Sonnet 5 gets 20 of 40 for $6.49, GPT-5 mini 19 of 40 for $0.525.

Movement V · Half one: evaluation design DRAFT

Miss causes become the regression set

Every miss is a case. Two causes explain 12 of 16 explanation misses: the answer is not in the slice, or the reply hit the 4,000-token cap before any JSON.

The bench · The hinge DRAFT

The bench, four rung rows empty

You hold a bench: card, key, gauge, bar, baseline row.

Movement VI · Half two: the ladder DRAFT

The meter: two cost-per-task strips

Price at a fixed capability falls. What we run at the frontier costs more each year, because we spend more tokens. A bench reads cost per task.

Movement VII · Half two: the ladder DRAFT

Context and hooks

The bottom rungs change in minutes and roll back for free.

Movement VIII · Half two: the ladder DRAFT

Routing: hull, ceiling, escalation

The portfolio already exists. Multi-model use is measured; routing adoption is not.

Movement IX · Half two: the ladder DRAFT

Small tuned models, teachers named

On a stable, bounded, labelled job a small tuned model can win.

Knowing it helped, and where I might be wrong · Half two: the ladder DRAFT

Knowing it helped: the release ledger and the room exercise

A fixed bench shows cost per correct, not that a practice improved: developers forecast 24% faster and measured 19% longer.