Talk four of four DRAFT
Take an LLM step you already run and build the evaluation that tells you which rungs pay
AI in Systems Engineering · Andrew Dunn
Movement I · Half one: evaluation design DRAFT
Your benchmark is the step you already run, written down: AP1 on the raw arm, one prompt, one call.
Movement II · Half one: evaluation design DRAFT
Cases come from real traffic and real misses, not invented ones.
Movement III · Half one: evaluation design DRAFT
A parser is a gauge: know what it credits.
Movement IV · Half one: evaluation design DRAFT
A bar has four parts, dated before the first call: metric, threshold, comparison, prediction.
Movement V · Half one: evaluation design DRAFT
Run the step as it runs today: Sonnet 5 gets 20 of 40 for $6.49, GPT-5 mini 19 of 40 for $0.525.
Movement V · Half one: evaluation design DRAFT
Every miss is a case. Two causes explain 12 of 16 explanation misses: the answer is not in the slice, or the reply hit the 4,000-token cap before any JSON.
The bench · The hinge DRAFT
You hold a bench: card, key, gauge, bar, baseline row.
Movement VI · Half two: the ladder DRAFT
Price at a fixed capability falls. What we run at the frontier costs more each year, because we spend more tokens. A bench reads cost per task.
Movement VII · Half two: the ladder DRAFT
The bottom rungs change in minutes and roll back for free.
Movement VIII · Half two: the ladder DRAFT
The portfolio already exists. Multi-model use is measured; routing adoption is not.
Movement IX · Half two: the ladder DRAFT
On a stable, bounded, labelled job a small tuned model can win.
Knowing it helped, and where I might be wrong · Half two: the ladder DRAFT
A fixed bench shows cost per correct, not that a practice improved: developers forecast 24% faster and measured 19% longer.