Talk four of four · November 2026 STUB

A Bench of Your Own

Take an LLM step you already run and build the evaluation that tells you which rungs pay

AI in Systems Engineering · Andrew Dunn

Movement I · Half one: evaluation design STUB

Write down the step you run

A bench starts as a record of one step as it runs today: inputs, the call, the output, who checks it, what a bad output costs

[EXHIBIT PLANNED] AP1 as a workflow card, from the pre-registration. Input: the 17-file Apollo 11 SysML v2 slice, 76,647 billed tokens on Sonnet 5. Call: fixed system prompt plus the answer-schema line, one call, 4,000-token output cap. Output: three typed fields, $0.154 per call. The question: "What is the maximum acceptable vertical velocity for the lunar lander soft landing, as defined in requirement HLR-R002? Give the value and unit." The lines for who checks it and what a wrong answer costs are the room's [STUB].

Movement II · Half one: evaluation design STUB

Cases from real work, and what right means

Take cases from real traffic and real misses; give each an answer key; separate what a parser checks from what needs a person

[EXHIBIT PLANNED] AP1's key: HLR-R002 (string), 2 (float, tolerance 0.05), m/s (string). Across the forty tasks: 126 fields, 79 floats, 12 exact strings, 35 string-contains checks on free text. A parser scores all of them; no model grades.

Movement III · Half one: evaluation design STUB

The scorer is an instrument

A parser is a gauge, a judge more so: know what it credits, report both error rates, calibrate against one expert

[EXHIBIT PLANNED] On the table arm, AP1 scores 0.33 on both models by echoing the requirement ID from the question and returning null for velocity and unit. On the raw arm, Sonnet 5 averages 0.585, yet 20 of 40 are fully right. Then the 35 string-contains verdicts against one person's reading: agreement, kappa, both error rates [STUB: not yet computed].

Movement IV · Half one: evaluation design STUB

Write the bar before the run

Date the metric, threshold, comparison and a prediction before the first call; criteria drift once you grade

[EXHIBIT PLANNED] The typed-view pre-registration, dated 2026-10-01, before any live call. Supported: at least 20 correct for at most a quarter of the slice's bill. Refuted: 10 or fewer. On Sonnet 5 the author predicted 10 to 14. Result: 13 of 40 on both models (GPT-5 mini had its own thresholds: 8 to 12 predicted, 19 or more supported, 9 or fewer refuted), $0.28 on Sonnet 5 against the slice's $6.49. Read against the criteria: "between", cheaper but not the correct path.

Movement V · Half one: evaluation design STUB

Baseline, then keep it honest

Run the step as it runs today, read transcripts, turn every new failure into a case; then the bench can price a rung

[EXHIBIT PLANNED] The raw-arm baseline. Sonnet 5: 20 of 40 for $6.49 ($0.32 per correct). GPT-5 mini: 19 of 40 for $0.53 ($0.028 per correct). The misses become the regression set. One class stands out: none of the 16 explanation tasks is correct on either model. They ask about the whole model, and the step sends only the slice. The input is at fault.

Movement VI · Half two: the ladder STUB

The meter

Price per token falls, cost per task rises, and so does the bill: a bench reads cost per task

[EXHIBIT PLANNED] Tokens and cost per task on the forty at two model tiers, with the spread across repeated runs.

Movement VII · Half two: the ladder STUB

Context to hooks

The bottom rungs change in minutes and roll back free; a hook takes the job away from the model

[EXHIBIT PLANNED] Count the sysml-bench items that a parser or validator answers exactly: what fraction needs no model at all.

Movement VIII · Half two: the ladder STUB

Routing and mixtures

Multi-model use is measured, routing adoption is not; the honest baseline is the best single model

[EXHIBIT PLANNED] Offline, from the recorded Sonnet 5 and GPT-5 mini rows: each model alone, the best-of-both ceiling, and a threshold route. Optionally sweep a Jev-style probability as the threshold.

Movement IX · Half two: the ladder STUB

Small tuned models and training

On a stable, bounded, labeled job a well-trained small model can win; training moves the cost to the teacher

[EXHIBIT PLANNED] A small tuned model against the zero-shot baseline on the forty: one run, baseline and year stated.

Closing STUB

Knowing it helped, and where I might be wrong

  • Price at fixed capability keeps falling, so the squeeze on rented frontier intelligence may not arrive.
  • Capability pull beats cost push: three vendors hold 88% of enterprise LLM API usage and open share fell from 19% to 11% (Menlo, S).
  • Tuned-small wins are mostly against zero-shot baselines.
  • The ladder itself predicts low rungs and may be what enterprises already do.

Out STUB

Knowing it helped

Name one decision in your project, the tasks you would put on your bench, and the rung you would test first. Write the bar before the run.