Talk four of four · November 2026 [STUB]
Take an LLM step you already run and build the evaluation that tells you which rungs pay
AI in Systems Engineering · Andrew Dunn
Renting frontier intelligence costs more per task even as tokens get cheaper, so enterprises will mix models and send bounded jobs to small tuned open ones. Which rung pays for you is a question only your own bench answers. Half of this talk is how to build that bench; half is the ladder it reads.
Two claims held open: rented frontier cost pushes enterprises to mix and route (evidence mixed), and a well-trained small open model can win a bounded job (supported, narrowly).
Half one: evaluation design · Half two: the ladder
Grades: PR peer-reviewed · PP preprint · V vendor · B blog or press · S survey or analyst.
Movement I · Half one: evaluation design [STUB]
A bench starts as a record of one step as it runs today: inputs, the call, the output, who checks it, what a bad output costs
The claim
A bench starts as a record of one step as it runs today: the inputs, the call, the output, who checks it, and what a bad output costs. A leaderboard measures someone else's step, and stops measuring once it is gamed.
Say it this way: Your benchmark is the step you already run, written down.
Best sources
PRZheng et al., ICLR 2025. A constant-output "null model" scores 86.5% LC win rate on AlpacaEval 2.0 (83.0 on Arena-Hard-Auto, 9.55 on MT-Bench).
PRGSM1k, NeurIPS 2024. A fresh human-written twin of GSM8k: "accuracy drops of up to 8%". Quote 8%, not 13%.
Keep off: GSM1k "13%" (a search summary; the abstract says 8%); "GPT-4 agrees with humans over 80%" unqualified.
Where I might be wrong
A card written from memory records the step as you think it runs. The running example is the Representation lesson's raw arm, one prompt and one model call, which is the step this audience runs: summarize a change request, classify a requirement, extract datasheet values, draft a trace link. A bench of your own step is also a bench of your blind spots.
Open questions for Andrew
Movement II · Half one: evaluation design [STUB]
Take cases from real traffic and real misses; give each an answer key; separate what a parser checks from what needs a person
The claim
Take cases from real traffic and real misses, not invented ones. Give each an answer key, and separate what a parser can check from what needs a person.
Say it this way: Start from real failures, and write down what right looks like.
Best sources
VAnthropic, Jan 2026. "20-50 simple tasks drawn from real failures is a great start."
BHusain, 2024. Run unit-test style checks before reaching for a model judge.
Keep off: Any claim that twenty to fifty is a statistically adequate size.
Where I might be wrong
Said aloud: sysml-bench was written against a real model, not taken from traffic, so it shows the method and not the sampling. Anthropic's line is vendor advice, not a measured threshold.
Open questions for Andrew
Movement III · Half one: evaluation design [STUB]
A parser is a gauge, a judge more so: know what it credits, report both error rates, calibrate against one expert
The claim
A parser scorer is a gauge, and a judge more so. Know what it credits, report both error rates, and calibrate it against one expert's reading.
Say it this way: Agreement is a number that lies. Calibrate the gauge before you trust it.
Best sources
PPNorman et al., 2026. 21 judges, about 541,000 judgments. "A judge reporting '85% agreement' on MT-Bench has kappa≈0.48."
BHusain, 2024. Judge as a binary classifier: report both rates. If 5% of cases fail, an always-pass judge scores 95% agreement.
Keep off: "GPT-4 agrees with humans over 80%" unqualified (a no-ties first-turn figure; with ties 66% vs 63%); Husain's "90%+ agreement" claims (self-reported).
Where I might be wrong
One expert is one person's opinion, and kappa depends on the base rate. Calibration moves the trust from the scorer to the expert; it does not remove it.
Open questions for Andrew
Movement IV · Half one: evaluation design [STUB]
Date the metric, threshold, comparison and a prediction before the first call; criteria drift once you grade
The claim
Date the metric, the threshold, the comparison and a prediction before the first call, because criteria drift once you grade outputs. Treat an evaluation as an experiment and report uncertainty, not a single score.
Say it this way: Write the bar before the run; criteria drift once you start grading.
Best sources
PPShankar et al., 2024. "Criteria drift": people define their criteria by grading outputs. Abstract only.
PPMiller, 2024. "Fundamentally, evaluations are experiments"; report standard errors. Abstract only.
PRBean et al., NeurIPS 2025. Of 445 benchmarks, 16.0% used any uncertainty estimate or statistical test.
Keep off: Any claim that pre-registration reduces evaluation over-optimism; pre-registration as established.
Where I might be wrong
Pre-registration is a proposal, not a proven method: I found only position papers claiming it reduces over-optimism. Declaring the bar first is my inference, not a tested practice.
Open questions for Andrew
Movement V · Half one: evaluation design [STUB]
Run the step as it runs today, read transcripts, turn every new failure into a case; then the bench can price a rung
The claim
Run the baseline as the step runs today. Keep twenty to fifty cases, have a person read transcripts, and turn every new failure into a case. Then the bench can price a rung.
Say it this way: Every miss is a case.
Best sources
VAnthropic, Jan 2026. Calibrate LLM judges to human experts; read transcripts; "20-50 simple tasks drawn from real failures is a great start."
BMETR, 2025. o3 hacked the reward on 21/21 runs of Optimize LLM Foundry (39/128 on RE-Bench, 8/1,087 on HCAST). METR says its detection likely undercounts.
Keep off: The METR counts as a general rate (they are per-task); NVIDIA's 8B router (one deployment, vendor authors) as proof the loop works.
Where I might be wrong
Twenty to fifty tasks give wide intervals. Small private sets go stale and can be learned by whoever tunes against them. The explanation-task diagnosis is one reading of the transcripts, and it needs a person to confirm it.
Open questions for Andrew
Hinge [STUB]
You now hold a bench with a baseline row (correct of N, bill, cost per correct) and a bar written before the run. In half two each rung is another row on that bench, and pays only when it moves your row past your bar.
Movement VI · Half two: the ladder [STUB]
Price per token falls, cost per task rises, and so does the bill: a bench reads cost per task
The claim
Keep three numbers apart. Price per token at fixed capability falls. Cost per task at the frontier rises, through tokens. The bill rises. A bench reads cost per task, which is the number a rung changes.
Say it this way: Price per token at fixed capability falls; cost per task at the frontier and the total bill rise.
Best sources
PPGundlach et al., 2026. Price at fixed capability falls "around 5x to 10x" per year; frontier run price rises 3x to 18x per year (benchmark-run cost; the 3x algorithmic gain excludes open models).
PPBai et al., 2026. Agentic coding uses about 1000x the tokens of chat; same-task runs differ up to 30x. Coding only (SWE-bench Verified).
PPChen et al., 2026. In 32% of model pairs the cheaper-listed model cost more, up to 28x (8 models, 12 tasks). Lead with "a third".
Keep off: Uber as the "entire AI budget"; Menlo "tripled" or "3.2x" (the 2024 figure excludes inference); Anthropic margins as a price-rise story; list-price trackers.
Where I might be wrong
Price at constant capability keeps falling, so the squeeze may not arrive. Epoch (2025, B) reports 9x to 900x a year at fixed capability, persistence "less clear". Prompt caching changed the baseline: 88.6% API cost cut on one developer's workload (Peng, 2026, PP, against nominal uncached rates).
Open questions for Andrew
Movement VII · Half two: the ladder [STUB]
The bottom rungs change in minutes and roll back free; a hook takes the job away from the model
The claim
The bottom rungs (context, system prompts, skills, tools) change in minutes and roll back for free. A deterministic hook goes further and takes the job away from the model: zero tokens, zero variance. Start there, and let the bench say when it is not enough.
Say it this way: Some of the work never needed a model.
Best sources
V/SLangChain, 2026. Label "agent builders", not enterprises: n=1340 self-selected, 63% tech firms; 57% do not fine-tune.
PPBai et al., 2026. Accuracy "often peaks at intermediate cost": more spend is not more correct.
PRAlphaGeometry, Nature 2024. Solved 25 of 30 vs symbolic alone 14: a checker the model cannot talk round.
Keep off: LangChain as an enterprise sample; Gartner's fivefold forecast as measured adoption; "cannot hallucinate" claims.
Where I might be wrong
The fraction may be small on our bench. The low rungs are also what enterprises already do, so the data confirms the order of the ladder, not a cost-forced shift. Surveys here are vendor-run and self-selected.
Open questions for Andrew
Movement VIII · Half two: the ladder [STUB]
Multi-model use is measured, routing adoption is not; the honest baseline is the best single model
The claim
The portfolio already exists, so you may have a routing problem without a router. Multi-model use is measured; routing adoption is not found anywhere. The honest baseline is the best single model, and a route is a hypothesis that may fail to beat it. A System One model (Jev, TypeSafe AI [CONFIRM]) is one candidate router signal: fast, typed, calibrated, vendor numbers labelled vendor.
Say it this way: Multi-model use is measured; routing is the horizon claim, not the measured one.
Best sources
PPRouterBench, 2024. KNN/MLP routers "do not significantly outperform the baseline Zero router" (the convex hull of single models).
PPLLMRouterBench, 2026. Several routers "fail to reliably outperform a simple baseline"; the best gain up to 4% accuracy or 31.7% cost reduction at matched performance.
Sa16z, 2026. 81% use three or more model families (68% prior), n=100, "not randomly sampled", investors in OpenAI.
V/BDataCamp, 2026. Jev, vendor-run table: 67.8% at $0.0004 and 0.4s vs GPT-5.6 Sol 74.1% at $0.0836 and 23.3s. No independent reproduction; architecture undisclosed.
Keep off: RouteLLM "26%" (unconfirmed); "routing is happening" as a measured fact; Jev as "cannot hallucinate" or "40x-200x"; Not Diamond and Martian claims.
Where I might be wrong
Routing adoption is not measured; surveys credit fit and capacity over cost; only 11% switched providers (Menlo, 2025, S). Peng (2026, PP, one developer) found hybrid routing trades defect rate for infrastructure savings along a cost-quality frontier, rather than dominating the pure-API baseline. Every Jev number is vendor-run.
Open questions for Andrew
Movement IX · Half two: the ladder [STUB]
On a stable, bounded, labeled job a well-trained small model can win; training moves the cost to the teacher
The claim
On a stable, bounded, labeled job, a well-trained small open model can beat a prompted frontier model, and curation does the work. It is a lever for format, style and narrow skill; knowledge stays with pretraining. Training does not escape the meter: it moves the cost to the teacher.
Say it this way: A small model trained well on a bounded task can beat a prompted frontier model on that task.
Best sources
PPChristou and Tsoumakas, 2026. Tuned Qwen2.5-0.5B F1 0.83 vs GPT-5.4 0.69 and Sonnet 4.6 0.66 zero-shot; an in-domain RoBERTa also beat both. 0.83 is the best of 30 configurations.
PPCelikik et al., Oct 2026. 0.8B tuned beats prompted GPT-5.4; the teacher is GPT-5.4; 16x is a modelled cost of grammar detection only; curation added +36.9 to +40.7 L3 recall points over raw teacher labels.
PRJeong et al., EMNLP 2024. Seven medical LLMs vs base models, 3-shot: better in 12.1% of cases, tied 49.8%, worse in 38.2%.
Keep off: LIMA as a small-model win; the Li 95.36 lead stat; LoRA Land as typical rather than best case; DeepSeek "545%"; Cast AI 5% GPU utilization.
Where I might be wrong
Wins are mostly against zero-shot or old baselines; the missing comparison is a tuned prompt, retrieval or a cheaper frontier tier. New facts are learned slowly in fine-tuning and raise hallucination (Gekhman, EMNLP 2024, PR) [CONFIRM]. RL post-training is "bounded by the base model" (Yue, NeurIPS 2025, PR), contested by ProRL (unverified). OpenAI ends self-serve fine-tuning in January 2027 (Tessl, B).
Open questions for Andrew
Closing [STUB]
A fixed bench shows cost per correct. It does not show that a practice improved.
PPBecker et al. (METR), 2025. Forecast 24% faster, measured "19% longer" (16 developers, early-2025 tools).
PRDell'Acqua et al., Organization Science 2026. +25.1% speed inside the frontier; about 19 percentage points less correct outside it (body; the abstract says "19% less likely").
[EXHIBIT PLANNED] Re-run the forty on each new release and plot cost per correct over time.