Talk two of three · October 2026
AI in engineering practice: meta-models, graphs, retrieval, and the evaluation loop
This talk lands in the week of 19 October 2026.
AI in Systems Engineering · week 8 · Andrew Dunn
Scroll to run it Read on: the plates are printed at their finished state
Plate I
A model pasted into a chat as prose loses the structure the work depends on. The same model as a graph keeps it.
One
An engineering model is parts, requirements, and the connections between them.
The rules for what counts as a part, what counts as a requirement, and what may connect to what are the meta-model: the model behind the model. Every tool in the room enforces them, and the work depends on them holding.
Two
Paste that model into a chat window and it arrives as prose.
The parts become nouns. The connections become sentences. A reader can follow them. A program cannot, so nothing downstream can check that the valve on line four is the valve in the requirement on line eleven.
Three
A knowledge graph is the same model written so a program can walk it.
Elements become nodes, relationships become edges, and each one keeps its own name. Nothing is added here that the model did not already say. The difference is that the links are now things a machine can follow, one at a time.
Plate II
Pasting more of the model into the context window buys correctness, and costs more than it buys.
One
A context window is everything the language model can see at once.
Your question, whatever was pasted in with it, and what it has already said. It is counted in tokens, the pieces of text a language model is billed by. Context stuffing is pasting the source text in whole, with nothing left out and nothing condensed, and letting it read the lot.
Two
Three stops were measured on the same forty questions.
Forty tasks on the deck built for the Apollo 11 SysML model this course uses. A view alone 0.17 and the raw slice 0.59 on Sonnet 5; the whole model 0.68 on GPT-5 mini.
A view of the model alone answers 0.17 of them. Seventeen files of source text, pasted in raw, answer 0.59. That jump is the biggest single gain anywhere in this talk.
Three
The whole model reaches 0.68, and the bill arrives with every question.
It is 85,554 tokens of source text, fifty to a hundred times what a view costs, and it is paid again each time somebody asks something. Nine hundredths of correctness for that is the trade on the plate.
a view: 0.17 correct, the baseline: one view of it.
Plate III
Retrieval reads the structure: about half the questions come back right, for a tenth of the tokens.
One
Retrieval means fetching the part of the model a question touches, and sending only that.
An index is built over the model's own structure, over the parts and requirements and the links between them rather than over pages of text. A question arrives, the index is searched, and the neighbourhood around the answer comes back.
Two
On the same forty questions it scores 0.51, at a tenth of the whole model's tokens.
Retrieval 0.51 on Sonnet 5, forty tasks on this deck, against the whole model's 0.68 on GPT-5 mini, for a tenth of its tokens.
Half the questions come back right, for a tenth of the bill, and against 0.68 for everything that is most of the score. It is the first stop on this rail where the cost stops tracking the answer.
Three
What it fetches is one neighbourhood, and the parts outside it stay missing.
Retrieval takes one pass and stops. If the answer needed a link two steps out from where it looked, nothing in the pass goes back for it, and the answer comes back confident and short of the fact.
a view: 0.17 correct, the baseline: one view of it.
Plate IV
A model calling tools over the structure across several turns recovers what one retrieval pass cannot, and which tool it is handed decides the score.
One
Retrieval takes one pass. An agent takes as many as the question needs.
It lists the parts, reads a property, traces an edge, and only then writes the answer. Each call is chosen after the last one came back, so the second question can be one the first answer raised.
Two
The paper's card reads 0.806 against 0.521, and both arms are the same loop.
Frozen card, the Apollo suite under Sonnet 4.5. Both arms run the same agentic loop; the winning arm was handed a report generator.
Nothing about the language model changed between those two numbers. What changed is the tool it was handed. The arm that could render a report scored where the other one did not.
Three
The second card, 0.900 against 0.569, is a different corpus and a different model.
Frozen card, the Eve corpus under Sonnet 4: a command-line tool against retrieval. This deck's own agent scores 0.90, Sonnet 5, the Apollo corpus, forty tasks.
It says the same thing about tools against one retrieval pass, and it is not the same measurement. Two cards, two labels, never averaged into one. The number this deck recorded for its own agent sits under them, labelled the same way.
Plate V
Every score so far exists because the same questions were graded against a fixed answer key. Run that loop on live work and it is the article's flywheel.
One
A scorer is a program that grades an answer against a fixed key.
The questions were written once and do not change. The answers are checked the same way every time. Swap the representation, rerun the same questions, and the number that moves is the only thing we keep.
Two
Without it, every claim about what worked is an impression.
0.17, 0.59, 0.51 and 0.90 are not opinions about the four arms. They are what this loop returned, on the same forty questions and the same model, with only the representation changed between runs. The whole model's 0.68 came off those same forty questions on the smaller model, and says so wherever it is printed.
Three
Run the same loop on live work and it keeps producing evidence.
Every answer accepted, corrected or thrown out lands on disk as a graded example, and that pile is the corpus the next decision gets read out of. Mining it is not built yet, and that is the honest state of it.
Plate VI
Choosing between a view, stuffed text, retrieval and tools is a scored decision on cost per correct answer, and those four sit at the bottom of an eight-rung ladder.
One
Read the rows in cost order and stop at the first one that clears the bar the task needs.
Forty tasks on this deck. A view 0.17, retrieval 0.51 and tools 0.90 on Sonnet 5; the whole model 0.68 on GPT-5 mini.
Move the bar and the pick changes. At 0.4 correctness, retrieval is the cheapest thing that clears it. At 0.8, only the agent does, and you pay for the turns.
Two
A rung is one way of steering a language model you did not train, and there are eight of them.
Context, memory and retrieval sit on rung 1. Above them: system prompts, skills, tools and interface, deterministic hooks, model routing, adapter training and LoRA, full training. Each step up buys durability and costs more to undo.
Three
The rule is the lowest rung that durably fixes the behavior, on evidence from the corpus.
Placing this deck's four arms on rungs 1 and 4 is this talk's reading, not the article's. The bottom four rungs change in minutes and roll back for free, and nobody in this room touched the top four today.
At 0.00, the cheapest row that clears the bar is a view.