The Hunt for Leverage

In Go Architecture for LLM-Maintained Codebases I explored the idea that the right structural choices can make wrong code fail to compile rather than fail code review. That article focused on type systems, generated code, and compiler-enforced contracts. What I hadn’t thought through was the test layer. Compilers catch type errors. Linters catch resource leaks. But what catches a test that looks correct and asserts nothing?

A colleague’s work on GitLab’s Knowledge Graph project recently showed me what pushing back at the test layer looks like in practice. The framework tests a graph query engine that compiles a DSL into ClickHouse SQL. Pure function: query in, results out. When a developer or agent writes a test, the framework reads the compiled query AST and derives a set of typed requirements. A query with order_by requires a call to assert_node_order. Queries with filters require assert_filter for each filtered field. Traversals with specific edge types require asserting those edges individually.

Miss one, and the test panics on drop. What caught my attention was the shape of the panic message:

unsatisfied assertion requirements:
  - OrderBy (call assert_node_order)
  - Filter on 'state' (call assert_filter for 'state')

That’s not reporting what went wrong. It’s naming what to do next.

What makes this work is Rust’s Drop trait. When a ResponseView goes out of scope, its destructor runs and checks whether all derived requirements were satisfied. If not, it panics with the checklist of what’s missing. This is the drop bomb pattern: a value that explodes on destruction unless you’ve done the right thing with it. The compiler guarantees the destructor runs, so the enforcement can’t be skipped, forgotten, or worked around. The test author (human or agent) either satisfies every requirement or gets a prescriptive panic naming exactly what they missed.

I find this genuinely elegant. Most languages require you to remember to call a verification step at the end of your test. Rust lets you build it into the type itself: the act of dropping the value is the verification. The enforcement is structural, not procedural. You can’t opt out because you can’t prevent a value from being dropped.

Go doesn’t have Drop, which is the gap I keep thinking about. The closest idiom is testing.Cleanup: you register a function that runs when the test finishes, and that function checks whether all requirements were satisfied. A test helper can register it automatically when it hands you the tracker, but nothing in the type system prevents constructing a tracker without registering the cleanup. In Rust the enforcement is inescapable. In Go it’s a convention that the helper establishes. I don’t know yet whether that’s good enough for LLM-generated tests, but it’s what lever is exploring.

Error messages as prompts

This is the thing I keep coming back to. When an LLM agent runs cargo test and gets a standard compiler error like “type mismatch on line 42,” it has to interpret what that means, reason about the code structure, and decide on a remediation. That’s a reasoning step the model might get wrong. When the same agent gets “call assert_node_order,” there is no interpretation needed. The error message is the prompt.

Every error message in the system is, in effect, a prompt to the next iteration of the agent’s repair loop. If you accept that framing, then designing test infrastructure is a form of prompt engineering: not in the sense of writing natural language instructions, but in the sense of shaping the signal that guides an autonomous process toward correctness. The quality of that signal determines whether the agent converges in one iteration or drifts for ten.

This has implications beyond testing. Every make check failure, every CI error, every linter message is a prompt to whatever process reads it next. If that process is increasingly an LLM agent, then parts of the developer toolchain start functioning as prompts. The ones that name the next action converge faster than the ones that describe the problem.

Three enforcement layers

The Knowledge Graph framework uses three layers, each catching a different class of vacuous test:

Layer 1: Specification-derived requirements. The framework reads the compiled query AST and produces a set of typed requirements. An assertion tracker records which are satisfied. On drop, it panics with the remaining list, naming the method to call for each.

Layer 2: Inspection enforcement. Methods like node_ids() return a MustInspect<T> wrapper that panics on drop if the caller never accesses the inner value. This catches the pattern where code calls a method to satisfy the tracker but ignores the result. You can’t just call the function and move on. You have to read what it returned.

Layer 3: Trivial predicate detection. assert_node("User", 1, |n| ...) constructs a blank node (same type and ID, no properties) and tests the predicate against it. If the predicate passes for the blank node, the framework rejects it. This catches |_| true and any check that doesn’t inspect a specific property. The test has to prove it’s checking something real.

A test can satisfy all three layers and still miss a real bug. That’s worth stating clearly. The framework catches structural vacuity, not semantic correctness. But the class of vacuity it eliminates is precisely the class that LLMs are most prone to producing: plausible-looking tests that assert nothing meaningful.

Three principles that generalize

Looking at this framework alongside the Go architecture patterns I’ve been using, I notice three principles that keep appearing across different projects and languages.

Derive obligations from the specification

In the Knowledge Graph framework, the query AST determines which assertions are needed. In a sqlc pipeline, the SQL query determines the Go function signature. In a huma API, the struct determines the OpenAPI spec. In synthesist, the spec determines the task requirements. The common thread: the system computes the obligations, so neither the developer nor the agent needs to remember them.

Make failure prescriptive

When the system rejects something, it should name the remediation. “Call assert_node_order” is something an agent can act on immediately. “Unsatisfied requirement” requires interpretation. The difference is the number of reasoning steps between error and fix: prescriptive failure reduces it to zero.

Bundle enforcement with the value

You can’t get a ResponseView without also getting the assertion tracker. You can’t get a sqlc-generated function without writing valid SQL. The enforcement isn’t a separate step you opt into. It’s built into the API surface. This matters for agents because they follow the path of least resistance. If the enforcement is opt-in, the agent will opt out.

I’ve been looking at how to apply these principles across all the projects I’m working on. nomograph/lever is where I’m collecting that exploration: what do reusable enforcement primitives look like across languages?

A gap that taught us something

We ran a small pilot: 3 test targets across different query types, 2 prompt conditions, 6 total completions using Claude Opus 4. All 6 compiled and satisfied all requirements.

Both prompt conditions missed the ASSIGNED edge in a neighbors query. The LLM found MEMBER_OF and AUTHORED but skipped ASSIGNED. The root cause: asserting any single edge type satisfies the generic Neighbors requirement, so other types go unchecked. The framework catches vacuity but not coverage incompleteness.

We contributed a fix: an opt-in assert_all_edge_types_covered() method that tracks which edge types have been asserted and compares against what’s in the response (MR !778). It follows the same prescriptive pattern on failure: if you miss an edge type, it names which one.

The framework didn’t just help us write correct tests. It helped us find where the framework itself was incomplete, by observing what LLMs consistently got wrong. The agent’s failure pattern was a signal about the infrastructure, not just about the agent. That inverts the usual relationship: instead of the framework testing the code, the agent’s behavior tests the framework.

What I’m watching

I don’t know yet whether prescriptive failure generalizes to test domains beyond query engines and compiled DSLs. The Knowledge Graph team built this to solve a practical problem, and the same design choices that make testing tractable for a small team happen to help with LLM-generated tests.

I’m curious whether the three principles (derive obligations, prescriptive failure, bundled enforcement) can be expressed as library patterns that other projects adopt. The Rust type system makes MustInspect and drop-based enforcement natural. I’m less sure what the equivalent looks like in Go or Python. That’s the question lever is exploring, and it’s the question I’m sitting with.