jig

jig is an agent-shape testing harness: it points a real agent at a real CLI inside a fixture and measures whether the commands the agent reaches for exist. The property under test is the one I worked on upstream in glab-api, a tool behaving the way a model expects a CLI to behave; jig turns that from an impression into a count for the tools under Nomograph, itself included. A subject tool declares its battery in an agent-shape.toml (TOML): fixture, agent models, turn cap, judge rubric, tuning and holdout tasks, and the subcommands the rubric claims exist.

1. Fixture setup runs before every trial; strip_env clears the variables that carry a session or token in.
2. The runtime spawns headless (Claude Code); every Bash call is read from the event stream in order, with turns, tokens and whether the run finished under the turn cap, which stays short because correction loops decay after the first few turns.
3. An LLM judge reads the transcript against the rubric and answers in strict JSON: the first command, whether it existed, completion, every invented command, whether it fell back to raw shell, and its reasoning. double_score runs it twice and records the delta.
4. Each (trial, verdict) pair appends to a JSONL checkpoint, so a killed run resumes; rejudge re-scores a checkpoint against an updated rubric without re-running the agent.
5. render and compare read saved JSON reports offline: per task and model, score, completion, tokens, turns and invented commands, and the delta between two reports.

The runner spawns one runtime; the schema is runtime-agnostic pending a configurable spawn command. Holdout tasks are a battery for an author who has not seen the tuning set. jig’s own battery stays on check, render and compare, since run and rejudge spawn an agent and cost tokens.

---