andrew.dunn.dev

sysml-bench

~3 min
sysmlaibenchmarkingpythonpre-registration
Source Results Dataset Pre-registration

sysml-bench is an evaluation harness for one question: does giving a tool-augmented language model more tools help it answer questions about a SysML v2 model. A Python package drives models over public SysML v2 corpora through tool sets called as command-line tools, with the search and graph tools backed by sysml, the knowledge-graph CLI that parses the corpus.

claimsFED INcorpora, taskspublic SysML v2 modelstyped answer schemasHARNESSthe runtool sets via CLIN runs per conditionGRADINGtyped scorerper field, againstthe published schemaCOMES OUTresultsresults repo, datasetdefects loggedGATEpre-registrationhypothesis, direction, snapshot and analysisfrozen before the confirmatory run
Corpora and tasks through harness to published results
  • A primary corpus carries every task category; a replication corpus carries its own confirmatory suites; the others carry discovery tasks for scaling and generalization.
  • Each task is a question with a typed answer schema, published with the dataset.
  • A condition is one model and one tool set, run N times.

The independent variable is how much of the graph the model may see.

addsaddsaddsBASELINEcli_searchsysml_searchread_fileGRAPHcli_graphtrace, check,query, inspectVIEWScli_renderrender: a reportbuilt server-sideFULLcli_fullstat: model healthplan: question to commandsCROSSED WITHa guided system prompt mapping task kinds to tools • vector search over the corpusno-tool and retrieval baselines • a turn budget per task
Four tool sets, each adding to the last
  • cli_search: find an element, open its source.
  • cli_graph adds traversal: trace chains, check structure and metamodel conformance, query by relationship, inspect an element.
  • cli_render adds a report rendered from the graph, so an explanation comes from a view, not hand assembly.
  • cli_full adds a model-health summary and a planner from question to commands. Each set grows the tool schema the model must carry before it reads any of the corpus, so the sets are ordered by that up-front cost as well as by capability.

Scoring, in eval/scoring.py, is per field and typed: booleans, floats within a tolerance, strings with qualified-name matching, lists by set F1 against a threshold. A task scores the mean of its fields, a condition the mean of its tasks over N runs.

Replication on an independent corpus runs under pre-registration.

thenyieldsFROZEN FIRSTthe registrationone hypothesis and its directionmodel snapshot, task minimumsanalysis script, in the repoTHEN RUNthe analysispaired one-sided permutation testHedges g, BCa bootstrap CIalpha 0.025, no task excludedVERDICTeither wayconfirmed, or a failureto replicate, foregroundedno amendment once scored
Pre-registered replication from freeze to verdict
  • The pre-registration commits one directional hypothesis, the model snapshot, minimum task counts and the analysis script before the first confirmatory run.
  • Inference is an exact one-sided paired permutation test with Hedges g and a BCa bootstrap interval at α = 0.025; no task is excluded.
  • The confirmatory suite matches frozen values mechanically, so the claim carries no scorer discretion.

The harness measures one modeling language on public models far smaller than a production system model, and the list scorer takes flat lists only.