Skip to content
andrew.dunn.dev

varve

Source

A varve is one annual sediment layer in a glacial lake. varve is that instrument pointed at an agent estate: a local lake of readable layers of everything the harness does. It normalizes the session records Claude Code, pi and opencode write to disk into Parquet partitions and queries them with DuckDB. No resident process, no listener, no egress. It is the transcript half of the-other-half-of-the-corpus, and the corpus the steering ladder in the-flywheel-and-the-meter takes its evidence from.

SESSION RECORDSTHE READINGSnever in the repositorypublic: adapters, queries, schema, docsClaude CodeJSONLpiJSONLopencodeSQLitePER RUNTIMEadaptersthe only writersPARQUETthe lakecanonicalDUCKDBviewsrebuildableVERSIONED SQLquery libraryread only
Session records to lake to query library
  • Adapters, one SQL file per runtime, are the only writers; JSONL is read in place and the opencode SQLite through a read-only clone, so no live store is written or locked.
  • Ingest is incremental, keyed on a manifest of path, size and mtime; backfill clears the lake and rebuilds it whole, and the DuckDB views beside it rebuild identically.
  • The query library opens the lake read-only, one file per question with its metric in lake columns: burn, cache hit, dispatch ratio, session shape, counterfactual re-pricing.

The lake serves a ladder of questions: efficiency wins from history; whether harness tuning is adequate, meaning a correction lands once and holds rather than recurring across sessions; and whether context engineering has saturated, the evidence for moving up to adapter training. The last two are the correction-mining loop.

LAKEmessagesapi_calls, tool_callsMINEcandidatesprecision-tuned detectorsOPERATOR DATAadjudicationsoutside the repositoryEVENTScorrection_eventsreserved durability blockRULEBOOK GIT LOGinterventionsbake date per ruleevents hold local pointers,never the content
Correction mining and the durability test
  • mine derives correction_candidates from the lake alone, detectors tuned for precision, and operator_messages as metadata only for the recall floor.
  • The operator adjudicates a sample; the ledger stays outside the repository, and confirmed candidates promote to correction_events.
  • interventions recovers bake dates from a rulebook’s own git history, read-only, no new capture surface.
  • The event schema reserves a durability block (intervention_ref, recurred_after_bake) for the durability test: a correction recurring after the rule meant to settle it would mark a rung that did not hold. The interventions recipe already recovers the bake dates that test needs.

Scope is one machine and one operator; a team stage swaps the engine, and Parquet and the SQL library port as they are.