andrew.dunn.dev

varve

~3 min
duckdbparquetsqlai-harnessanalyticsobservability
Source

A varve is one annual sediment layer in a glacial lake, and reading a sequence of them reconstructs a chronology. varve is that instrument pointed at an agent estate: a local lake that accumulates readable layers of everything the harness does. It normalizes on-disk session records from Claude Code, pi, and opencode into Parquet partitions and queries them with DuckDB. Zero resident processes, no listeners, zero egress.

SESSION RECORDSTHE READINGSnever in the repositorypublic: adapters, queries, schema, docsClaude CodeJSONLpiJSONLopencodeSQLitePER RUNTIMEadaptersthe only writersPARQUETthe lakecanonicalDUCKDBviewsrebuildableVERSIONED SQLquery libraryread only
Session records from three agent runtimes pass through per-runtime SQL adapters into a Parquet lake that never enters the repository, and a versioned query library reads it back through rebuildable DuckDB views.

The design is deliberately plain: no server, no daemon, no vector database, just files and a binary. Adapters are per-runtime and close to pure SQL, and they are the only writers; analysis opens the lake read-only. A full backfill of years of sessions takes a few seconds. Lake data never lives in the repository, landing instead at ~/.local/share/varve/lake/ as Parquet partitions with a rebuildable DuckDB file beside it.

Highlights

  • Reconciling its cost numbers against an independent accounting tool agreed within 0.2%, which is most of the reason to trust the rest of what it reports.
  • The versioned SQL query library is the real analytics artifact; the engine underneath it is treated as disposable.
  • Goal ladder from the PRD: efficiency wins from history, then harness-tuning adequacy (do corrections land once and hold, or is the same decision being re-litigated), then whether context engineering has saturated.
  • Correction mining and cost analytics are live; adapters are validated to differing depths and each documents its own status, with the pi adapter the least exercised.
  • CI backfills against a synthetic three-runtime corpus in tests/fixtures/, so the gate runs on fixtures rather than on anyone’s real sessions.
  • Architecture decisions are dated and carry their evidence in docs/adr/, with open designs kept separately in docs/rfc/.
  • mise install provisions the pinned DuckDB and just; every recipe assumes both on PATH.
  • MIT licensed.

Why this exists, what the first question put to the lake actually answered, and the two-layer argument it came out of are in The Other Half of the Corpus.