Inference on a Schedule

I wanted an activity page on this site. Something that shows what I’m building across GitHub and GitLab, what I’m listening to via ListenBrainz, how contributions distribute over the year. The data would come from five APIs, get merged into a single JSON file, and ship with the static build. The catch: some of the source data contains private project names, internal namespaces, and employer-specific paths that can’t appear in public output. The data needs judgment, not just reshaping.

The obvious first pass was to send everything to a language model. Claude gets the raw API responses, a prompt describing the privacy rules and output schema, and produces the final JSON. Clean, simple, one moving part.

It cost $0.58 per run and took two minutes. For a nightly job, that’s $17/month to update a JSON file.

That number bothered me. I currently pay $20/month for Claude handling a wide range of complex work. Spending nearly the same amount for it to update a single JSON file felt disproportionate. I started reading through the prompt to understand what the model was actually being asked to do. Most of the rules are deterministic: if a project lives in a blocked namespace, suppress it; if visibility is public and the namespace matches a personal org, categorize as Personal OSS; merge two calendar objects by summing values per date. String comparisons and arithmetic. The model was doing jq’s job at Sonnet prices.

Finding the boundary

I went through each section of the output and asked whether it genuinely needed a model or just felt like it did.

Section	What it does	Needs judgment?
calendar	Merge GitHub + GitLab daily counts	No
stats	Count events, compute percentages	No
listening	Reshape ListenBrainz response	No
categories	Classify projects by namespace + visibility	Mostly no
projects.nodes	Group events by project, apply privacy rules	Partially
projects.links	Infer relationships between projects	Yes
now	Pick current activity highlights	Partially

Six of the seven sections turned out to be fully or mostly mechanical. Calendar merging is addition. Stats are counts. Listening data is extracting fields from a nested JSON response. Even categorization is namespace prefix matching for every project that exists right now.

The tempting move was to rewrite the whole thing in Python and eliminate the model entirely. That would work today. What gave me pause was maintenance: every new namespace, every new organization I contribute to, every new source of activity data would require updating a lookup table. A model handles novel inputs gracefully because it reads the rules and applies them. Python handles novel inputs by falling into a default case and silently producing wrong output.

The question isn’t “can a model do this?” or “can Python do this?” It’s the same question from The Hunt for Leverage: where does the boundary between deterministic and inferential work actually fall?

The hybrid

I split the pipeline into three phases. Phase 1 is Python: merge calendars, count events, reshape ListenBrainz data, and build a slim project summary. It runs in under a second. No inference, no cost. Phase 2 sends only the project summary to Claude Haiku: 9KB instead of 166KB. The model categorizes projects, generates the relationship graph, and picks the “now” highlights. Phase 3 merges the two outputs, validates the result (blocked namespaces, required keys, valid JSON), and commits to the repo if anything changed.

The numbers shifted more than I expected:

	Full LLM	Hybrid
Input tokens	~42,000	3,774
Output tokens	~5,000	2,019
Model	Sonnet	Haiku
API time	96s	14s
Total runtime	128s	50s
Cost per run	~$0.58	~$0.005
Monthly (daily)	~$17.40	~$0.15

The cost reduction is dramatic, but I think the more interesting result is architectural. Each phase does what it’s suited for. Python doesn’t hallucinate calendar math. The model doesn’t spend context window on arithmetic, freeing its attention for the judgment calls where it actually adds something: is this new namespace professional or personal? Should these two projects be linked?

What the model is actually good at here

I keep coming back to the distinction between intelligence and flexibility. The categorization rules could be a Python dictionary today, but the model absorbs new projects without code changes. A new namespace appears in the source data, and the model categorizes it on the next run because the prompt describes the principles, not the cases. A lookup table would silently dump it into the default bucket, and I wouldn’t notice until I happened to check the page.

The project relationship graph is where inference seems to genuinely earn its cost. “These projects share a namespace” is mechanical. “This project is upstream of that one” requires reading the namespace structure and understanding what gitlab-org/ means in context. The model generates relationship edges from namespace patterns and activity co-occurrence. I could hardcode those edges, but they’d go stale the moment my work shifts. Whether Haiku is getting these relationships right every time is something I’m still evaluating.

Running it

The pipeline runs as a GitLab CI scheduled job, triggered nightly at 3 AM Eastern. The job fetches from five APIs, runs both phases, and pushes the updated JSON back to the repo. That push triggers a normal build pipeline that deploys the site with fresh data.

A CI variable CLAUDE_MODEL lets me swap models without a code change. If Haiku’s categorization quality drops, I can point it at Sonnet for ~$0.08/run instead. The architecture doesn’t care where the judgment comes from, only that it arrives as JSON matching the schema.

The broader pattern

Static sites have a reputation for being, well, static. But the build step is a function: inputs in, HTML out. There’s no rule that says the inputs have to be checked into the repo by a human. A scheduled pipeline can fetch external data, apply whatever mix of deterministic and inferential processing makes sense, commit the result, and trigger a deploy. The site stays static (fast, cheap, cacheable) while the data stays fresh.

The design question I find interesting is where to draw the line between conventional compute and inference. Send too much to the model and you’re paying for arithmetic. Send too little and you’re maintaining brittle lookup tables that break on novel inputs. It’s almost never “all model” or “no model.” For this pipeline it turned out to be about 5% of the input data and 3 of 7 output sections. I suspect most LLM-in-CI use cases have a similar ratio, and I suspect most of them haven’t looked.

The next thing I want to try is running an ephemeral model inside the CI runner itself. The Gemma family is interesting here: small enough to load in a container job, capable enough for structured categorization tasks. The project summary is under 4,000 tokens, well within range for a quantized model. If that works, the inference cost drops to zero and the only question is whether CI runner compute time is cheaper than the API call. The architecture already supports the swap (point the endpoint at localhost instead of Anthropic). Whether a 9B model matches Haiku’s judgment on genuinely novel inputs is the open question, and the one I’m most curious about.