Birthday at the Theater

Alasdair turned three and we threw him a party at a local theater. Actually, the theater inside Bamboo, which is where I have a co-working space thanks to GitLab’s generous coworking policy. It was Alasdair’s first time in a theater, and a lot of the parents in our peer group hadn’t taken their kids to one yet. We liked the idea of him sharing his favorite moments from a couple digital estates and giving his peers a chance to be in a theater setting.

Originally we were going to let him pick a handful of favorite moments and hand them to the projectionist as a playlist. As we started asking him, we realized he had a wide variety of memorable things he wanted to share based on different social contexts. He knew what he wanted to show people and could explain why. This was taking the form of between five and six specific pieces of content totalling roughly 50 minutes. For the flow of a party we knew we’d need to break this into at least two discrete acts.

I thought it would be even more interesting if we could create a time capsule of sorts by recording his statements about each thing and placing them before it, so there was a rational flow of titles within each act. The challenge was that we’d need to splice together several disparate pieces of media content into something coherent. TV episodes from different shows, movie clips, phone-recorded introductions, title cards, music. We landed on a two-act structure with title cards and intermissions. (This was a private screening for a few families. We hope that mention of BBC and Disney digital estates doesn’t cause them to come for us.)

We figured all kids would like to see themselves on a big screen, so my wife went through and assembled a 100+ picture set from the last year focused on photos with his peer group. We were going to do a simple slideshow, but Alasdair has always been extremely interested in the “numbers and names” at the end of a film (the credits). So the idea came up to make him and his peer group their own credits: music, images, and something visually engaging. Alasdair likes the idea of “Numbers” or “Letters” in credits but doesn’t read yet, so we needed a way to show the photos in a high-engagement format. The idea of panning around a larger image took form, and that became a Ken Burns-style pan across vintage Route 66 maps with 100+ family photos composited as aged Polaroid snapshots, set to some thematic music.

The aesthetic

The French Dispatch was the loose inspiration, a collection of distinct stories held together by presentation style rather than narrative. White text on black, symmetrical framing. The intro cards put Alasdair on the left (portrait video from a phone) with the show title, subtitle, and his commentary on the right. He introduced each show in his own words. “Because it’s Silly. The Frog took Ducks Chair.” “He was burying things and peeking out of his holes.” “I like the little spider, because, the spider helps.” Having him introduce each piece provided two things: a logical introduction for the viewers, and an effective time capsule of what he liked at this point in his life.

The program

What emerged was a two-act structure defined in a TOML file:

Act 1
  Title card ("Alasdair's Third Birthday — Party at the Theater")
  Act card
  Alasdair intro → Sarah and Duck: Tummy Talk
  Alasdair intro → Sound Collector: Sound Detector
  Alasdair intro → Kiri and Lou: The Something Something
  Intermission (30s — "Please go paint your own cars and race")

Act 2
  Act card
  Alasdair intro → Cars on the Road: Trucks
    ("Because Mater goes upside down, good song to dance to.")
  Alasdair intro → Cars: Cruising Radiator Springs
  Dancing title card (30s — "Grab a light from the front")
  Cars Sh-Boom scene → flows into photo wall
  Credits (scrolling photos + friend video quotes + Our Town)

Each segment type (title card, intro, episode pass-through, scene, photo wall, credits with friend quotes) is rendered to a temp file and concatenated with fade-to-black transitions. The whole thing was driven by program.toml, a declarative segment list. Change the text, swap an episode file, reorder segments, re-render. The constraint of a real deadline (the party was on a specific date) meant I needed to iterate fast on the program structure without rebuilding everything from scratch each time.

Choosing to have segments exist on disk rather than a full render provided a sort of cache to sample from. If I’d rendered something effectively I didn’t need to re-render it to pull it into a composition. Sometimes modification could occur in place (adding or changing audio tracks within the container).

Two tools hiding in a monolith

The birthday project was a single private repository (alasdair-3rd-credits) with two scripts. pipeline.py (~1800 lines) was the photo wall: scan photos, scatter them on a background, weather them to look vintage, render a Ken Burns video. assemble.py (~1450 lines) was the program assembler: read the TOML, render each segment type, concatenate.

These two things had nothing to do with each other. They didn’t import each other. They didn’t share data structures. The only coupling was that the assembler could reference the photo wall’s output file as a segment. After the party, they became Pinboard and Playbill.

Photo wall layout

105 family photos scattered across two panoramic Route 66 map backgrounds, aged to look like vintage Polaroid snapshots. A camera that zooms in, meanders through them, and zooms back out. Two backgrounds, two songs, about ten minutes of video at 4K.

The backgrounds needed to be cohesive 8K-wide panoramic illustrations in a vintage Route 66 map style. The Cars universe was the visual anchor. I compared DALL-E (maxed out at 1024x1024, seaming issues), Midjourney (best illustration quality but couldn’t suppress text generation, rendering gibberish on road signs), and Ideogram (supported wide aspect ratios natively, respected “no text” instructions). I went with Ideogram. The tradeoff showed up at the party: Ideogram’s backgrounds looked great to adults, but Alasdair spotted the cars immediately. The vehicles didn’t look like anyone from the Cars universe, and he called each one out. Three-year-olds have zero tolerance for unfamiliar objects in familiar worlds.

Each photo gets weathered to a subtle vintage feel (20% desaturation, warm color shift, 15% sepia, 15% vignette, cream Polaroid border with paper noise, soft drop shadow) and then needs to be scattered “organically” across the backgrounds. The initial weathering settings (40% desaturation, 25% sepia) were too aggressive and the composite became a beige smear. The restraint matters more than the effect.

Scattering photos sounds simple until you try it. The Polaroid frames have thicker bottom borders and random rotation angles. They need to not overlap, fill the space without clumping, and look like someone pinned them up over time rather than arranged them on a grid.

A grid layout was the first thing I tried. It works mechanically but looks mechanical. Poisson disk sampling via Bridson’s algorithm gives organic scatter with a guaranteed minimum distance between points. The problem was overlap detection for rotated rectangles.

AABB (axis-aligned bounding boxes) produced false positives: two frames with opposite rotations can have overlapping bounding boxes without their actual shapes touching, placing frames too far apart to fit 105 photos. Circle-based detection was too conservative, wasting roughly 36% of the placement area. The Separating Axis Theorem was the only correct approach: project both shapes onto each edge normal, four axes total for two rectangles. If any projection axis shows a gap, they don’t overlap.

After initial placement at 80% of the worst-case rotated bounding box diagonal, a force-based relaxation loop pushes overlapping frames apart. Each pair gets a constant repulsion force (rather than proportional to overlap, which prevents oscillation). Boundary clamping keeps frames inside the placement zone. Converges to zero overlaps in 20 to 100 iterations depending on density.

Camera path and rendering

Each background segment follows four phases: wide establishing shot with gentle drift, cosine-eased zoom to 40% of background width, zig-zag meander visiting waypoints with a 20% loiter ratio (the camera decelerates into a waypoint, dwells briefly, accelerates toward the next), and zoom back out to wide. Two backgrounds because laying 100+ photos out on one was too dense. The camera mirrors direction: left-to-right on the first, right-to-left on the second.

ffmpeg’s zoompan filter couldn’t handle multi-phase camera paths or per-frame effects like fades. The replacement: Python generates each frame as raw RGB bytes and pipes them directly to ffmpeg’s stdin. Pillow crops the composited 8K image at each camera position, resizes to 4K, writes the result. Complete control at the cost of doing the work in Python.

The initial implementation ran at 9.2 fps, which meant a 43-minute render. Three optimizations combined for a 4.3x speedup. Swapping LANCZOS (6x6 sinc kernel) for BICUBIC (4x4 cubic kernel) was visually identical at 4K viewing distance and about 2x faster. Converting the composited 8K images from PIL Image objects to NumPy arrays at render start gave thread-safe read access via zero-copy slicing (PIL’s crop() is not thread-safe due to lazy loading). That unlocked the third optimization: 4 worker threads preparing frames into an 8-frame lookahead buffer, the main thread pulling completed frames in sequence for ffmpeg’s stdin. Pillow and NumPy release the GIL during C-level computation, so threads achieve genuine parallelism.

I would not have built any of this optimization on my own. Not because the concepts are beyond understanding, but because I don’t carry knowledge of PIL’s lazy loading internals, the GIL behavior of NumPy’s C extensions, or the relative cost of resampling kernels at these dimensions. The model in my AI harness proposed all three optimizations in sequence, each one either directly improving throughput or removing a blocker for the next. What it needed from me was the problem statement and the validation that the output looked identical.

Heterogeneous video

The TV episodes were HDR (BT.2020 color primaries, PQ transfer). Phone clips and title cards were SDR (BT.709). Concatenating them without conversion produces washed-out or oversaturated segments depending on which color space the player assumes.

The assembler probes each input and routes HDR through Apple’s VideoToolbox hardware-accelerated tonemapping:

format=nv12,hwupload,
scale_vt=w=3840:h=2160:color_matrix=bt709:color_primaries=bt709:color_transfer=bt709,
hwdownload,format=nv12

The final output is SDR BT.709 regardless of source. We didn’t know what projector or media player the theater would have, and HDR playback depends on the entire chain negotiating correctly. SDR just works.

Heterogeneous audio

The audio disagreements were worse. Sample rates (44.1kHz, 48kHz), channel layouts (stereo music, 5.1 surround from TV, mono from phone), sample formats, codec containers. The fix was a canonical format forced onto every stream:

aformat=sample_fmts=s32:sample_rates=48000:channel_layouts=5.1

Intermediate files use pcm_s24le (uncompressed 24-bit) to avoid generation loss between pipeline stages. The final container carries both a 5.1 surround track and a stereo downmix. This turned out to matter: the center channel on the playback system was malfunctioning. Having the stereo track baked in meant we could switch outputs rather than troubleshooting theater hardware with a room full of three-year-olds waiting.

Never going silent

Title cards and intermissions have no inherent audio content. An early approach used anullsrc (generated silence), but some audio processors detect the transition from real audio to digital silence and switch modes, producing an audible click. Without access to the theater to test ahead of time, we treated this as a risk.

The fix was a background music track that runs continuously, ducked to 20% volume during title cards and intermissions. The music thread maintains continuity through concat boundaries, so the audio processor never sees a format change or silence gap. This simplified the concat logic (no special cases for silent segments) and gave the whole program a subtle sense of continuity.

Loudness normalization

Sources came in at wildly different levels. Studio-mastered music, TV audio mixed for home theater, phone recordings of a three-year-old in a living room. The pipeline normalizes every segment to -14 LUFS with a -1dB true peak ceiling using ffmpeg’s loudnorm filter in two-pass mode. Conservative enough that nothing clips on unknown amplification, consistent enough to take the load off the projectionist.

Trailing silence and crossfades

The music files lied about their duration. Sh-Boom’s container reported 194 seconds but the actual audio ended at 143. Route 66 reported 282 but ended at 201. Our Town reported 350 but ended at 243. Over 100 seconds of trailing silence per track, baked into container metadata.

ffmpeg’s acrossfade filter consumed inputs based on container-reported durations rather than actual sample counts, producing output 126 seconds shorter than expected. The replacement was manual mixing from primitives: atrim to strip trailing silence (using content duration from silencedetect), individual afade filters, adelay to position tracks at correct start times, apad to extend to total duration, then amix with normalize=0 to combine without the default per-input volume division.

Building this filter graph was where the AI harness earned its keep on the audio side. ffmpeg’s filter syntax is powerful and opaque. The model could hold the full graph in context, reason about signal flow, and propose changes that accounted for upstream effects. I still had to listen to the output and judge whether it sounded right, but the iteration cycle was fast.

Audio ducking

The credits section played Our Town underneath while friend video quotes appeared. When a friend’s video plays, the music ducks down via a per-frame volume expression:

volume='1.0-(0.85)*min(envelope,1)':eval=frame

The envelope is built from clip() ramp functions for each voice clip: smooth 2-second ramp-down at start, ramp-up at end. Where clips overlap the envelopes add, clamped to prevent ducking deeper than 15% of original volume. The voice clips themselves use the same adelay + aformat + apad + amix primitives as the photo wall crossfade.

The split

The birthday project was built under deadline pressure for a real event. A monolith was the correct architecture for that context.

Pinboard was the photo wall pipeline. The extraction was mostly about removing birthday-specific assumptions. Background matching by filename keywords became alphabetical sort order. Title card text moved from hardcoded strings to a CONFIG parameter. Route 66 shield badge keep-out zones became configurable corner exclusions. The core algorithms transferred unchanged. Pinboard shipped as v1.0.0.

Playbill was the program assembler. This extraction required more work because the segment types were designed around a birthday party’s structure. The intro type genericized from Alasdair introducing shows to any portrait video with text overlay. The photowall type became unnecessary (users pass through any video via scene). The credits type (scrolling photos with friend video quotes, audio ducking) was the most birthday-specific but also the most generally useful. The quote type works for testimonials, interviews, or any face-next-to-words layout. Playbill is still bootstrapping, but the idea of a TOML-driven video program assembler feels more generally useful to people than the photo wall does. I want to circle back and get it to a proper DSL: Pydantic typed config, testable modules, and eventually a declarative scene graph where the scene description is the program.

The original alasdair-3rd-credits repository stays private. The birthday-specific program.toml with Alasdair’s commentary, the helper scripts, the 105 family photos and three music tracks, all of that is a historical record of a specific birthday party. It doesn’t need to be open-source. It needs to be a snapshot. The two open-source tools carry forward the interesting engineering work.

Building with the harness

The birthday party had a date. The video needed to exist by that date. The harness exploration post describes the arc of learning to work with AI agents. The birthday video was one of the first projects where the harness felt load-bearing rather than experimental. OpenCode with structured context (AGENTS.md, session memory, observation logs) was the working environment for all of it.

The model was good at iteration documentation, ffmpeg filter graphs, boilerplate and plumbing, and the render optimization described above. What required human judgment was aesthetic tuning (weathering intensity, camera timing, background selection) and program structure (the two-act design, the intermission timing, the decision to have Alasdair introduce each show in his own words).

The observation logs turned out to be the most valuable artifact. When I came back after the party to extract Pinboard and Playbill, the logs contained the full context for every design decision. Why SAT instead of AABB. Why amix instead of acrossfade. Why BICUBIC instead of LANCZOS. The model’s first session on the extraction work had the same context as my last session on the birthday project because the observations bridged the gap.

The pattern that emerged: I describe what I want at the level of intent and constraints. The model proposes an implementation. I evaluate the output (watch the video, listen to the audio, look at the layout). I describe what’s wrong in terms of the experience (“the photos look muddy”, “the crossfade has a volume dip”, “the camera lingers too long”). The model translates that into parameter changes or architectural fixes. Repeat. The harness (structured context, memory, observations) is what makes this sustainable across sessions rather than requiring re-explanation each time.

On a short timetable the important thing was getting results first, then recognizing which bottlenecks were worth accelerating. The render optimization only happened because I noticed the 43-minute render was the thing blocking iteration on camera timing and weathering. Getting to a working output quickly, then focusing on the tight loops where rapid iteration mattered most, was more valuable than optimizing anything upfront.

The birthday video shipped on time.