Immutable — andrew.dunn.dev

A signed bootc image estate for homelab hosts that need ZFS pools and NVIDIA GPU compute. Built FROM-scratch on Fedora with source-built kernel modules MOK-signed at build and a CycloneDX SBOM per image. :stable is earned: every candidate boots in a nested-KVM gate before the tag moves, and a failed boot freezes it. Validated in a VM before deploy, never assumed.

The kernel modules article covers the multi-stage build pattern in detail.

Fedora and kernel module source go in, and nothing carries the :stable tag until the image boots in a nested-KVM gate.

The estate

pipeline and bootc-base-imagectl build the basef base image, each host forks it into an instance image, and validation boots that image in a VM before it ships.

basef: FROM-scratch Fedora bootc base images. Source-built ZFS and NVIDIA kernel modules in throwaway builder stages; only the compiled .ko files and userspace tools land in the final image. composefs for content-addressed filesystem layers and an opinionated pure-systemd posture.

bootc-base-imagectl: Rust tool that transforms an upstream rootfs into the bootc-shaped layout basef builds on. Frozen at v0.1.4 against a known-good upstream surface so the chain has a stable foundation.

pipeline: Reusable GitLab CI/CD components covering the full lifecycle: upstream image mirroring, kernel version detection, module compilation, MOK module signing, image assembly, SBOM generation, and CycloneDX attestation.

instance: Forkable reference configuration on top of basef: quadlet container definitions, systemd services, network configuration, secrets. Fork it, customize for your hardware, push to build.

validation: VM-based validation harness. Boots the built image in a VM, runs smoke tests, reboots, verifies the system comes up clean. Catches boot-time failures that container-level bootc container lint cannot.

basec (CentOS Stream 10 + GRUB) is archived for reference. The FROM-scratch Fedora approach in basef supplanted it once composefs stabilized.

Key patterns

Supply chainEvery image’s source-built kernel modules are MOK-signed at build (CN immutable.dunn.dev) so hosts can run them under Secure Boot, and each image carries a CycloneDX SBOM. Full image signing with Sigstore cosign is the goal but is currently blocked upstream; the registry policy stays permissive in the interim.

The estate is built around a few patterns:

FROM-scratch images. No upstream image is taken on faith. basef builds on top of a Fedora rootfs transformed by bootc-base-imagectl into a bootc-shaped layout. Every byte that ends up on disk is traceable to a source we control.

Multi-stage kernel module builds. ZFS and NVIDIA modules compile in throwaway builder stages with the full toolchain. The production image receives only compiled .ko files and userspace tools via manifest-driven file copy. The kernel modules article covers this in depth.

Supply-chain provenance. Source-built kernel modules are MOK-signed at build, and every image in the chain (base, instance) carries a CycloneDX SBOM generated by syft, living in the same registry as the images. Full image signing with cosign is tracked but blocked upstream, so the registry policy is permissive in the interim.

Layered validation. Builder stages validate their own output (modinfo, ldd, manifest file counts). The final image validates no build dependencies leaked, all tools are present, all services are enabled, and bootc container lint passes. The validation harness boots the image in a VM and runs smoke tests.

What’s next

Development moved to cairn, the successor estate. The move from GRUB to systemd-boot with Unified Kernel Images that this estate planned is where cairn starts, with the estate’s own Secure Boot keys in place of shim and MOK, and promotion gated on an actual sealed boot in CI. What generalized from here: the FROM-scratch build discipline, the multi-stage kernel module pattern, and the principle that a tag is earned by a boot. The bootc bug investigation that shaped this estate’s validation harness shaped cairn’s gates too.