MELBOURNE RHUG · OCTOBER 2026

How much OS trust can you bake in at day 0?

A field report from a homelab, with receipts.

Andrew Dunn · GitLab working title · draft every link in this deck is public
02THE HOOK

My phone updates itself overnight.
My servers still get patched by hand.

My phone
  • One image, built and tested before it ships
  • Updates overnight, atomically
  • Rolls back by itself when something is wrong
  • Nobody patches it by hand

Millions of devices · one image · no hands

My servers
  • Installed once, then configured forever
  • Patched in place, host by host
  • Drifts, quietly, one edit at a time
  • Audited by scanning and reconciliation

Every host, its own snowflake

How much of the phone treatment can a server OS take, and how much of it can CI bake in before first boot?

03WHAT IS VERIFIED, WHAT IS WATCHED

Above the kernel, integrity rests on a baseline someone maintains.

/etc · /var configuration and state root filesystem libraries · binaries · my content dashed unless IMA appraisal is running kernel modules the drivers I added kernel signature checked bootloader signature checked firmware UEFI · the root of it all after the handover Tripwire-class FIM · AIDE · OSSEC hash files, reconcile against a baseline database IMA — measurement, or appraisal measurement attests to a verifier; appraisal denies the read outright SCAP scans periodic compliance sweeps, report what they found verified at boot Secure Boot checks each signature, then hands over solid = enforced in the boot or read path dashed = watched after the fact, or writable by design

Enforcement above the kernel exists, per file, against reference values someone regenerates.

04THE LAY OF THE LAND

The moving parts, and who ships them.

BUILD Containerfile the recipe · mine, reviewed like code GitLab CI builds + signs every commit Container registry holds the images · named by content digest DELIVER bootc pulls the image · boot, update, roll back composefs + ostree lays it on disk, content-addressed BOOT UEFI Secure Boot verifies each signature at power-on systemd-boot signed loader, loads the UKI UKI · unified kernel image kernel + initramfs + cmdline · one signed PE binary TPM · measures what actually booted ■ GitLab■ Red Hat■ platform■ mine FALLS OUT: ATOMIC UPDATE · ROLLBACK · ONE DIGEST · SEALED ROOT

None of these boxes is the feature on its own. The properties fall out of how they compose.

05THE FACTORY

I got CI to build the whole OS, drivers included.

GITLAB CI · EVERY COMMIT · COMPONENTS: cairn/pipeline commit the recipe changes build the whole OS seal digest root, sign UKI boot gate prove it boots promote :stable moves INSIDE THE BUILD STEP the exact kernel the one this image ships compile ZFS + NVIDIA against that kernel sign every .ko ship no toolchain THE MODULES ARE COMPILED AND SIGNED IN THE SAME BUILD THAT PRODUCES THE OS /etc · /var root filesystem built from a recipe · named by a digest kernel modules compiled + signed in the build kernel bootloader firmware ROOTFS + MODULES: BUILT, SIGNED, NAMED

The expensive part was the out-of-tree modules: compiled against the image's own kernel and signed in the same build.

06THE BOOT GATE

I gate promotion on a real boot.

boot-gate · in a VM · inside the pipeline
install this exact image to a disk ............ ok
boot it · reach a login prompt ................. ok
load zfs .............................................. ok
nvidia · signature accepted (no GPU on this VM) ... ok
promotion unlocked · :stable → this digest
THE ARTIFACT

The exact bytes, not a rebuild

THE INSTALL

A real install, to a real disk

THE BOOT

A real boot, to a login prompt

THE MODULES

zfs loaded, not just present

the same bytes that pass are the bytes that ship · a failed gate moves nothing

watch one:https://gitlab.com/dunn.dev/cairn/base/-/jobs/16417649382

that job: the boot gate on cairn/base main, 10 September 2026, green · the whole gate is one script and it is in the repo above · the run it belongs to is backup face B7

The gate tests the bytes that ship, not the recipe.

07THE RIG · A BOOT LAB INSIDE CI

The smoke test is a VM inside the job.

THE FACTORY · THIS PIPELINE RUNNER HOST · ITS OS IS A whetstone GOLDEN IMAGE, BUILT TO qcow2 FROM cairn/base ↺ GITLAB-RUNNER · ROOTFUL PODMAN SOCKET · HANDS THE JOB /dev/kvm CI JOB · tags: [crucible] · THE GOLDEN IS READ-ONLY: qemu + swtpm BAKED IN, NO JOB-TIME INSTALLS QEMU/KVM virtual machine · fresh scratch disk · the image under test, with its own swtpm qemu-system-x86_64 -machine q35,accel=kvm -cpu host -m 4096 · console → serial.log
ONE GATE RUN · sealed-boot.sh · EVERY EDGE IS IN THE SCRIPT advance: grep -aq "login:" · ≤10 min PULL EXTRACT ENROLL INSTALL BOOT GATE GATE GREEN digest recorded UKI · composefs= db ← UKI cert to-disk · loopback SB on · swtpm assert-sealed.sh :stable → digest no UKI · no composefs= karg sealed root not found SB rejection · panic · verity · exit · timeout ssh dead 120 s → exit, no verdict assertion fails → VERDICT=FINDING GATE RED · :stable UNMOVED · RECEIPTS + serial.log KEPT AS ARTIFACTS

SaaS shared runners do not expose /dev/kvm · this is one small self-managed runner, doing one job · every state and edge above is sealed-boot.sh behavior

The runner host boots an image this same pipeline built.

08THE SEAL

I sealed the image under one digest.

"What am I running?" · detection
  • Tripwire-class FIM · baseline database
  • IMA + Keylime · attest after the read
  • Patch management + inventory agents
  • Vulnerability scanner
  • SCAP compliance scans

All reconciliation, after the fact, against a baseline someone maintains.

Sealed image · enforcement
/etc · /var root filesystem every read checked by the kernel kernel modules kernel bootloader firmware one digest signed at build, verified at run
composefs = EROFS metadata + overlayfs + fs-verity objects the root digest rides in the signed UKI command line dm-verity seals a device · fs-verity seals files /etc and /var stay writable, by design

The baseline is now one digest I signed at build. The kernel checks it on every read.

09WHOSE KEYS, WHOSE MACHINE

What happened when I enrolled my own keys.

THE FIRMWARE'S KEY SLOTS · REPLACED WITH MINE IN A CEREMONY PK · platform key the root of the hierarchy · mine signs KEK · exchange key authorizes db updates · mine signs db · what may boot verifies systemd-boot + the UKI at power-on the same db key signs every .ko at build too · the kernel enforces it at load shim · MOK leave the trust chain together: I took shim out of the image, and MOK goes with shim Microsoft's CAs stay in db: my option ROMs are signed against them, my boot chain is not module trust moved BOTH ARRANGEMENTS BOOT-TESTED · SHIPPING ONE KEY · FINGERPRINTS: cairn.dunn.dev/keys

NSA · UEFI Secure Boot Customization · CTR v1.2 · §3.2:

https://media.defense.gov/2023/Mar/20/2003182401/-1/-1/0/CTR-UEFI-SECURE-BOOT-CUSTOMIZATION-20230317.PDF

I took shim out, MOK went with it, and module trust moved into db.

10SIDEQUEST · MODULE TRUST

Should my own key load a kernel module?

MAINLINE LINUX ✗

No

The platform keyring does not verify modules, deliberately. Four proposals since 2019: two rejected on principle, one died, and the 2025 RFC got no second version.

NEARLY EVERY DISTRO ✓

Yes

A downstream patch trusts the platform keyring for modules. Of 9 families I surveyed, 8 carry it.

MY LAB ⚖

Boot-tested both

Both directions, with a negative control. The behavior is real and load-bearing; the docs state the rule but never that it is a patch.

CARRY THE PLATFORM-KEYRING PATCH · 8 OF 9

Fedora · RHEL · Alma · Rocky · Ubuntu · Debian · SUSE · Oracle RHCK

MAINLINE .machine PATH ALONE · 1 OF 9

Oracle UEK 7/8. The split runs inside one vendor: Oracle's own RHCK carries the patch, UEK7 dropped it.

my RFE · .mokkeys · withdrawn, the why:https://github.com/systemd/systemd/pull/43638#issuecomment-5611922484

The behavior fleets rely on is a distro patch, and the docs state the rule without saying it is one.

11THE LADDER

Four rungs, and what each one costs.

4 · state-bound the TPM releases the disk key only for the signed boot · wrong OS, no disk COST: TPM ENROLLMENT + A RECOVERY DRILL 3 · own keys the only loader on the disk is mine · their CAs stay in db, so option ROMs load, and a Microsoft-signed loader would too COST: A KEY CEREMONY · KEYS TO KEEP SAFE 2 · sealed one digest over the whole image · the kernel verifies every read COST: UKI + COMPOSEFS PLUMBING 1 · stock signed boot as the distro ships it + a CI boot gate · running in public today COST: ORDINARY CI

/etc and /var stay writable by design at every rung: the scoped, auditable mutable surface · at rung 4 a recovery passphrase always works.

Where to stop is a requirements question.

12CLOSE

What I found along the way.

systemdTPM2 unseal failed on ECC P-384 TPMs; fix zero-pads ECC point coordinatesmerged 08/2026 https://github.com/systemd/systemd/pull/43396
systemdRFE: .mokkeys module-signing key enrollment; withdrawn after review, the reasons in one commentwithdrawn 09/2026 https://github.com/systemd/systemd/pull/43638#issuecomment-5611922484
bootccomposefs fails with Docker v2s2 media typesopen https://github.com/bootc-dev/bootc/issues/1703
bootcinstall to-disk with LUKS + TPM brokenopen https://github.com/bootc-dev/bootc/issues/421
image-builderLUKS partition type in disk customizationsopen https://github.com/osbuild/image-builder/issues/2609
bootc · composefschunkah re-chunking regressions, filed at backout; four of my fixes merged, two reports openfiled · 4 merged https://github.com/bootc-dev/bootc/issues/2408 · https://github.com/composefs/composefs-rs/issues/383

the estate:https://gitlab.com/dunn.dev/cairn

the docs:https://cairn.dunn.dev

The two-box grammar and the sealing frame are borrowed with thanks from Mark Russell and Colin Walters of Red Hat, "Trust at every layer: how sealed images extend OS integrity from boot to runtime," May 2026:

https://www.redhat.com/en/blog/how-sealed-images-red-hat-enterprise-linux-extend-os-integrity-boot-runtime

B1BACKUPUPDATES OVER A CONSTRAINED LINK

What re-chunking saved on the wire.

MEASURED IN THIS ESTATE · 12 PRODUCTION BUILDS · TWO skopeo COMMANDS REPRODUCE IT full pull · 1084 MiB median real day, chunkah re-chunked into 64 content-keyed layers · 585 MiB · 46% less controlled one-package update · 87% less THE GAP BETWEEN 87 AND 46: MY OWN DRIVER BLOBS RE-PULLING EVERY BUILD
How it ended
  • chunkah (coreos/chunkah) did not survive production use in this estate
  • I backed it out and filed the regressions to bootc and composefs
  • These are historical measurements, not live claims
Why it still matters
  • Content-keyed layering is the right shape for narrow links
  • The seal is unaffected: the root digest is computed over content, not transport layers

I measured it, then backed the tool out. Four of the fixes are merged.

B2BACKUPTHE DISK GETS AN OPINION

I bound the disk to the boot I signed.

The boot state I signed
  • TPM checks the measured, signed boot
  • Releases the disk key
  • No passphrase typed, ever
  • A signed update unlocks without ceremony

One image, no per-machine ceremony

Anything else
  • No signature authorises this boot state
  • The unlock is declined before the TPM is asked
  • Disk stays sealed
  • Recovery passphrase always works

Refusal is the default; recovery is the exit

systemd-cryptenroll binds the LUKS slot to the signed boot measurement first boot re-binds that slot to the machine the image actually landed on on ECC P-384 TPMs this did not work until a fix of mine landed in systemd, 08/2026 the refusal is systemd's, before the TPM is asked: no signed policy for this boot's PCRs, no unseal attempt a replayed signature and a foreign policy key are indistinguishable from the console

one image, three install postures: TPM auto-unlock + recovery (edge) · passphrase only (attended) · no encryption (ephemeral, CI)

I demonstrated both directions on a real rig.

B3BACKUPFIELD NOTE

A green gate that proved nothing.

WHAT I HAD ●

A lane that never ran

One path of the gate had never executed. Its checkmark was green anyway.

WHAT THAT MEANS ◍

An assertion that could not fail

Structurally green, whatever the artifact. Green was telling us nothing.

THE FIX ◆

Prove the gate can fail

Hand it a known-bad artifact and require a red result before trusting a green one.

the fix, rendered as a test
hand the gate a known-bad artifact .......... red · as required
the gate has now been seen red

I decide what a gate must prove, then break that exact thing on purpose.

B4BACKUPFIELD NOTE

What the build committed that I did not write.

WHAT HAPPENED

A stale boot ID rode into the image

A leftover /run/libpod file from the build container was committed into the OS image and surfaced at first boot.

WHY

Two builders, two commit semantics

rpm-ostree compose snapshots a clean tree; buildah RUN commits the container's runtime residue too. Same Containerfile, different bytes.

THE CHOICE

Compose semantics, deliberately

This estate builds rpm-ostree compose from scratch: a curated tree is snapshotted, so build-container residue never rides along.

rpm-ostree compose rootfs ...... a snapshot of the tree I curated
buildah build · RUN dnf … ......... that tree, plus what the run left in /run

the checklist line I took from this: pin the base by digest · know the builder's commit semantics

my choice is on the record as D1:https://gitlab.com/dunn.dev/cairn/base/-/blob/main/docs/decisions.md

Layering on someone else's base inherits its builder's commit semantics.

B5BACKUPDAY 2 · ROLLBACK

I gave the host a way to roll itself back.

THE SUCCESS BAR ⚑

boot-complete.target

The boot either reaches it or it does not. systemd-native, no agent.

THE MEMORY #

A machine-local counter

Failures accumulate across boots. The machine remembers its own bad mornings.

THE ACTION ↩

bootc rollback

The counter trips it; the previous image boots. No operator on the console.

runs on a live host in the estate · one deliberate failed-update rollback demonstrated

why I built my own · greenboot-rs is still GRUB2/grubenv-coupled; systemd-boot is an open RFE there:

https://github.com/fedora-iot/greenboot-rs/issues/175

and bootc's own maintainer named boot counting as wanted for the composefs backend:

https://github.com/bootc-dev/bootc/issues/1190

I maintain this gate myself. The maintainers have named the gap upstream.

B6BACKUPTHE WRONG KEYRING

I put a key in the wrong keyring.

WHAT I TRIED §

.mokkeys

A UKI section; the boot stub wrote shim's MOK table from it, no shim present. It worked in the VM.

WHAT IT COST ✗

A week and three maintainers

The table's GUID is Linux's; the entry names are shim's. I had written under shim's names without asking.

WHAT I FOUND AT HOME ⌂

.machine is empty

My design record said a CA-shaped db certificate is promoted into .machine. It never is. Modules load through Fedora's db patch.

.platform OPT-IN · BUILT · +75 LINES

Trust db for module signatures only when the signed command line says so. .secondary untouched. Wart: root can kexec the token onto a stock kernel.

.vendor AS WRITTEN · BUILT · +275 LINES

A new table, any loader may fill it, linked into .secondary unconditionally: modules, kexec, IMA, vouching. No consent step. Parsed with Secure Boot off.

the withdrawal, in one comment:https://github.com/systemd/systemd/pull/43638#issuecomment-5611922484

six fixes to the .vendor branch, public:https://github.com/andrewdunndev/linux/tree/vendor_keyring-fixes

Both worked in my VM. I have filed neither.

B7BACKUPTHE REAL RUN

What the pipeline actually did.

cairn/base main, 10 September 2026 · every pill links that job's log · the boot gate is sealed-boot, the disk binding is autoenroll, and promote-sealed is where the tag moves

the run:https://gitlab.com/dunn.dev/cairn/base/-/pipelines/2836618446

This is the run the rest of the deck describes.

B8BACKUPIPE · THE OPEN QUESTION

Why I still sign my modules.

THE QUESTION AT EVERY MODULE LOAD may this module load? TODAY · A SIGNATURE ON THE FILE does the file carry a signature? so I sign every .ko with the same key still one more step at day 0 IPE · A PROPERTY OF THE STORE where did the file come from? no signature on the file at all which store counts is being settled now IPE IS A KERNEL POLICY · IT ASKS WHERE A FILE CAME FROM, NOT WHAT IS ATTACHED TO IT
WHAT I MEASURED · STOCK FEDORA 44

ipe is already an active security module, kernel modules are in scope, and a policy has to be signed by a key the firmware already trusts — which mine is. It ships switched off, with no policy loaded.

WHAT THE IPE AUTHOR TOLD ME · 14 SEP

The anchor he plans for composefs is a signed dm-verity backing store, and my ext4 lock buys fs-verity, not that. Where kmod decompresses in userspace it hands over a buffer, and IPE denies it. An RFC is coming.

I asked on Walters' tracker; Wu answered:https://github.com/composefs/composefs/issues/360

Module signing stays for now.

speaker notes · n to hide