cairn, stacking primitives

One

An Android phone runs from one slot and writes the update into the idle one.

Android calls them slots. Newer phones use Virtual A/B, where the idle slot is a snapshot.

An A/B update runs while you keep using the phone. A background daemon writes the next system into the idle slot, which nothing running touches, then tells the bootloader to boot that slot next time. Nothing in your hand has changed yet: the new system is written and waiting.

Two

Before the new slot runs, it is checked. Each boot stage verifies the next before handing over.

An iPhone has one signed system volume; Apple calls its hash tree similar to a Merkle tree.

Verified Boot is the chain: the bootloader checks the kernel, and the kernel checks the slot with dm-verity, a hash tree with one hash per block under a root the vendor signed. A block that does not match is refused. Android and iPhone arrange the system differently and share one rule: check before you keep.

Three

The new slot is kept only if it boots. Boot it, mark it successful, or fall back.

Android says several attempts and names no number, so neither do I.

The bootloader tries slot B. Once the system is up, a user space daemon runs its checks and marks the slot successful, Android's word; until then it is only bootable. If several attempts never get there, the bootloader marks B unbootable and falls back to A, the system the phone had before. My servers did none of that.

One

A kickstart laid the host down once, from a recipe, and nothing read that recipe again.

A kickstart file is a text recipe: disks, packages, users and a script that runs when the install ends. Anaconda installs from it with no prompts. Day zero was the only day my host and its recipe agreed. After the first reboot the file stayed in a repo; the host went on alone.

Two

Every patch window after that rewrote the running host in place, and hand edits landed beside the packages.

dnf history keeps every transaction and can undo one. Restoring from backup was my practice, not the tool's limit.

dnf upgrade resolved the new package set and swapped files on the live filesystem while services kept running. A config line or a unit dropped into /etc sat beside those packages with no record beyond the shell history. I never exercised the result before it went live, and a window that stopped halfway left a host matching neither list.

Three

What the host actually ran, I learned by scanning it, after the fact.

Configuration management calls the recipe a baseline; a scan measured the distance from it, reading packages and files back from the host. Every finding was already live, and had been since some patch window I could date only from the logs. The phone in chapter 1 had four properties this host never had.

One

The list of what a host is did not change: packages, kernel arguments, users, units, drivers.

Every host I ran was those five things, and each landed on the running machine: a package installed, a kernel argument edited, a user added, a unit enabled, a module compiled against whatever kernel was there. The list stays. What changes is where it runs.

Two

The list left the host for Containerfiles, built by buildah in a GitLab CI pipeline.

The schedule only detects moved inputs; a change starts a build. The cadence is a setting, not a file.

Four of the five are steps in base's Containerfile, and each step sits under a labelled block; chapter 6 reads one. The drivers have their own Containerfiles, layered on top. buildah runs them in a GitLab CI pipeline and pushes the image to the registry. The pipeline runs on a push that touches the image paths, and on a schedule that watches upstream inputs. The image exists before any machine has run it.

Three

The host does one thing with all of this: it pulls the image and reboots into it.

What happens when the new image fails to boot, and how the host rolls back, is chapter 27.

bootc fetches the image as it would any container image, stages it as the next deployment, and the reboot boots it. Nothing in CI reaches a host in service today; a job logs in only to machines it booted to test. That is practice, not enforcement; a fork could wire a pipeline to a host. I wanted a host I could lose without losing anything I had typed into it.

One

A container is an ordinary process that the kernel has shown a narrower machine. There is no second kernel.

Mount came first, in Linux 2.4.19. Time came last, in 5.6.

The narrowing is done by namespaces, eight kinds today: process ids, mounts, network, hostname, ipc, users, the cgroup root and clocks. Each one wraps a global resource so the process sees its own copy. Inside, the process is PID 1 and can see nothing it did not start. From the host it is one more entry in the process table, with a different number. podman hands the container to an OCI runtime, which asks the kernel for the shells with clone, one flag per namespace.

Two

cgroups are the budget: how much CPU and memory a container may use, and how many processes it may start.

Cgroup v2 became official in Linux 4.5, in 2016. Fedora 31 made it the default in 2019.

A control group is a directory in a filesystem the kernel keeps. Three files hold the ceilings: cpu.max, memory.max and pids.max. A new group starts with max, which is no ceiling, and cpu.max also carries a period of 100000 microseconds: the quota is how much of each period the group may run. Every systemd service gets a group like this too, so a container and a service are budgeted by the same files.

Three

The root filesystem is the only part that travels. Everything else here belongs to the host kernel.

base Containerfile: FROM scratch, then COPY the rootfs to /.

The mount namespace lets pivot_root turn one directory into /. That is the root filesystem, only files, and it is all an image ships: no kernel, no process, no limits. In cairn it is the whole operating system, copied FROM scratch. Docker showed this combination in 2013; the kernel already had the pieces. Chapter 5 takes the slab apart.

One

An image is an ordered list of layers. Each layer is a tar archive, named by the digest of its bytes.

sha256 over the bytes: same input, same name, on any machine.

The OCI image format is three kinds of blob. A layer is a tar of files, gzip or zstd compressed, holding what one build step added or changed. A config records how to run the result and the order of the layers. The manifest lists both by digest, and the digest of the manifest is the image's identity. Nothing in the format knows about a kernel or a host.

Two

Height is size. The base-zfs-nvidia variant is 64 layers and 832.5 MiB, and a new host needs all of it once.

Measured on cairn/base; the same numbers sit behind the deck.

A host keeps every layer it has ever pulled, filed by digest. A pull starts with the manifest, which is small, and compares each layer digest against that store. Only the names it does not have are fetched; the registry serves blobs by name and never has to know what the host already holds. So the first pull onto a new machine is the whole stack, and every pull after it is a diff.

Three

A same-commit rebuild changes five layers. The host pulls 43.8 MiB, about 5% of the image.

Five of 64 layers changed; 43.8 of 832.5 MiB travelled.

Identical files make identical tars, and identical tars have identical digests. The 59 layers whose bytes did not change keep their names, so the host already has them and skips them. Only the five new names travel. This holds because rpm-ostree writes the root filesystem as chunked layers, so a change to one part of the system lands in the layers that part lives in and leaves the rest alone. Turns out the stack is the update mechanism.

One

A Containerfile is the recipe. Mine builds an operating system's root filesystem from nothing, one block at a time.

Each block says what it does, why, and which upstream document it follows.

A Containerfile lists build steps, and buildah runs each one in a fresh container and keeps what changed as a layer. Mine starts FROM scratch: an empty filesystem. A builder stage runs rpm-ostree compose, which lays a root filesystem down from a package list, and the rest of the file adjusts that tree. Every step sits under a cf: header, and there are more than forty of them; the sheet shows six. I wanted a reader to be able to audit the whole system from one file.

Two

A build yields a digest, the image's identity. A tag is a name that can be moved later.

:<short-sha> is the candidate: one tag per commit, written by the build.

Content addressing means the same bytes always give the same name, so a digest names exactly what was built. A tag is a different thing: a mutable name in the registry that points at one digest today and another tomorrow. When I need what was tested, I pull by digest. When I want whatever is current, I pull a tag and accept that it moves.

Three

Three tags: the candidate, the one that passed the gates, and the one a host installs from.

Tags named keep- sit outside registry cleanup, along with :latest and :stable.

Every commit gets :<short-sha>, and nothing is tested but that digest. Once the sealed build boots under the gates, promote works through the list below and moves :stable to the digest that passed. The sealed lane moves :stable-sealed when both gates are green. A failed gate moves neither, and the last good image stays where every host can still find it.

One

A bootc host keeps up to two root trees on one filesystem; the loader picks one by entry order.

cairn's sealed hosts boot bootc's other backend, composefs: same verbs, no OSTree. This chapter draws the default.

bootc boots a container image as the operating system and by default stores it as an OSTree deployment: a whole root tree checked out as a directory, hard-linked into one repository with the other. Not a partition. Both share one stateroot with exactly one /var, mounted into whichever tree booted. Each deployment gets one Boot Loader Specification entry; the loader takes them in order.

Two

bootc upgrade pulls the next image beside the running one and stages it; nothing running changes.

bootc upgrade fetches the image and checks it out as a new deployment; the manual calls it staged, and bootc status shows it. The /etc merge and the boot entries wait for shutdown, when OSTree's finalize service completes the staged deployment and writes the entries, B first. Today the reboot is the whole switch: the loader starts B, and A sits untouched beside it.

Three

bootc rollback reorders the two boot entries. Nothing is copied back, because A was never touched.

Chapter 27's opt-in health gate calls bootc rollback after repeated failed boots.

bootc rollback reorders the entries and creates no deployment, so A boots as it last ran and B stays as the rollback entry. OSTree keeps a writable /etc per deployment and fills a new one by a three-way merge when it finalises that deployment. Edits made while B ran stay in B; A's /etc returns as A left it. /var is shared and never moves.

One

The base image starts in a throwaway builder stage, where rpm-ostree composes a root filesystem from a treefile.

rpm-ostree's docs say its focus is moving to bootc and dnf; the compose is the part I use.

The Containerfile is a multi-stage build. The builder installs rpm-ostree and runs rpm-ostree compose rootfs against base.yaml, a treefile that names the package sets and turns weak dependencies off. The compose writes a plain directory, the repo's overlay files are copied over it, and CI pins the compose to the commit's source date. Then the builder is discarded.

Two

The last stage is FROM scratch: an empty image and a copy of the composed root. Then bootc lints it.

bootc container lint --fatal-warnings runs in the pipeline's build job, before the push.

There is no parent image. FROM scratch is an image with nothing in it; a COPY from the builder puts the composed root at /, and one more lands a proof marker from a test stage. Every file is one the build asked for: a treefile package, a repo overlay file, a Containerfile step, or that marker. Then bootc container lint checks the layout bootc expects, and any warning fails the job.

Three

One composed root, five flavors. The other four add one layer each and earn their tags by booting.

base-vm boots as a VM guest and earns :stable only, as the README states.

cairn/base publishes five flavors; flavors.yml names each one's parent and build directory. base is the root above. base-zfs adds source-built OpenZFS kernel modules, base-nvidia adds the GPU modules, base-zfs-nvidia adds both (Chapter 9 builds them). Every metal flavor earns :stable and :stable-sealed by booting its own sealed artifact under the gates of Chapter 6.

One

A kernel module is compiled against one kernel build and fits no other; the match is made somewhere.

The builder is Fedora's bootc image of the same release, with kernel-devel pinned to that version.

A loadable kernel module is code the kernel links in, built against one kernel's headers; the kernel refuses one built for another. DKMS keeps the source on each host and recompiles after every kernel update. I make the match once: the pipeline reads the kernel version out of the image it just composed and hands it to the module builder.

Two

The modules are compiled, signed and checked against a committed file list in one pipeline, never on a host.

Decision D29: the modules sign with the boot image key, one signer to trust.

OpenZFS and the NVIDIA open modules live outside the kernel tree, and Fedora ships neither. The builder compiles them from source; sign-kmods.sh runs the kernel's own sign-file over each .ko and fails on one without the trailer. A manifest gate diffs what each layer ships against its expected.txt; a path added or missing fails the job.

Three

At boot the kernel checks each module's signature against keys it trusts; the CI boot asserts that it does.

The GPU module is signed, its signature accepted, never exercised: the runner has no GPU.

Upstream module signing checks against the kernel's own keyrings, never the firmware's. Our signer is the boot image's certificate, enrolled in the firmware's db, and Fedora's kernel falls back to the platform keyring, where db is loaded, when its own cannot. With lockdown on and signatures enforced, an unsigned module does not load; assert-sealed.sh asserts both in the CI boot.

Claim

A host image is one thin FROM line on a base variant plus a little config; it bakes only what is safe to publish, and the private part of a host arrives after first boot.

Say it this way

A host repo is tiny: FROM a cairn variant at :stable, add an account, a kernel argument, a unit or two, push. Secrets land after first boot, so the image can be public even when the host is not.

Sources

• cairn/base README: fork and roll, the variant table
• the instance component: four jobs, lint, detect a base change, build, promote
• Running it: the cascade stops at a promoted image, and the seam between public and private is verification
• cairn/example, four forkable lanes, keep one; whetstone and roon, two public instances on base-vm
• sysusers.d and bootc kernel arguments: what a thin layer adds
• GitLab CI variables: the signing key is a masked variable, never a layer

Where I might be wrong

• "Promotes it like base" overstates. Base earns :stable by booting in CI; the plain instance lane has four jobs and no boot. Sealed instances get a separate lane. The chapter must keep the two apart.
• "SSH keys arrive after boot" holds for day-to-day keys. A host may bake a recovery key on purpose. Say "day-to-day".
• The baked-versus-placed list comes from the deck and the public example lanes. Private host Containerfiles cannot be linked, so the chapter shows the example instead.

Open questions

• What is the smallest honest instance Containerfile to draw inline: the example's plain lane, or the README's three-line fork?
• Does this chapter show the example's four lanes, or leave them whole for chapter 35?
• The fleet figure: a rack of stations labelled by role and variant. Is a role label enough to make the point without naming machines?

Claim

A host's CI file is a few includes at a major and a base image name; the pipeline, its jobs and the runners they land on all come from a component I version.

Say it this way

The host repo writes no jobs; it names a base image and includes a component. GitLab turns that into a pipeline, and the component is mine.

Sources

• Pipelines, jobs and stages, GitLab docs
• GitLab Runner and tags, GitLab docs
• include and CI/CD components, GitLab docs
• example/.gitlab-ci.yml, a consumer's whole CI file
• pipeline/templates/instance.yml, the component it includes

Where I might be wrong

• The stage count. The instance component runs detect-changes, build and promote; the shared preset now declares more stages for consumers that opt into sealing and disks. The built chapter has to pick one component and count its stages.
• Lint and detect-changes run on GitLab-hosted runners. That is hardcoded in instance.yml today and could move.
• "A protected runner" is my shorthand for a runner that takes jobs only from protected refs. Check GitLab's own wording before it goes on a slide.

Open questions

• Show the real example file or a redrawn strip? The real file also names runner tags, which the figure should not.
• How much of a runner to explain: polling, the executor, and that mine run on hardware I own.

Claim

The pipeline is a catalog of versioned components; a release is one tag, and every consumer takes it together.

Say it this way

Ten components, one version number: when the major moves, every repo moves. A repo left on the old major has drifted.

Sources

• CI/CD components and releases, GitLab docs
• cairn/pipeline, the component table in its README
• CONTRIBUTING.md, versioning and the one-transaction rule
• CHANGELOG.md, breaking changes and their migrations
• The catalog entry and its releases
• Semantic Versioning 2.0.0

Where I might be wrong

• The count. Nine components on 31 October; a tenth landed on 1 November with v19.2.0. The built chapter fixes a snapshot date and counts at it.
• "@19 resolves at parse time to the newest release" is the catalog's CONTRIBUTING wording. Check it against GitLab's semantic range rules before quoting it.
• "Every consumer bumped together" is a rule I set and a check I ran once over each repo's CI file. Most of those repos are private; the page can show example, whetstone and roon.

Open questions

• Nine blocks or ten in the parts list?
• How to draw a transaction without borrowing the database picture.
• Chapter 17 also claims the catalog-major transaction. One of them should own it.

Claim

A promoted base rings each instance once and never waits; the cascade ends at a promoted image, never at a running host.

Say it this way

Base rings the doorbell and walks away; each instance rebuilds and earns its own tag. Nothing in CI touches a running host.

Sources

• Pipeline triggers and the job token, GitLab docs
• Protected variables and allow_failure, GitLab docs
• base/.gitlab/variant.yml, the trigger job after promote
• base/docs/decisions.md, D30: base rings the doorbell, never watches
• bootc-upgrade(8)
• Running, cairn docs

Where I might be wrong

• "No standing credential." The trigger uses the job token, which I read in the job and in D30. What else that token can reach is a per-project GitLab setting I have not audited.
• "No public file names a consumer" is too strong. The base README lists whetstone, and roon says it is built on base; the full instance list is a protected variable. The public consumers I know of are example, whetstone and roon.
• The trigger job also runs on the daily schedule, where there is no build to wait on. Whether a ring with nothing new behind it is a bug or a feature needs a sentence.

Open questions

• How many instance stations to draw when the count is not public.
• Does the keep-out between registry and host belong here or in Part VI, where the operator's hand is the subject?

Claim

Every image-changing push to main builds and seals a candidate, boots it twice, walks it through an upgrade and back, and only then moves the stable tags.

Say it this way

Nothing gets a stable tag for being built. It gets one for booting twice under enforcing Secure Boot and surviving an upgrade and a rollback with its data intact.

Sources

• Container registry, GitLab docs
• Unified kernel image, UAPI group specification
• sealed-boot.sh and upgrade-path.sh, the harness in base
• bootc-switch(8) and bootc-rollback(8)
• pipeline/templates/promote.yml, digest re-read and retag
• Pipeline and Sealing, cairn docs

Where I might be wrong

• "Boots twice" is one install boot and one reboot in place of the same disk, firmware variables and TPM state. Whether that reads as two boots on stage is wording to settle.
• The upgrade walk gates only the flagship variant, per base's CI file when I first outlined this. Re-read it at the snapshot.
• Promote re-reads the digest after the registry round trip and verifies it again. I have that from the template's header, not from watching a live run.

Open questions

• Which of the five stations carries the figure's weight: seal or boot?
• Chapter 15 owns the boot's insides. How much of the boot station is drawn here?

Claim

The boot test runs a machine inside a machine: a runner VM built from my own base, and inside it the candidate under enforcing Secure Boot with its own throwaway keys and a software TPM.

Say it this way

The runner is my own OS, and the test machine inside it gets fresh keys, a software TPM and Secure Boot enforcing. If it does not reach login, the tag does not move.

Sources

• Nested guests, KVM wiki; QEMU, Wikipedia
• swtpm, a software TPM emulator; OVMF, UEFI firmware for VMs
• UEFI specifications; UEFI on Wikipedia for Secure Boot and its key databases
• cairn/whetstone, the runner VM image built from base
• sealed-boot.sh, lib.sh and assert-sealed.sh, the harness
• Custom executor, GitLab Runner docs

Where I might be wrong

• The depth. My first outline counted three machines by including the host the runner sits on. The page says nothing about that host, so the claim counts two.
• "No Microsoft keys in db" is true of the harness, which writes PK, KEK and db itself. The metal install enrols differently; that is Part V's subject.
• The runner uses a custom executor that boots a VM per job. I have read the component's tag input and the runner image, not the executor's own configuration, which is private.
• The login wait under nested KVM has a cap in the harness. I read the loop; I did not time a run.

Open questions

• How to draw the runner VM's floor without drawing the hardware under it.
• The serial readout: real console lines, or a schematic of the four events (login, assert, reboot, assert)?

Claim

A gate earns trust only after I broke, on purpose, exactly what it must catch, and watched the run go red.

Say it this way

A green run says nothing about a check that cannot fail. So I break the thing on purpose, and if the run stays green the check is decoration.

Sources

• Mutation testing and negative controls, Wikipedia
• Flaky test, Wikipedia; retry, GitLab docs
• pipeline/.gitlab-ci.yml, retry only on runner failure
• base/scripts/test-host-assert.sh, a probe that runs on every push
• Pipeline, cairn docs, on what a red gate leaves in place
• libvirt, where the boot gates take their probes

Where I might be wrong

• "I never retry a red run." CI retries when the runner broke or the job stalled, never on a script failure. I read the retry blocks in the catalog, not in every consumer.
• The deck names three public red runs by date. I did not reopen those job pages for this draft.
• The health gate's probe (rollback swapped for a no-op, the gate must hold) ran in a lab VM, not in CI. The gate itself ships in base, disabled by default.
• An equivalent mutant proves nothing: a probe the old code also survives. Two of mine were, and the merge request waited until the probes could tell.

Open questions

• A real red job page as the exhibit, or a drawn barrier?
• How much of the health gate belongs here and how much in chapter 27.

Claim

One shared Renovate preset watches every pin in the stack, and the two moves that touch everything, a catalog major and a Fedora release, each land as one transaction.

Say it this way

A bot proposes every bump, and small ones merge themselves after a week's wait and a green run. The big ones land everywhere at once or not at all.

Sources

• Renovate: minimumReleaseAge, automerge, groupName, the regex manager
• dunn.dev/pipeline/renovate-config.json, the shared preset
• base/renovate.json, the regex managers for ZFS, the GPU driver and the Fedora release
• Component versions, GitLab docs, on including by major
• Fedora releases and Fedora bootc
• base/images/base/Containerfile and modules/zfs/Containerfile, two of the pins

Where I might be wrong

• The seven-day wait is for third-party packages. The preset waives it for my own catalog and base images. I read the preset once; re-read it at the snapshot.
• "A major waits for a person" is inferred from automerge being set only for minor and patch, not from a rule that forbids it.
• "Four sites" for the Fedora release: two in base, one in each module builder. Counted in the files today; a built chapter counts again.
• The release number is a moving target. My first outline said 44; the page should say the number at the snapshot, or none.

Open questions

• Does the figure show the bot's merge request or the pins it moves?
• Chapter 12 also claims the catalog-major transaction. Here or there, not both.

Claim

Three layers of firmware run before any OS exists, each in its own silicon with its own owner, and the OS only ever sees a report of them.

Say it this way

By the time Linux gets a say, the boot ROM, the board firmware and every card's option ROM have already run. Linux never reads that flash; it reads what the firmware wrote down about it.

Sources

• Boot ROM, Wikipedia: the first code the processor runs, held in ROM
• UEFI PI spec, SEC phase: the first instructions, before permanent memory
• UEFI spec, runtime services: variables, Secure Boot keys included, live in firmware storage
• Option ROM, Wikipedia: a card's own firmware, run by the host firmware
• TCG PC Client Platform Firmware Profile: the event log the OS receives
• cairn: the trust chain, "What the firmware runs before the OS exists"

Where I might be wrong

• "Three layers" is a teaching cut. A board management controller is a fourth computer with its own flash, and a storage controller may be a fifth.
• Whether the processor verifies the firmware's signature before running it depends on the vendor and the board's configuration. The chapter must say "boards that verify", not "boards verify".
• The claim that a signature database holding only my certificate would stop add-in cards loading is cairn's own finding, checked by one script against my hardware, not a general rule for every card.

Open questions

• How much of the SPI flash layout (vendor region, UEFI code, variable store) to draw without naming a vendor.
• Whether to show the BMC as a peer computer or leave it to a margin note.
• Where the hand-off to chapter 19 sits: at the signature check, or at the first Secure Boot database.

Claim

UEFI replaced "run whatever is there" with a signature check, one vendor's certificate ended up in nearly every firmware, and revoking a signed binary turned out to be the hard part.

Say it this way

A BIOS ran the first sector of the disk and trusted it; UEFI keeps lists of who may boot and who may not. Most firmware trusts Microsoft and boots Linux through shim, so cairn enrols its own keys and skips shim.

Sources

• BIOS, Wikipedia and UEFI, Wikipedia
• UEFI spec 2.10, Secure Boot and driver signing: PK, KEK, db, dbx
• shim and its SBAT.md: generation-number revocation
• CVE-2020-10713, BootHole, and BlackLotus, Wikipedia
• Microsoft: Secure Boot certificates expiring in 2026
• systemd-boot loader.conf, secure-boot-enroll; cairn: the trust chain

Where I might be wrong

• "Nearly every firmware" is a claim about market practice, not a spec requirement. Microsoft's lever was its Windows certification programme; a board can ship without its certificate.
• The 2011 certificates expire on three dates across 2026, not one, and Microsoft says devices keep booting past expiry. "Expiry bricks machines" is wrong and the chapter must not imply it.
• BootHole was a GRUB bug; shim mattered only because shim trusts GRUB. BlackLotus used a hole already patched; it worked because the old signed binaries were not yet revoked.
• "dbx takes one entry per binary" is true of hash entries, which is why SBAT exists; dbx can also hold certificates.

Open questions

• How much of the timeline to draw: 2011 CA, 2020 BootHole, 2023 BlackLotus, 2026 expiry is the candidate set.
• Whether to show PCR 7 here or hold it for chapter 21.
• Whether secure-boot-enroll force and its soft-brick warning belong in this chapter or in the install path.

Claim

A TPM is a second, smaller computer that keeps two keys and a bank of registers, and hands a sealed secret back only while a rule holds.

Say it this way

The host asks the chip; it never looks inside. You give it a secret and a rule for giving it back, and my CI boots against the software one.

Sources

• Trusted Platform Module, Wikipedia: parts, forms, endorsement and storage root keys
• TCG TPM 2.0 Library Specification, Part 1 Architecture
• systemd: TPM2 PCR measurements
• systemd-cryptenroll(1): sealing a disk key to a policy
• swtpm: the software TPM cairn's boot gates run against
• cairn: sealing and docs/tpm-rebind.md

Where I might be wrong

• "Two keys" simplifies. The endorsement and storage root keys derive from seeds the chip holds; the spec describes hierarchies, not two fixed keys. The chapter should say "two roots" if the picture is to hold.
• Which registers clear at reset, and when, is spec detail (Part 1, Architecture). I have not reread the exact section; chapter 21 owns the register sheet.
• Firmware and software TPMs differ from a discrete chip in what an attacker with the board can do. The cut-away should not suggest they are the same thing in a different case.

Open questions

• Whether to show the bus and command set, or keep the chip opaque until the lid lifts.
• How to draw a policy on a signer without pre-empting chapter 24.
• Whether the swtpm detail belongs here or in chapter 15, booting inside CI.

One

A PCR is a register you can only extend: the new value is the hash of the old value and the measurement.

A hash chain: replay the log and the register comes out again.

The TPM holds a bank of them, each the width of a hash. Nothing can write a value in. Firmware and the boot loader feed measurements through extend, one after another, and the event log records what went in and in what order. Each step folds the previous value in, so the final number depends on every measurement and their sequence. Two machines that measured the same things arrive at the same register.

Two

The PC Client map gives each register a job, so a policy can name what it cares about.

Rows 6 and 8 are the TCG PC Client profile's; systemd leaves them undefined.

Firmware measures its code into 0 and its configuration into 1, option ROMs into 2 and 3, the boot loader into 4 and the partition table into 5. The Secure Boot policy lands in 7. systemd-stub measures the UKI into 11 and the kernel command line into 12; the machine's identity goes in 15. systemd-cryptenroll carries the table. The split is the choice: bind a disk to the firmware, the kernel, or both.

Three

A policy on a value breaks on every kernel update. A policy on a signer survives it.

cairn binds the slot to the signer: sealing.

Every new UKI measures to a new PCR 11, so a slot sealed to today's value needs re-enrolling after each update. systemd-measure does the arithmetic ahead of time instead: at build it predicts the PCR 11 the new UKI will produce and signs it with a key I hold. The signature ships inside the UKI as a .pcrsig section, and the slot is enrolled to the public key. At boot the policy asks whether the measured value is one that key signed.

Claim

A signature says who built the image; the seal says the root that booted is the one the signature covers.

Say it this way

Secure Boot checks a signature once, before the kernel runs. The seal puts the root digest on the signed command line and the kernel checks every block it reads against it, so one changed byte fails to read.

Sources

• UEFI spec 2.10, Secure Boot: the signature is checked before execution
• Unified Kernel Image specification: kernel, initrd, command line and policy sections in one signed PE
• composefs and fs-verity: a Merkle tree over fixed-size blocks, usually 4 KiB, checked on read
• ukify(1) and systemd-measure(1)
• cairn: the seal recipe, assert-sealed.sh
• cairn: sealing

Where I might be wrong

• "Every block" is fs-verity's model; what composefs checks per object, and what the kernel checks per page, needs one careful sentence from the fs-verity docs, not my paraphrase.
• The seal proves bytes. It does not stop a verified binary doing something bad; that is chapter 25's point and should not leak forward.
• The CI check compares the digest signed into the UKI's command line against the fs-verity digest on the installed disk, three ways inside the guest. I should quote the script, not summarise it.

Open questions

• Whether the PCR 11 policy appears here as one locked plate or waits for chapter 24.
• How to show "a changed byte fails to read" without drawing a fault that looks like a bug in the figure.
• Whether the hash tree detail is its own figure or an inline drawing in the prose column.

Claim

One offline ceremony mints every key the estate signs with; two stay offline, three ride in CI.

Say it this way

I wanted CI to sign everything and the firmware to trust that signer directly, no shim. One script, run once on a box with no network, mints the keys: platform keys to escrow, signing keys to masked CI variables.

Sources

• cairn: sb-ceremony-mint.sh: offline, enrols nothing, emits keys, payloads, CI blobs and public halves
• Key ceremony, Wikipedia
• UEFI spec 2.10: PK signs KEK, KEK authorises db
• GitLab CI/CD variables: masked and protected variables
• Kernel module signing and cosign signing
• cairn: the trust chain, /keys/, decisions D18 and D29

Where I might be wrong

• Each cairn certificate is self-signed; trust flows through the signed enrolment payloads, not an issuer chain. "PK signs KEK" is shorthand for "PK signs the payload carrying KEK" and the figure should draw it that way.
• The module-signing route: with no shim, the signer lands in the kernel's platform keyring by a Fedora fallback. One path measured on my hardware, per D29. Not a general statement about Fedora.
• Key sizes and a fifty-year validity are choices in the mint script, not requirements. Firmware does not check certificate expiry; I should say so once and move on.

Open questions

• Whether the enrolment order (db, then KEK, PK last) earns a line or a figure mark.
• How to show escrow without drawing anything that looks like a real location or device.
• Whether the cosign key counts as a fifth key on stage or a footnote, since image signing is opt-in per repo.

Claim

The root slot trusts the key that signed the boot measurement, not the measurement itself, so kernel updates survive; and a TPM alone proves what booted, not where the machine is.

Say it this way

Every build signs the expected PCR 11 value and ships it in the UKI; the disk slot trusts the public key, so a new kernel still opens it. Enrol the wrong key and you fall back to the passphrase, degraded, not bricked.

Sources

• systemd-measure(1): signing the expected PCR 11 value
• systemd-cryptenroll(1), --tpm2-public-key-pcrs; systemd-stub(7), the .pcrsig hand-off
• systemd: TPM2 PCR measurements: what lands in PCR 11 and PCR 15
• cairn: assert-pcr11-anchor.sh against the committed pcr11.pub.pem
• cairn: sealing, "Unlock bound to the signer"; cairn: install, the postures
• cairn base README, "Root unlock from a Tang server"; Tang

Where I might be wrong

• The public docs still describe the TPM-bound posture as the main path. On my own hosts I later moved root unlock to Tang and retired the TPM slot, because a stolen machine would unlock itself. The public docs do not say that yet; the chapter has to say which is current.
• A data volume whose unlock measured into PCR 15 failed with a changed-PCR error once root also measured there. I have one case, on one host, and have not confirmed the general cause from the spec.
• "Degraded, not bricked" assumes the recovery passphrase was added first, as the install path says. Skip that step and the claim is false.

Open questions

• Whether this chapter ends on the TPM posture or on the Tang hand-off that Part VII picks up.
• How to draw a "policy on a signer" next to chapter 21's "policy on a value" so the two read as one register sheet.
• Whether the promotion gate on the committed anchor is figure material or one line of prose.

Claim

cairn checks every file it reads; the next step checks every file it runs, and Red Hat is shipping the same chain.

Say it this way

Today a changed byte fails to read; the next step, IPE, refuses to run anything that did not come from a verified volume, and I do not run it yet. Red Hat ships sealed images as a Technology Preview in RHEL 10.2 and calls them where OS management is heading.

Sources

• composefs and fs-verity: checked on read
• IPE, Integrity Policy Enforcement, a Linux security module
• composefs issue 360: IPE integration tracker, open since September 2024
• Red Hat: sealed images in RHEL 10.2
• bootc-composefs(7)
• cairn: ext4 by default, because composefs needs fs-verity and XFS has none; cairn: sealing

Where I might be wrong

• IPE checks what the kernel executes or maps. A script run by an interpreter is checked only if the interpreter asks; I have not tested how far that reaches.
• "Red Hat ships the same chain" rests on one blog post. The release notes naming the Technology Preview were not readable anonymously when I checked, so I link the post and not the notes.
• "Not in what I run" is a grep of the base repo for the word, not a boot-time check that the module is absent from the kernel.

Open questions

• Whether to draw the RHEL stack as a ghost twin behind mine or keep the comparison in prose.
• How to show "read" and "run" as two rungs of one ladder without inventing a layer that does not exist.
• Whether this chapter closes Part V or hands straight to day two.

Claim

CI stops at a promoted image. A person pulls it into the spare slot and reboots.

Say it this way

The factory ships a tag and nothing in CI touches a running host. I run bootc upgrade, the new image lands in the spare slot, and the reboot is my hand on the lever.

Sources

• bootc upgrade: download and queue, visible as staged
• bootc status: booted, staged, rollback
• bootc rollback: boot the previous deployment
• OSTree deployments: the stateroot shares one /var
• OSTree atomic rollbacks
• cairn, running: the cascade stops at a promoted image
• cairn/base: how a variant earns :stable-sealed

Where I might be wrong

• I say the swap happens at shutdown. That is bootc's default today, and its manual says it is likely to change so that only bootc upgrade --apply applies an update. The built chapter should pin the version it describes.
• Two slots is a picture, not a limit. OSTree keeps a list of deployments, and a pinned one makes a third beside booted and rollback.
• The signature check at pull is a lane choice. The evidence lane bakes a policy that bootc upgrade enforces; a plain lane pulls without one.

Open questions

• Chapter 5 measured about 5% re-shipped for a same-commit rebuild. What travels on a Fedora release bump? Measure one before claiming a number here.
• Does the manual reboot read better on stage than --apply, or does it look like a step I forgot to automate?
• What does the operator see when the pull is refused by the signature policy? The refusal is the point; the chapter has no exhibit for it yet.

Claim

A new deployment stays only while its required checks pass. Three failed boots roll it back, and with nowhere to go the host holds.

Say it this way

bootc does not count boots yet, so I count them: three strikes and the previous image comes back. With nothing to roll back to, the host halts and waits for a person, because a reboot loop is worse than a halt.

Sources

• systemd: boot-complete.target
• systemd: OnFailure= and Requisite=
• systemd: automatic boot assessment, the idea this gate borrows
• cairn/base: run-checks.sh, required.d must pass, wanted.d is logged
• cairn/base: on-failure.sh, the counter, the threshold, the hold
• cairn/base: confirm.sh, reset only on a confirmed boot
• cairn/base README: health gate, shipped disabled

Where I might be wrong

• "bootc keeps no boot counter" was true when I wrote the gate. If bootc or systemd's boot assessment now covers it, the gate is a stopgap and the chapter should say so.
• The counter survives its own reboots because OSTree shares one /var per stateroot. I have not checked that every cairn variant keeps that default.
• Rollback, hold and a hung check were proven in a lab VM, not on metal. The bound on a hung script landed after that run and is not re-proven.
• The gate ships disabled. I should not imply any fleet runs it.

Open questions

• Does this belong upstream, in bootc or in systemd's boot counting, rather than as scripts in the image?
• How does a host choose its required checks, and who decides that a check is required rather than wanted?
• Three failed boots means three measured boots. Does the gate interact with anything sealed to PCR values, or is the rollback path clean?

Claim

One volume key encrypts every sector; LUKS2 keeps that key wrapped in slots, and a cairn volume keeps two of them, a typed recovery passphrase and a clevis token.

Say it this way

The disk has one key, and the header holds copies of it, each locked with a different secret. I keep two locks: the words I type, and a question the host asks the network.

Sources

• dm-crypt, kernel documentation
• LUKS, Wikipedia
• LUKS2 on-disk format, cryptsetup
• cryptsetup(8)
• clevis
• cairn/base README, root unlock from a Tang server

Where I might be wrong

• clevis does not wrap the volume key itself. It mints a strong passphrase, adds it as a keyslot, and encrypts that passphrase with the pin. The figure's "three wrappings of one key" is one level short of the truth.
• I say the passphrase slot runs Argon2. That is cryptsetup's LUKS2 default as I remember it; the built chapter has to cite the version that set it.
• The reasoning for dropping the TPM slot is in chapter 24. The public record of that decision is thin.

Open questions

• Draw three slots and strike one out, or draw the two that exist?
• The header's token metadata is readable without a key. What does a Tang URL in the clear tell a thief?
• Does the figure need to show the clevis-minted passphrase as its own block?

One

Tang publishes a public key. The client fetches it once at enrol time, and that is all it keeps.

GET /adv is the advertisement; POST /rec/{kid} is one answer.

Tang is a small HTTP service with two endpoints. GET /adv returns a signed set of public keys. One is the exchange key S; its private half s never leaves the server. No client is registered and nothing is stored per host. Tang keeps no per-host state: the whole server state is its signing and exchange keys.

Two

Binding: a random key, one elliptic-curve step, and the private half is thrown away.

clevis picks a random scalar c and its public point C = c·G. One ECDH step with the server's key gives K = c·S, and K wraps the LUKS passphrase. Then c and K are deleted. What stays on the disk is C and the wrapped blob. From those, K needs c, which is gone, or s, which only the server has.

Three

Unlock: the client blinds C before it asks, so the server works on a point it never saw.

The McCallum-Relyea exchange, in the Tang README.

At boot the client picks a fresh random e and sends X = C + e·G. The server returns Y = s·X, which is s·C + e·s·G, which is K + e·S. The client knows e and S, subtracts e·S, and has K back. The server did one multiplication and learned nothing: X is random to it, and so is Y.

Four

The server never holds the disk key. Deleting s on the server is the revoke.

Steal the server and you have s, which alone unlocks nothing: K also needs C, and C is on the disk. Steal the disk and you have C and the blob, which need the server. keystone runs Tang as a Cloudflare Worker; a revoked key set answers 404, and a host that already unlocked keeps its key until it reboots.

Claim

The initrd brings up a link, resolves a name and asks Tang for the key before root exists, and it waits for the answer without a timeout.

Say it this way

Before there is a root filesystem there is a small Linux in memory, and I teach it to take a lease, look up one name and ask one question. If nobody answers it keeps asking instead of dropping you into a rescue shell.

Sources

• Kernel arguments, bootc
• dracut.cmdline(7), rd.neednet
• dracut systemd-networkd module
• systemd-networkd-wait-online, --any
• systemd.unit(5), JobTimeoutSec
• cairn/base tang-initrd

Where I might be wrong

• "No timeout" is one drop-in: JobTimeoutSec=infinity on the root device unit, in the initrd only. Other clocks still run, wait-online's own and DHCP's. I have not traced what each does when the server stays silent for an hour.
• clevis-dracut pulling in NetworkManager is a packaging dependency at the time the script was written. If the package splits, the builder stage is doing work for nothing.
• I say resolved takes DNS from the lease. That is the script's own comment, not something I watched on the wire.

Open questions

• Re-measure the hold with the server down. The lab run kept the prompt up for about two minutes; the built chapter needs a number it can cite.
• Does wait-online --any race the askpass path on a machine with two links?
• Show the catch-all .network file and its mask in the root, or leave that to the text?

Claim

One Cloudflare Worker serves many named key sets with stock tangd's protocol; a set is written once, and deleting its secret is the revoke, after which rec answers 404.

Say it this way

Tang is a small protocol: publish a public key, answer one blinded question. That fits in a Worker, where a key set is one secret, rotation is a new name and revoke is delete.

Sources

• Tang README, the protocol and the maths
• keystone
• keystone setup, rotate and revoke, reference
• keystone changelog
• Worker secrets, Cloudflare
• RFC 7517, JSON Web Key

Where I might be wrong

• "Stock tangd's protocol" rests on clevis binding against it in the image build and in the lab. There is no conformance suite; a client other than clevis has not been tried.
• Secrets cannot be read back through the API, but Worker versions carry them. A rollback past a revoke serves the set again. "Delete is revoke" holds only with the tombstone check keystone's deploy runs.
• P-521 needing the paid plan comes from keystone's own timing against the free plan's CPU limit, not an independent measurement.

Open questions

• How much of the Worker is Tang and how much is the gate from chapter 32? A line count would settle what the figure gives each.
• Does the tombstone belong in this figure or the next?
• Where does the keyset tool stand in the picture: mint, check, deploy, revoke?

Claim

A gated set recovers a key only for a network I named, or one an approver admitted for ten minutes; everything else is refused before a private key is read.

Say it this way

The server used to answer anyone; now a set carries a list of networks, and a request from off the list is held while someone gets an email. Press approve and that network is in for ten minutes; everything else is a 403.

Sources

• keystone README, security model
• keystone setup, origin gate
• keystone src/index.js
• Durable Objects, Cloudflare
• Email Routing, Cloudflare
• CIDR, Wikipedia

Where I might be wrong

• "Before a private key is read" is what the handler does at one commit. Nothing pins the order; a refactor could move it. The built chapter should point at a test that does.
• A held host unlocks in about fifteen seconds because clevis-luks-askpass keeps retrying. That number is keystone's own documentation, not a stopwatch.
• The gate narrows who reaches the key. It is not authentication: whoever reads an approver's mailbox holds that power until the request expires.

Open questions

• Hold-then-approve as one node of the decision tree or two?
• Where do the caps go: three held origins per set, five mails an hour per approver, a pending request that expires after twelve hours?
• Is deny worth a branch, given it refuses the same way as never answering?

Claim

A stolen disk opens only while the Worker answers it, so the thief's list is the disk plus one thing I still control.

Say it this way

Take the disk; what else do you need? An address I allowed, an approver's inbox or my Cloudflare account, and each of those is mine to pull.

Sources

• keystone README, security model
• keystone setup, rotate and revoke
• Rollbacks, Cloudflare Workers
• Workers Logs, Cloudflare
• Workers limits, Cloudflare
• WAF rate limiting rules, Cloudflare

Where I might be wrong

• "One thing I still control" fails for a thief who takes the whole machine and plugs it in on an allowed network. That is the trade the allow list makes; the bill is then just the disk, until I notice.
• Revoke protects a disk at rest. A host that already unlocked keeps its key until it reboots.
• A rollback past the revoke serves the set again. The platform does not stop that; keystone's deploy check does.
• "Never data" holds only while the recovery passphrase lives somewhere other than the stolen machine.

Open questions

• Does the bill list the domain and its DNS, which the Worker URL depends on?
• How to draw "already unlocked keeps its key" without a running machine in the frame?
• Does a quota flood belong here as denial, or in chapter 32 as a cap?

Claim

Every claim in this walkthrough sits in one of three trays, proven live, proven in a lab, or open, and each carries a date.

Say it this way

Here is the ledger: what a machine I run has booted, what a lab VM has passed, and what I can prove and cannot yet, each with a date on it.

Sources

• cairn/pipeline v19.1.0, 31 October 2026
• CI/CD components, GitLab docs
• cairn/base !149: the health gate as a disabled opt-in
• Mutation testing, Wikipedia
• Tang
• keystone #3: a walk from a fresh account

Where I might be wrong

• The snapshot is 31 October and the ledger has already moved. The catalog shipped v19.2.0 on 1 November and the health gate merged into base the same day. The built chapter needs a rule for which date a row shows.
• "Proven live" for the Tang root unlock rests on logs from hosts I run. A reader can check a release page and a merged request; they cannot check that my host booted. The row should say so.
• The health gate's lab run turned every planted fault red, but the fix for a hang found in that run has not been re-proved. One lab row is a half row until it is.

Open questions

• Which date goes on a row: the day the receipt first existed, or the day it was last re-checked?
• What counts as a receipt a stranger can check? A release page and a merged request do; a host log does not. Does the ledger show the difference?
• Does the ledger live only here, dated once, or also on the cairn site, where it would have to stay current?

Claim

A fork of cairn/example is a working instance in one push, and the ladder above it is a list of things to bring, one per rung.

Say it this way

Fork the example, keep one lane, point it at your registry, push; CI builds it and promotes it. Everything above that is one more thing to bring, and you can stop at any rung.

Sources

• cairn/example: four lanes, "Fork and roll"
• Running it, cairn.dunn.dev
• CI/CD components, GitLab docs
• The include: keyword, GitLab docs
• OCI distribution spec: what a registry is
• bootc install

Where I might be wrong

• "One push" assumes a GitLab project with a container registry and a runner that can build images. The example's include blocks also carry runner tags that fit my runners; a fork changes them or its first pipeline waits forever.
• The seven rungs come from the deck's figure, not from anything the repos enforce. The example's lanes are postures side by side: plain, attended and edge climb one disk-encryption ladder, and evidence composes with any of them. Seven rungs and four lanes have to be reconciled.
• The instance component never boots the image. A green build against a promoted base tag is the criterion, so a fork can promote an image its own Containerfile broke.
• The one-machine door, bootc install with no CI, rests on upstream docs. I have not walked it from a fork of the example.

Open questions

• How many rungs, and which lane stands on which?
• What does a fork change beyond instance_image: runner tags, registry login, the base variant?
• Where does a reader with one machine and no GitLab stop, and is that a door or a dead end?