On Building a Complicated Foot Gun

I’ve wanted an immutable server OS for a long time. Fedora Silverblue showed what the desktop side could look like: atomic updates, rollback in one command, no configuration drift. Fedora CoreOS brought that model to servers, but honestly, the types of hosts I manage made it a difficult fit. CoreOS is built around ephemeral infrastructure. Machines are cattle, not pets. You provision them declaratively with an Ignition file, they auto-update and reboot themselves, and if one dies you recreate it from the config. That works well for a Kubernetes cluster or a fleet of interchangeable nodes. My infrastructure is the opposite: a single long-lived bare metal server with ZFS pools that represent irreplaceable state (pool topology, snapshot schedules, replication history) and NVIDIA GPUs with persistent compute configuration. This is pet infrastructure, and CoreOS’s philosophy assumes you don’t have pets.

The technical gaps reinforced the philosophical ones. Out-of-tree kernel modules like ZFS and NVIDIA’s proprietary driver had no clean path on ostree-based systems for years. The standard Fedora approach (akmod) compiles modules at boot against the running kernel, but ostree’s immutable root filesystem makes that impossible. There are no kernel-devel headers on the deployed system, and you can’t write to /lib/modules. CoreOS’s automatic kernel updates made it worse: even if you hand-loaded modules, the next update could deliver a new kernel with no mechanism to rebuild them before reboot. ZFS was particularly stuck because its CDDL license means it can never ship in the kernel or in RPM Fusion, so the usual Fedora packaging paths don’t apply.

The breakthrough was ostree native container support (2022-2023), which let you define your entire OS as an OCI container image built in CI. Kernel modules compiled during the image build, signed, baked in. Universal Blue proved this at scale with projects like uCore, shipping Fedora CoreOS derivatives with both NVIDIA and ZFS pre-compiled. I used uCore and appreciated what it proved possible. But their NVIDIA packaging depends on negativo17’s upstream repository, and I’d rather package directly from NVIDIA’s distribution when I can control the build. Their akmods pipeline is impressive engineering, but it’s another link in a supply chain I don’t control.

When bootc landed with official CentOS Stream base images, the calculus changed. bootc keeps the atomic update model but drops the ephemeral assumptions. Your OS is a container image. You build it with a Containerfile. bootc install to-disk writes it directly to a physical disk. bootc upgrade pulls the next version. No Ignition files, no netboot infrastructure, no HTTP server hosting provisioning configs. The image is the source of truth for what the system looks like, and /etc and /var (your state, your ZFS pools, your machine-specific config) persist across updates. Long-lived hosts become manageable without pretending they’re disposable.

Reading mrguitar’s posts about running bootc with GPU drivers on CentOS Stream and Fedora convinced me this was ready to build on. I started Immutable Base, a GitLab project that builds hardened CentOS Stream 10 images with ZFS kernel modules and NVIDIA drivers compiled directly from NVIDIA’s packages and baked in. Instance projects layer host-specific configuration on top. The first instance was carmine, a single physical server running infrastructure for my family.

What we built

Carmine runs self-hosted services for my family. Authentik for SSO. vLLM for local AI inference. Vaultwarden for passwords. Immich for photos. GitLab for source control. Mealie for recipes. Each service runs as its own non-root user in rootless Podman containers with Caddy sidecar proxies. Envoy handles L4/L7 edge routing.

The configuration management layer is pyinfra (agentless, SSH-based). Static files, no templates. Secrets are stored in Bitwarden so my family can find them if something happens to me, but pyinfra pulls them from Bitwarden during deployment and persists them on host via podman secret. The whole deployment is in git. The image supply chain has cosign signatures, CycloneDX SBOMs, and signed attestations. Storage is mirrored-topology ZFS pools across NVMe and SAS drives with zrepl replication and off-site backups.

Six months of refining these ideas, drawing on 10+ years of running different distributions from Debian to Arch to the Red Hat ecosystem. We tried to pour every gotcha we’d accumulated into the design. The harness exploration earlier this year changed the velocity of all of it significantly. I finally scheduled a Friday to deploy. Delightfully, we found new gotchas.

The deployment

Friday afternoon. Installed the image from a live USB onto NVMe. First boot worked. SSH in, run pyinfra to harden the system: SSH lockdown, nftables, sysctl tuning, SELinux, NetworkManager. Started deploying the first service.

By the time I wrapped up for the day, concerns were surfacing. The pyinfra validation was weak. We were testing that quadlet files parsed correctly and that the pyinfra code linted clean, but we weren’t validating what the system looked like after pyinfra ran. I made a note to circle back to that.

I rebooted. The system never came back. No ping. No SSH. No network at all. ARP-level unreachable, the NIC was not even responding at Layer 2.

I reproduced this three times, each time assuming the problem was in my pyinfra harness. I’d revert changes, re-run the deployment, reboot, and lose the system again. The sticky part was the non-determinism: sometimes the system would survive a reboot. Sometimes it wouldn’t. That inconsistency kept pointing me toward my own configuration as the culprit, not something deeper.

The investigation

The server has a BMC with console access. I got a console session up and could see the system hanging post-boot, sitting at what appeared to be a login prompt that never fully rendered. Services were not starting.

I dropped into rd.break (break into the initramfs before switch_root) and confirmed: initramfs is fine, kernel is fine, root filesystem mounts and looks correct. The hang happens after the root filesystem takes over, somewhere in the systemd boot sequence.

To get proper forensics I booted a Fedora CoreOS 43 live USB, port-forwarded SSH through a Ubiquiti router, and mounted the dead system’s root filesystem. Now I could read journal logs from the failed boots.

The smoking gun was in the journal:

dbus-broker-launch[2193]: ERROR launcher_run_child @ ../src/launch/launcher.c +326: Permission denied

dbus-broker was crashing immediately after start. Without dbus, nothing works. NetworkManager needs dbus (no network). systemd-logind needs dbus (no login). Every service that registers on the bus stalls waiting for a connection that will never arrive. The system appears to hang because everything is blocked on a dead bus.

Tracing the +326 offset in launcher.c pointed to sd_id128_get_machine(), which reads /etc/machine-id. The dbus-broker-launch process forks a child that drops privileges to UID 81 (the dbus user), and that child tries to read the machine ID file. I checked the SELinux label on /etc/machine-id:

ls -Z /etc/machine-id
# system_u:object_r:container_file_t:s0 /etc/machine-id

The label is container_file_t. It should be etc_t. The SELinux policy for dbusd_t (the domain dbus-broker runs in) allows reading etc_t but denies reading container_file_t. When the child process drops to UID 81 and tries to read the file, SELinux blocks it with EACCES, and dbus-broker exits fatally.

The full crash chain:

bootc install (runs inside a container context)
  -> creates /etc/machine-id with container_file_t label
  -> system boots, dbus-broker starts
  -> dbus-broker-launch forks child, child drops to UID 81
  -> child calls sd_id128_get_machine() -> reads /etc/machine-id
  -> SELinux denies dbusd_t reading container_file_t -> EACCES
  -> dbus-broker exits fatally
  -> NetworkManager, logind, everything stalls on dbus
  -> system hangs: no network, no login, no SSH

The twist: I checked the journal from the first boot (the one that worked) and found the same mislabeled file. setroubleshoot logs showed hundreds of systemd-userwork AVC denials reading container_file_t on machine-id throughout that entire session. The bug was present from the first moment. It manifested non-deterministically based on SELinux AVC cache timing. The first boot happened to cache the allow decision before dbus-broker needed it. The second boot did not.

The upstream story

Once I understood the root cause, finding the upstream issue was straightforward. bootc#1647 (“Ensure layers are canonically labeled”) has been open as a meta-issue tracking exactly this problem: the install process runs inside a container, and files it creates inherit container_file_t labels instead of the labels the SELinux policy expects. The bootc maintainers knew about this. They had been working on it. I had spent months building a supply chain project on top of bootc and never seriously evaluated bootc’s own open issues. I was too focused on getting kernel modules built, images signed, and services deployed to look closely at the foundation I was building on.

Three PRs addressing various aspects of the labeling problem were merged in the week leading up to my deployment:

• #2025 (Feb 26): deferred relabeling of container storage
• #2045 (Mar 6): walk API with noxdev
• #2035 (Mar 7): full SELinux relabel pass as the final install step

The last one would have prevented my specific failure. It was merged to bootc main two days before my Friday deployment, but the latest bootc release is v1.13.0 from February 23, before any of these fixes. Even when they ship as a bootc release, CentOS Stream 10 has to package it, immutable-base has to pick up the new package, and carmine has to rebuild. That chain has real latency, and none of it was the upstream team’s fault. They were doing their job. I just wasn’t paying attention to the right part of my own supply chain.

An interesting aside: #2035’s commit message notes “Assisted-by: OpenCode (claude-opus-4-6).” The upstream developers were using AI tooling to fix this. I was using the same tooling (OpenCode with Claude) for the investigation itself, fetching upstream source, mapping error line offsets to function calls, cross-referencing journal entries across multiple boots, and pulling up the relevant GitHub issues and PRs. AI as a pair-debugging partner for systems forensics turned out to be genuinely useful here.

The foot gun

The root cause of the outage is the SELinux mislabeling bug. But the root cause of the impact is architectural. I built a sophisticated supply chain and had zero verification that the final installed system actually boots.

The image pipeline:

CentOS Stream 10 (quay.io)
  -> immutable-base (GitLab CI)
    -> carmine (GitLab CI)
      -> install to disk
        -> ???

That ??? is the foot gun. The CI pipeline builds the image, signs it with cosign, generates SBOMs, validates quadlet files, and lints the pyinfra code. What it does not do is test that the image boots after being installed to disk.

The mislabeling happens during the install process, not during the image build. No amount of container-level testing catches it. You can run the image as a container, start services inside it, run a full integration test suite, and everything passes. The bug only exists in the installed system because the install process itself introduces it.

The more things you test, the more confident you feel. I was testing several things: image builds succeed, signatures verify, SBOMs generate, quadlet syntax is valid, pyinfra lints clean. None of those things covered “does the installed system come up with a working dbus.” The sophistication of the pipeline created confidence that was not warranted. That is the foot gun. The more complicated you make the system, the more surface area exists for failures you did not think to test, and the more your passing tests convince you that you have coverage you do not actually have.

What we are building now

Image version pinning

The trust chain flows from the source. Immutable-base should have :latest (CI auto-builds on every commit) and :stable (manually promoted after testing). Instance projects like carmine consume :stable, never :latest. Each source distro (CentOS Stream, Fedora) is a separate stability lineage with independent tagging. A bad base image build breaks :latest but never reaches :stable until someone verifies it. The CI template in the carmine project was already pinned to a specific version (v1.2.0), but the actual container image reference was not. Both need to be pinned, and both need to be updated deliberately.

QEMU multi-boot smoke test in CI

This is the one that would have caught this exact bug. A CI job that:

1. Installs the image into a qcow2 disk image
2. Boots the qcow2 in QEMU (headless, serial console)
3. Waits for SSH to become available
4. Runs health checks: is dbus alive, is NetworkManager up, does systemctl is-system-running return running
5. Reboots the VM and runs the checks again
6. Pass/fail gates the :stable tag promotion

Step 5 is critical. My bug only manifested on the second boot. A smoke test that boots once and checks SSH would have passed. The test needs KVM access, which means bare metal or nested virtualization CI runners. Worth the infrastructure cost.

restorecon as defense in depth

Add restorecon -Rv /etc/ as the very first pyinfra operation. This resets SELinux labels on /etc to whatever the loaded policy says they should be. It fixes any mislabeled files before anything else runs. It is harmless on correctly-labeled systems and respects semanage fcontext customizations. Even after the upstream fix ships, this costs almost nothing and catches an entire class of labeling drift.

.autorelabel in the Containerfile

RUN touch /.autorelabel in the Containerfile triggers a full SELinux relabel on first boot. About 30 seconds of additional boot time. Open question: does bootc preserve this file through the install process? If it does, this is a simple backstop. If it does not, the QEMU smoke test catches it anyway.

If you are building critical infrastructure on bleeding-edge technology, you need to engineer your own safety net. Upstream’s safety net does not exist yet. That is not a criticism of upstream. bootc is actively developed, the maintainers identified and fixed this bug, and the fix is thorough. But the fix has not shipped in a release, the release has not been packaged by the distro, and the distro package has not been picked up by my base image. That chain takes time. If your family’s infrastructure depends on the output, the gap between “fix merged” and “fix deployed to your system” is your responsibility to cover.