bulkhead — andrew.dunn.dev

Bulkhead is my public, vendor-neutral reference project for running least-privilege self-managed GitLab CI runners on a fixed set of RHEL-family Linux hosts: AlmaLinux 8, 9, and 10. No cloud autoscaler, no dynamic scaling, just VMs an administrator provisions once and keeps. I was exploring how best to isolate runners when Kubernetes isn’t in the picture, and around the same time a customer asked nearly the same question, starting from RHEL 8. Bulkhead is where the two met: containment shown holding on the host family in question, rather than asserted from a policy document. Anyone operating self-managed GitLab Runner on RHEL-family hosts who needs to isolate CI jobs from each other and from the host runs into that question eventually, and Bulkhead answers it with one small demo workload, a Flask service, run through two runner postures side by side: one for ordinary CI, one for jobs that genuinely need root, which a least-privilege default cannot grant without becoming something else entirely. It stands on the same rootless-Podman ground as two other things I’ve written about here: the subordinate UID range each container runs inside is the same mechanism behind a filesystem quirk in Rootless Podman pods leaking subordinate UIDs onto the host filesystem, and the question neither runner posture here settles, whether job containers can reach each other over the network, is the one Network Plane Isolation in Rootless Podman actually works through.

A CI job runs under rootless Podman by default, or a throwaway VM when it needs root, and the host cgroups version decides whether Podman's CPU and memory caps hold.

The runner-matrix pipeline sends one Flask app down two tags: the rootless Podman runner builds it inside a user namespace with an SELinux label and resource caps, the ephemeral-VM runner boots a throwaway libvirt guest for root jobs and deletes it after, and the host cgroups version decides whether those caps are real.

The rootless Podman runner, the default posture for ordinary CI, points GitLab Runner’s Docker executor at a per-user rootless Podman socket, so there is no root daemon in the loop: a Linux user namespace with a subordinate UID range, an explicit SELinux MCS label rather than disabling it, and CPU and memory caps set in the runner config. The ephemeral-VM runner, for the jobs rootless containment can’t cover, points GitLab Runner’s Custom executor at shell hooks that clone a copy-on-write VM image, boot it under libvirt and QEMU, stream the job script in over SSH, then destroy the VM and delete its disk; in one measured run the VM was up at 36 seconds and gone by 66.

The project’s central finding, the cgroups trap, lives entirely inside the rootless Podman runner. Podman’s --memory and --cpus flags are enforced only on cgroups v2: AlmaLinux 8 defaults to cgroups v1, where the same flags print a warning and are then silently accepted and ignored, while AlmaLinux 9 defaults to v2 and enforces the identical limit exactly. AlmaLinux 10 removes the v1 hierarchy from the kernel entirely, so the failure that could ship silently on AlmaLinux 8 becomes structurally impossible on 10, not merely off by default.

I found the cgroups trap in the lab, running rootless Podman against real AlmaLinux 8 and 9 hosts, and the evidence files sit in the repo next to the finding. It’s independent of any customer work, built on my own demo Flask app, with no customer data anywhere in it.