Kernel Modules as Multi-Stage Container Builds

bootc images are container images that become your operating system. You build them with a Containerfile, push to a registry, and deploy to bare metal. The image is the system: every package, every file, every config. Updates are atomic image swaps, not package-by-package mutations.

ZFS and NVIDIA are out-of-tree kernel modules. They need to be compiled against the exact kernel version in your image. The traditional approach is DKMS: install dkms, gcc, kernel-devel, and the module source, then build at image build time.

That approach leaves ~500MB of build tooling in your production OS image. On an immutable system, that tooling serves no purpose after the build completes. It is attack surface you do not need, weight you do not want, and a violation of the principle that the image should contain exactly what runs in production and nothing else.

This post describes a pattern for building kernel modules in multi-stage container builds that keeps the build tooling in throwaway builder stages and copies only the compiled modules and userspace tools into the final image. The pattern came out of building immutable home server infrastructure and debugging a bootc bug that taught me how much the build pipeline matters for immutable systems.

The pattern

The build has three stages. Two throwaway builder stages compile the kernel modules (one for ZFS, one for NVIDIA). A final stage assembles the production image by copying only the compiled output.

Both builders carry the full toolchain and neither survives the build. Only the compiled modules, the userspace tools and their manifest cross into the production image.

The builders are full container images with the complete build toolchain. They compile the modules, install the userspace tools, generate a manifest of installed files, validate their own output, and then are discarded. The final image never sees gcc, make, dkms, or kernel-devel.

Kernel version pinning

The builder and the final image must use the exact same kernel version. If the builder compiles modules against kernel 6.12.8 and the final image runs 6.12.9, the modules will not load. DKMS exists to solve this problem at boot time, but on an immutable system the kernel should never change without a new image build.

The CI pipeline handles this by querying the available kernel-devel version from the repos and passing it as a build argument:

# CI upstream-sync stage: query available kernel-devel
KMOD_KERNEL=$(buildah run "${CONTAINER}" -- bash -c \
  "dnf repoquery --available --latest-limit=1 \
   --qf '%{VERSION}-%{RELEASE}.%{ARCH}' kernel-devel \
   2>/dev/null | tail -1")

dotenv artifactsGitLab CI dotenv artifacts pass variables between pipeline stages. The upstream-sync job writes KMOD_KERNEL to build.env, and downstream jobs inherit it automatically.

Both the builder and the final image pin to this version. If the base image ships a newer kernel, the Containerfile downgrades it:

ARG KMOD_KERNEL
RUN CURRENT=$(rpm -q --qf "%{VERSION}-%{RELEASE}.%{ARCH}\n" kernel-core) && \
    if [ "${CURRENT}" != "${KMOD_KERNEL}" ]; then \
        PKGS="kernel-${KMOD_KERNEL} kernel-core-${KMOD_KERNEL} \
              kernel-modules-core-${KMOD_KERNEL} \
              kernel-modules-${KMOD_KERNEL} \
              kernel-modules-extra-${KMOD_KERNEL}" && \
        CMP=$(rpm --eval \
          "%{lua:print(rpm.vercmp('${CURRENT}','${KMOD_KERNEL}'))}") && \
        if [ "$CMP" = "1" ]; then \
            dnf downgrade -y ${PKGS}; \
        else \
            dnf upgrade -y ${PKGS}; \
        fi; \
    fi

The rpm.vercmp Lua function handles RPM version comparison correctly, including epoch and release fields. Comparing version strings with shell operators would break on versions like 6.12.8-200.fc42 vs 6.12.10-100.fc42.

This pattern handles both directions: downgrade when the base image is ahead, upgrade when it is behind. The CI pipeline verifies the kernel match after building by extracting the built kernel version and comparing it to the target.

DKMS build in the builder

The builder installs the full build toolchain, the module source, and runs the DKMS build:

# From build-kmod-nvidia-open-dkms.sh
ARCH=$(uname -m)
K_VRA=$(rpm -q --qf "%{VERSION}-%{RELEASE}.%{ARCH}\n" kernel-core)

# Add NVIDIA CUDA repository
curl https://developer.download.nvidia.com/compute/cuda/repos/\
rhel${VERSION_ID}/${ARCH}/cuda-rhel${VERSION_ID}.repo \
    -o /etc/yum.repos.d/nvidia.repo

# Install DKMS package (source + build scripts)
dnf install -y kmod-nvidia-open-dkms

# Build and install against the target kernel
DRIVER_VERSION=$(rpm -q kmod-nvidia-open-dkms --qf '%{VERSION}')
export MAKEFLAGS="-j$(nproc)"
dkms build -m nvidia -v $DRIVER_VERSION -k $K_VRA --verbose
dkms install -m nvidia -v $DRIVER_VERSION -k $K_VRA --verbose

The -k flagThe -k flag is critical. Without it, DKMS targets the running kernel (the container host’s kernel), not the kernel installed in the image. The builder’s kernel-core RPM provides the target version.

The -k flag tells DKMS to build against a specific kernel version rather than the running kernel. This is essential in a container build where the host kernel (the CI runner) is different from the target kernel (the bootc image).

The ZFS build follows the same pattern with the OpenZFS repository:

# From build-kmod-zfs-dkms.sh
dnf install -y \
    https://zfsonlinux.org/epel/zfs-release-2-8\
$(rpm --eval "%{dist}").noarch.rpm
dnf install -y zfs-dkms

ZFS_VERSION=$(rpm -q zfs-dkms --qf '%{VERSION}')
export MAKEFLAGS="-j$(nproc)"
dkms build -m zfs -v $ZFS_VERSION -k $K_VRA --verbose
dkms install -m zfs -v $ZFS_VERSION -k $K_VRA --verbose

After dkms install, the compiled .ko files are in /usr/lib/modules/${K_VRA}/extra/. The builder’s job is not done yet: it also needs to install the userspace tools.

Manifest-driven file copy

Why not COPY --from?A naive COPY --from=builder /usr /usr copies everything in the builder’s /usr directory into the final image, including gcc, make, kernel-devel, and every other build dependency. The manifest approach copies only the files that belong to the packages you want.

The builder installs userspace tools (like nvidia-smi, zfs, zpool) using rpm --nodeps to avoid pulling in DKMS as a runtime dependency. Then it generates a manifest of exactly which files those packages installed:

The RPM database already knows which files each package installed, so the final stage copies that list rather than a whole directory tree. Copying /usr wholesale would bring the build toolchain with it.

# Install userspace tools without DKMS dependency chain
dnf download --destdir=/tmp nvidia-driver nvidia-driver-cuda
dnf download --destdir=/tmp libnvidia-ml

# Resolve library dependencies dynamically
NVIDIA_LIB_DEPS=$(rpm -qRp /tmp/nvidia-driver*.rpm \
    | grep -E '^nvidia-driver-(cuda-)?libs' \
    | awk '{print $1}' | sort -u)
for dep in $NVIDIA_LIB_DEPS; do
    dnf download --destdir=/tmp "$dep"
done

# Install all packages without dependency resolution
rpm -ivh --nodeps /tmp/nvidia-driver*.rpm /tmp/libnvidia-*.rpm

# Generate manifest from RPM database
INSTALLED_PACKAGES=$(rpm -qa \
    | grep -E '^(nvidia-driver|nvidia-modprobe|nvidia-kmod-common|libnvidia-)' \
    | sort)
for pkg in $INSTALLED_PACKAGES; do
    rpm -ql "$pkg"
done > /nvidia-files.txt

Library resolutionThe rpm -qRp query extracts library dependencies from the downloaded RPM before installing it. This is resilient to upstream package name changes: if NVIDIA renames nvidia-driver-libs to nvidia-driver-libs-560, the query still finds it.

The manifest is a plain text file listing every file path that belongs to the installed packages. The final stage copies only those files:

RUN --mount=type=bind,from=nvidia-builder,target=/tmp/nvidia-builder-root \
    while IFS= read -r file; do \
        [ -z "$file" ] && continue; \
        if [ -f "/tmp/nvidia-builder-root${file}" ]; then \
            mkdir -p "$(dirname "${file}")"; \
            cp -a "/tmp/nvidia-builder-root${file}" "${file}"; \
        elif [ -d "/tmp/nvidia-builder-root${file}" ]; then \
            mkdir -p "${file}"; \
        fi; \
    done < /tmp/nvidia-files.txt

--mount=type=bindBuildKit bind mounts mount a builder stage’s filesystem into the current stage without copying it into a layer. The files are available during the RUN command but do not persist in the image unless explicitly copied.

This gives you nvidia-smi, the shared libraries, the GSP firmware files (required for Turing+ GPUs), the systemd units, but not gcc, not dkms, not the kernel source tree.

The ZFS manifest follows the same pattern, capturing zfs, zpool, the shared libraries (libzfs, libzpool, libnvpair, libuutil), systemd units, udev rules, and the zfs-import-cache service.

Builder validation

Each builder validates its own output before the final stage copies it. Three checks:

Module presence: modinfo confirms the compiled module exists and reports its version:

depmod -a ${K_VRA}
if ! modinfo -k ${K_VRA} nvidia > /dev/null 2>&1; then
    echo "FAIL: DKMS did not produce an nvidia kernel module"
    find /usr/lib/modules/${K_VRA} -name '*.ko*' | head -20
    exit 1
fi
echo "PASS: nvidia module present \
($(modinfo -k ${K_VRA} -F version nvidia))"

Library resolution: ldd confirms userspace tools have all their shared library dependencies satisfied:

for bin in nvidia-smi nvidia-modprobe; do
    if ldd "$(which $bin)" 2>&1 | grep -q "not found"; then
        echo "FAIL: $bin has unresolved library dependencies:"
        ldd "$(which $bin)" | grep "not found"
        exit 1
    fi
done

Manifest sanity: a minimum file count check catches cases where rpm -ql missed packages or upstream package names changed:

MIN_FILES=50
FILE_COUNT=$(wc -l < /nvidia-files.txt)
if [ "${FILE_COUNT}" -lt "${MIN_FILES}" ]; then
    echo "FAIL: manifest has only ${FILE_COUNT} files \
(expected at least ${MIN_FILES})"
    exit 1
fi

Single kernel enforcement

bootc requires exactly one kernel version in /usr/lib/modules/. The builder stages may have had different kernel versions during their build process (the builder’s base image kernel versus the target kernel). After copying modules from both builders, the final stage cleans up:

ARG KMOD_KERNEL
RUN cd /usr/lib/modules && \
    for dir in */; do \
        dir_name="${dir%/}"; \
        if [ "$dir_name" != "${KMOD_KERNEL}" ]; then \
            echo "Removing: $dir_name"; \
            rm -rf "$dir_name"; \
        fi; \
    done && \
    DIR_COUNT=$(ls -1d /usr/lib/modules/*/ 2>/dev/null | wc -l) && \
    if [ "${DIR_COUNT}" -ne 1 ]; then \
        echo "FAIL: expected exactly 1 kernel, found ${DIR_COUNT}" && \
        exit 1; \
    fi

Final image validation

The build is gated twice. Each builder proves its own output before anything leaves it, and the final image proves the toolchain never came along.

The production image validates itself as the last build step. Five checks, any failure stops the build:

RUN # No build dependencies leaked into final image
    for pkg in dkms gcc kernel-devel; do \
        if rpm -q "$pkg" &>/dev/null; then \
            echo "FAIL: build dependency '$pkg' found" && exit 1; \
        fi; \
    done && \
    # Kernel modules present
    modinfo -k ${KMOD_KERNEL} zfs > /dev/null && \
    modinfo -k ${KMOD_KERNEL} nvidia > /dev/null && \
    # Userspace tools present
    for bin in zfs zpool nvidia-smi nvidia-ctk zrepl; do \
        command -v "$bin" > /dev/null || \
            { echo "FAIL: $bin not found" && exit 1; }; \
    done && \
    # Shared libraries resolved
    for bin in zfs zpool nvidia-smi nvidia-ctk zrepl; do \
        if ldd "$(which $bin)" 2>&1 | grep -q "not found"; then \
            echo "FAIL: $bin has unresolved libraries" && exit 1; \
        fi; \
    done && \
    # systemd services enabled
    for svc in zfs-import-cache zfs-mount nvidia-powerd; do \
        systemctl is-enabled "$svc" > /dev/null || \
            { echo "FAIL: $svc not enabled" && exit 1; }; \
    done && \
    # bootc structural validation
    bootc container lint --no-truncate

Cross-distro DKMS packages

noarchDKMS packages are typically noarch: they contain source code and build scripts, not compiled binaries. This means a DKMS package built for one distro often works on another, as long as the build dependencies are satisfied.

NVIDIA’s kmod-nvidia-open-dkms is a noarch package: just source files and build scripts. The RHEL 10 CUDA repository often has newer drivers than the Fedora CUDA repository. Since the DKMS package is just source, it compiles against whatever kernel-devel is available. This is how you get NVIDIA 595.x building on Fedora 42 when the Fedora repo only has 590.x.

# Use RHEL 10 CUDA repo on CentOS Stream 10
source /etc/os-release
curl https://developer.download.nvidia.com/compute/cuda/repos/\
rhel${VERSION_ID}/${ARCH}/cuda-rhel${VERSION_ID}.repo \
    -o /etc/yum.repos.d/nvidia.repo

ZFS has a similar cross-distro story. EPEL provides zfs-dkms for CentOS, but Fedora sometimes needs a source build with libtirpc-devel as a non-obvious dependency.

What is in the final image

The production image contains:

Kernel modules: zfs.ko, nvidia.ko, and their dependencies in /usr/lib/modules/${KMOD_KERNEL}/extra/
ZFS userspace: zfs, zpool, shared libraries, systemd units, udev rules, scrub timers
NVIDIA userspace: nvidia-smi, nvidia-modprobe, shared libraries, GSP firmware, nvidia-powerd service
zrepl: ZFS snapshot replication daemon
nvidia-container-toolkit: GPU access in Podman containers via CDI

The production image does not contain:

gcc, make, or any compiler toolchain
kernel-devel or kernel source headers
DKMS framework
Build-time RPM repositories
Any file not in the manifest or explicitly installed in the final stage

The size difference is roughly 1GB: ~2GB final image versus ~3GB with the build toolchain included.

The multi-stage pattern is an instance of the composable tools approach: each builder is a self-contained unit that produces a well-defined output (compiled modules + manifest), and the final stage composes those outputs into a production image. The builders can be updated, replaced, or extended independently. The foot gun post covers what happens when this kind of pipeline validation is missing.