andrew.dunn.dev

The Key in the Wrong Keyring

~23 min
linuxbootcsecure-bootkernelimmutablesystemd

The homelab runs cairn, a system we compose from parts with the goal of doing as much as we can in CI before boot. We want to build kernel modules against a specific kernel, smoke test that the system would boot on metal by using nested virtualization in CI, and sign everything as simply as we can. Further, we want to align to modern/progressive upstream activities as much as we can by using bootc, composefs, and the smattering of systemd subsystems that are not quite golden path today. By traversing these we can look for opportunities for upstream contributions, but this is a story of when we didn’t look at enough context before attempting a contribution.

We wanted to sign everything in CI, and to have the kernel trust our module key, an extra key for our kernel modules. We also did not want to add that key by hand, at a console prompt, on each physical host.

IN CIbuild the imagekernel and modulessign everythingboot chain and modulessmoke testnested virtual machinepromoteif it bootedON EACH HOSTpull the imagewith bootcbootadd our module key by handat each host’s consolethe step we did not want

CI builds the image, signs the boot chain and the kernel modules, boots it in a nested virtual machine, and promotes it. Each host then pulls the image and boots it, without a step at its console to add our module key by hand.

Our interest in this started during our time in public service, working in secure computing environments, where we were always curious about just how many layers of a system could be secured by the people running it. The DoD, for example, runs its own large public key infrastructure for nearly everything. The homelab already runs its own certificate authority, step-ca, for the certificates our services use to prove who they are to each other, so cairn is where we wanted to see how far down that can go: to the hardware that boots and the software that runs.

How cairn boots

bootc delivers and updates the whole operating system the way a container image is delivered. What boots is a Unified Kernel Image, a UKI: the kernel, its startup filesystem and its command line in one signed file. The firmware checks the signature on systemd-boot before running it, and systemd-boot asks the firmware to check the UKI the same way before loading it. Both are signed with our boot key.

A UKI is an ordinary program file for the firmware, with each part in its own named section, so one signature from our boot key covers every section.

ONE SIGNED FILEUKIsystemd-stubstarts the kernel.osrelwhich system this is.cmdlinekernel command line.unamekernel version.pcrsigsigned boot measurements.pcrpkeykey that checks them.initrdstartup filesystem.linuxthe kernelour boot keysigns the finished file once;that one signature coversevery sectiona second keysigns the bootmeasurements; the securitychip checks them before itunlocks the encrypted disknothing checks the kernelon its own; the UKI’s onesignature covers itOUTSIDE THE UKIkernel modulesfiles in the cairn image,also carried in the startup filesystemsigned with our module keyour module key isnot in the UKI

The sections inside a UKI and the key that signs it, with our kernel modules outside the UKI under our module key.

For the firmware to run systemd-boot, it has to trust our boot key. The firmware keeps the keys it trusts to run at boot in a list called db, a variable in the firmware’s storage on the motherboard, beside the settings people still call the BIOS. A change to the list must be signed, except in setup mode, when no platform key (the key at the top, which controls who may change the list) is present and the firmware accepts any change.

The list can be changed in four ways:

  • At the firmware setup screen, someone clears the Secure Boot keys, which restores setup mode.
  • The running system applies an update signed with a key-exchange key, one of the keys allowed to change the list. This is how fwupd updates the deny list, the signatures the firmware must refuse.
  • In setup mode anything can write the list, which is how systemd-boot added our keys from signed files on the boot partition.
  • A server’s management controller, a small computer on the motherboard reached over the network, can change keys through Redfish.
FIRMWARE STORAGE ON THE MOTHERBOARDplatform keydecides who may changethe key-exchange keyskey-exchange keyssign changes to thefirmware’s listthe firmware’s listcalled db: keys trusted to run at bootour boot keyour module keyMicrosoft’s certificatesdbxthe deny listWAYS THE LIST CAN CHANGEfirmware setup screensomeone at the consolecan clear every keythe running systeman update signed witha key-exchange keyhow deny-list updates arrivesetup modeno platform key:anything may writesystemd-boot addedour keys this waymanagement controllerremote, over Redfishsome can clear keysor add one

The firmware’s list sits in the motherboard’s firmware storage, under the platform key and the key-exchange keys and beside the deny list. Four ways can change it, and setup mode is how systemd-boot added our keys.

The firmware’s list on cairn also holds four Microsoft certificates. Microsoft’s third-party certificate from 2011 signs option ROMs, the drivers the firmware runs from plug-in cards such as a GPU, a network card or the SAS controller the disks connect to. The firmware runs them while it sets up the hardware, before any boot loader starts. If the firmware stops trusting the certificate that signed a ROM, it does not run the ROM; in Microsoft’s own test, an empty list leaves nothing on the display.

EARLY BOOT, AS TIME RUNSfirmware on the motherboardchecks each signature against the firmware’s listbefore running itIF THE FIRMWARE’S LISTDOES NOT TRUSTTHAT CERTIFICATEOPTION ROMGPUshows setup, boot menuOPTION ROMnetwork cardboots from the networkOPTION ROMSAS controllerboots from its disksON A DISKboot loaderstarts the operating systemGPU stays darkno picture before the OSno network bootits option ROM never runsno disk bootfrom the disks behind itsigned with Microsoft’s 2011 certificateMicrosoft Corporation UEFI CA 2011,renewed as Microsoft Option ROM UEFI CA 2023or trust each option ROM by its hashthe firmware’s list holds these exact ROMs’ hashes,not Microsoft’s certificate

Early boot runs from the firmware through each card’s option ROM to the boot loader. The dashed boxes show what a server loses when its firmware’s list does not trust Microsoft’s 2011 certificate.

EVERY BOOTfirmware’s listkeys it trusts,called dbsystemd-bootsigned withour boot keyUKIsigned withour boot keykernelinside thesigned UKIthe firmware checks each signed file against its list before it runsFIRST BOOTboot partitionsigned key filessystemd-bootadds our keysfirmware’s listnow holdsour boot keynobody types anything at a console

Every boot, the firmware checks each signed file against the keys in its list before running it. On first boot, systemd-boot adds our keys to that list from signed files on the boot partition.

Why not shim

shim has done a great deal for Linux. Windows 8 hardware shipped with Secure Boot turned on and Microsoft’s key trusted, so a distribution needed a boot file signed by Microsoft to boot on an ordinary PC without the owner changing firmware settings. Matthew Garrett described Fedora’s plan: a small first-stage loader, signed by Microsoft, that carries the distribution’s own key and checks what comes after it. shim is that loader, and Debian and Ubuntu boot through it too.

shim also gives the owner of a machine a way to add keys of their own. MOK, for Machine Owner Key, is the set of keys a machine owner adds through shim’s console prompt, MokManager. The owner asks for a key with mokutil and confirms it at the console on the next boot, which Debian’s wiki explains keeps malware in the running system from adding one.

We left shim out of cairn to find out how much of the boot chain we could own ourselves. shim can start systemd-boot, but MOK would still mean a console prompt on every machine. We may also build vanilla kernels later, from the upstream source with no distribution patches, so we want a chain that does not depend on a distribution’s signed pieces. Without shim there is no MOK, so we needed another way for the kernel to trust our module key.

What the kernel trusts for modules

The kernel comes from Fedora, and our modules are built outside the kernel source: cairn compiles them into the image and signs them as a unit. A kernel we did not build has to trust our module key to load them.

Our module key is a different key from the boot key, and its certificate, the file the key is carried in, is also on the firmware’s list. Fedora and RHEL ship a kernel that accepts module signatures from keys the firmware trusts, through a downstream patch. Mainline, the shared upstream kernel, does not include it.

Each side gives its reasons. Debian added the patch to fix a bug in which modules built on the machine, such as ZFS and NVIDIA drivers, stopped loading under Secure Boot. Vitaly Kuznetsov of Red Hat raised it for mainline in a 2025 RFC, a patch posted for discussion, pointing to cloud images that bring their own firmware keys.

In that thread James Bottomley wrote that the firmware’s list guards the firmware’s part of boot, and “the kernel is a completely separate security domain.” He added that most people would not want Microsoft or the hardware maker able to sign module code. Mimi Zohar wrote that the kernel first loaded the firmware’s keys only to check a kernel started by kexec, the call that starts a new kernel from the running one.

The kernel keeps the keys it trusts in keyrings, lists of keys it accepts signatures from, and four matter here:

  • .builtin holds the keys compiled into the kernel.
  • .secondary holds keys added while the system runs. The kernel verifies module signatures with it and whatever is linked into it.
  • .machine holds keys that reach the kernel through MOK.
  • .platform holds keys read from the firmware’s list, and mainline does not use it for modules.
SOURCES AT BOOTKEYRINGSCONSUMERS.secondary.builtinkernel buildmodule signaturesmainlineMOK tableMokListRTMokListTrustedRTSecure Boot on?no · nothing loadsMokListTrustedRT set?no · to .platformCA with keyCertSign? (Fedora)no · to .platform.machinewhat the kernel acceptsfor an owner’s keyonly ifMokListTrustedRTis setfirmware’s list.platformkexec_file_loadIMAFedora and RHEL:downstream patch,not mainline

Where each source of keys lands among the kernel’s keyrings. On mainline, module signatures read .secondary, which takes a MOK key through .machine only when MokListTrustedRT is set, and only a downstream patch, which Fedora and RHEL carry, lets .platform verify them.

Mainline fills .machine from MOK, which shim hands to the kernel after the owner confirms each key at the console, and we wanted to avoid that prompt.

What we wanted was our module key carried in the image, set at build time, and unable to boot a kernel. Can a UKI carry our module key, hand it to the kernel at boot with no shim present, and have the kernel trust it for modules?

The note in the TODO

systemd-stub is the small EFI program inside a UKI that starts the kernel. systemd’s TODO file has a few lines proposing a .mokkeys section that the stub inserts into the MOK keyring, “by overriding/extending whatever shim sets in the EFI var,” so a local key can be added to an upstream kernel without recompiling it. The MOK keyring the note means is .machine.

The mechanism is simpler than we expected: shim passes MOK keys to the kernel through an EFI configuration table, a named blob in memory. So we wrote a prototype, patching both ends of systemd’s UKI tooling. ukify, the tool that assembles a UKI, put our module key’s certificate into the new .mokkeys section, and at boot the stub built that same table from it, under the entry name the kernel already reads. Secure Boot has verified the image before the stub runs, so the key is covered by the same signature as the kernel.

We tested it on QEMU with OVMF, the open-source UEFI firmware for virtual machines. The kernel did not carry the downstream patch, shim was not in the boot chain, and no keys had been added through MOK.

  • A module signed by our module key loaded.
  • A module signed by a key the kernel had no route to trust was refused.
  • With the section removed, both were refused.
A · STOCK DISTRIBUTIONvmlinuz + initrdfirmwareMicrosoft’s keyshimsigned by Microsoftboot loaderGRUB or systemd-bootMokManageronce, at the consoleMOK tablewritten by shimkernelfirmware’s list + MOKB · THE HOMELABno shimfirmwaretrusts our boot keysystemd-bootour boot keyUKIour boot keyMicrosoft’s certificates stay on the listkernelfirmware’s listno MOK table anywhere: the kernellearns our key from the firmware’s listC · THE WRONG TURNno shim, no MokManagerfirmwaretrusts our boot keyUKI.mokkeys sectionkernelfirmware’s list + MOKMOK tableshim’s entry names,written by our stub

Three boot chains side by side. A stock distribution gets its MOK table through shim, the homelab has no MOK table and trusts the firmware’s list alone, and the wrong turn has our stub write shim’s MOK table from a .mokkeys section in the UKI.

Opening it without asking

We opened an issue describing the approach, then a pull request and a matching spec change. We should have marked them draft and raised the idea in the right channels before writing code, and we did neither. The first replies asked what the section was for and whether we had asked shim’s maintainers.

Luca Boccassi asked why the key that signs the UKI cannot also sign the modules. He also made a point we had not considered: the variable has one producer, shim, whose maintainers are not systemd’s, and a message to them would have cost little and bought goodwill. Lennart Poettering defended the feature, since an image builder signing the UKI with their own key may still want the distribution’s module key in it, and suggested that the stub extend shim’s table when one is already installed.

Adrian Vovk agreed, and said that writing shim’s variables would be controversial with the shim-review maintainers. He said the keys we were carrying are OS vendor keys, not machine-owner keys, so a user who later runs mokutil --untrust-mok to stop trusting their machine-owner keys would lose trust in the vendor keys too. He pointed at a kernel patch of his own that adds a separate .vendor keyring, fed by a new configuration table any boot loader may fill, and said the shim maintainers he had talked to were on board with it.

Our first instinct was to argue that the TODO note said to do exactly what we did. The note was a sketch, and it did not say whose entry names the stub would write. MOK is named for a machine owner’s keys, and the key we carried is an OS vendor’s key, so the names did not fit.

What we had missed

We read the kernel source. The configuration table’s GUID, a globally unique identifier, belongs to Linux, and the header says any EFI boot loader may provide the table. The two entry names the kernel reads from it come from shim. MokListRT is the list of keys shim hands the kernel. MokListTrustedRT is the marker that tells the kernel to trust them, and shim sets it unless the owner has turned it off with mokutil --untrust-mok. We used shim’s names without asking its maintainers.

UEFI has no call that extends an installed table. InstallConfigurationTable() with a GUID that is already installed replaces the pointer, as the UEFI specification says. On the kernel side, efi_mokvar_table_init() reads one table. So extending shim’s table means rebuilding it with our entries added and reinstalling it under the same GUID, which leaves the table shim installed unreachable. We said so in the thread, in case there is an append path we missed.

THE MOK TABLE AND WHO OWNS EACH PARTEFI configuration tableowner: LinuxLINUX_EFI_MOK_VARIABLE_TABLE_GUIDMokListRTEFI signature list of certificatesMokListTrustedRTone-byte markerboth names: shim’sA SECOND TABLE UNDER THE SAME GUIDfirmware table registryone row per GUID -> one pointeranother table’s GUIDptrLINUX_EFI_MOK_VARIABLE_TABLE_GUIDptrreadsefi_mokvar_table_init()reads exactly one table for this GUIDthere is no list to append toInstallConfigurationTable()again, same GUIDshim’s tableunreachable,nothing points herethe new tablesame GUID, new pointer

The MOK table is owned by Linux, but the two entries the kernel reads from it carry shim’s names. The firmware keeps one pointer per GUID, so a second install under the same GUID leaves shim’s table unreachable, and the kernel reads only one table.

When the kernel added the .machine keyring, it did not load every MOK into it. The kernel requires a second signal, MokListTrustedRT, and looks for it both when it loads the table into .machine and when it links .machine into .secondary. Bottomley, who had replaced his firmware’s keys with his own, replied to the patch that added that signal. He noted that an owner like him could not add keys to the kernel this way, and asked how the kernel would know whether the owner trusts the firmware’s keys. Keys from the firmware’s list always go to .platform. A key on the MOK list was added by the owner or built into shim, and the owner can withdraw the kernel’s trust in that list. A key on the firmware’s list says only that the firmware will run what that key signs.

Our prototype answered Bottomley’s question by setting MokListTrustedRT itself. The firmware runs a UKI because it trusts our boot key. So when the stub set MokListTrustedRT, trust that came from the firmware’s list reached module signing without the owner ever consenting, across what Bottomley called separate security domains.

Two paths that need no upstream change

A signed loader installing that table does not need to be a systemd patch. We tried two on one QEMU machine, with Secure Boot on using our keys. The kernel was Fedora’s, rebuilt with the downstream module-trust patch reverted so it behaves like mainline.

The first is a pre-loader, a small EFI binary of about a hundred lines signed with our boot key. It installs the configuration table and then starts systemd-boot, which starts the unmodified UKI. Our module loaded, and the untrusted one was refused. The second is our own shim, signed with our boot key. We built it with our module key’s certificate as its vendor certificate, the certificate built into shim itself. The key landed in .machine with no MokManager prompt, and shim was back in the boot chain.

Both runs loaded the module, so a signed loader can carry the table. We run neither, because we do not want to maintain a signed boot binary, and our own shim would bring back the component we chose to leave out.

The table route has two limits:

  • With Secure Boot turned off, the table still installs but the kernel refuses the key, because mainline gates MOK loading on Secure Boot being enabled. We had assumed there was no such gate.
  • The certificate has to be a certificate authority (CA), one allowed to vouch for other certificates. Under INTEGRITY_CA_MACHINE_KEYRING_MAX, which Fedora’s kernel sets, it must not also allow digital signatures. Any other certificate is silently demoted to .platform.

The two kernel-side answers

That left a kernel question: how should a signed loader that is not shim tell the kernel an owner consented to an extra module key? We built both candidate answers on the same kernel and booted each.

The first is the 2025 RFC’s idea with a boot-time gate added: trust .platform for module signatures only when the signed command line opts in. In our runs it loaded our module and left .secondary holding only .builtin, so the new trust reached module signatures and nothing else. It needs no UKI section.

The second is Adrian’s .vendor keyring, a new keyring fed by a new table any loader may fill, which answers the naming problem the thread found. The branch is work in progress; with six small fixes it booted and loaded a module signed by a self-signed CA certificate in .vendor. IMA, the Integrity Measurement Architecture, is the kernel’s file-integrity checker. Reading the branch as written, before our fixes, showed what the design trusts:

  • .vendor links into .secondary unconditionally, so the same key is trusted for kexec_file_load, for IMA, and for vouching for further runtime keys.
  • There is no consent step of the kind .machine needed before it got its link.
  • The table is parsed with Secure Boot off, where mainline refuses MOK keys.
  • .vendor accepts only a certificate signed by a key already in .builtin or .secondary, so a self-signed key was refused until one of our fixes gave it the CA check .machine uses.

Some of that may be intended, and we said so in the thread.

THE TWO KERNEL-SIDE ANSWERS · WHAT EACH KEYENDS UP TRUSTED FOR.platform opt-in2025 RFC, WITH A GATE ADDEDfirmware’s list.platformgatesigned command line opts inmodule signaturesone consumer.secondary.builtin only, untoucheduntested: root adding the opt-in token through kexec.vendor keyringAS WRITTEN, BEFORE THE SIX FIXESVSK tableany loader may fill it.vendorunconditional.secondarymodule signatureskexec_file_loadIMAvouching for runtime keysfour consumersno consent stepparsed with Secure Boot offrefuses self-signed keysbooted with six small fixes

The .platform opt-in trusts an added key for module signatures alone, behind a gate; .vendor as written links into .secondary unconditionally and trusts it for four consumers.

We closed the pull request, the issue and the spec change with one comment on what we found.

How cairn tests a boot

Every cairn image that can be promoted first boots in QEMU on a nested-virtualization runner, under OVMF and a software TPM from swtpm. A TPM is the security chip that records what booted. A script in the guest prints the keyrings to the serial console. A second run empties the variable store where the firmware keeps its trusted keys, so systemd-boot adds our signed key files on every build, as it did on the hardware.

The prototype went through the same shape of test in a throwaway lab, a Google Compute Engine virtual machine with nested virtualization, and nothing went from the lab to the hardware. Each run loaded at least two test modules, one signed by the key under test and one by a key the kernel had no route to trust, and we read the results from the serial log. On the systemd runs we also checked that PCR 11, the TPM’s record of the UKI, matched what systemd-measure predicted. Three kernels went through it: the patch-reverted Fedora kernel, and the two kernel-side answers above.

A · THE LABpatch + build kernelbuild image · sign UKIwrite variable storethrowaway keys onlybootQEMU · OVMF · swtpmserial logkeyrings (keyctl)module signed by our module key: loads?module signed by an untrusted key: refused?PCR 11 = systemd-measure?next runnothing goes from the lab to the hardware directlyB · THE PIPELINE GATEsealed imageits own keysboot on anested-KVM runnersame OVMF, same swtpmone verdict linepromotehost pulls itbootc

The lab loops from a kernel build to one serial log and back for the next run. The pipeline gate boots a sealed image to one verdict line before promotion, so nothing goes from the lab to the hardware directly.

Where cairn is now

A CA certificate on the firmware’s list does not reach .machine. On this kernel only the MOK list reaches .machine, read from the MOK table or, when there is no table, from the MokListRT variable.

On a production host, .machine is empty, .secondary holds only .builtin, and .platform holds our boot key, our module key and the four Microsoft certificates, all read from the firmware’s list. The host has never booted through shim. systemd-boot added our keys on its own after a warning with a countdown, because its configuration file, loader.conf, says secure-boot-enroll force.

So for now cairn relies on Fedora’s patch: the modules load because Fedora’s kernel trusts the firmware’s list for module signatures. Our module key’s certificate is on that list, so the key could also boot a kernel, which is what we wanted to avoid. If we build vanilla kernels later, they would not carry Fedora’s patch, and the question of how the kernel trusts our module key would come back. What we still do not know is whether the kernel’s integrity maintainers want a consent signal from a loader that is not shim at all, and what shape it would have to take to be worth reviewing.