Rootless Podman Pods Leak Subordinate UIDs
I’ve been running rootless Podman for a long time. Long enough that the tooling has changed shape underneath me more than once, from handwritten systemd units to podman generate systemd to quadlet files. The appeal has always been the same: rootless containers with no daemon, native systemd integration, no Docker socket to worry about. And for nearly as long as I’ve been running it, one thing has quietly driven me up the wall.
User namespaces leak subordinate UIDs onto the host filesystem. You ls -la a data directory expecting to see your service user and instead get 200200069:200200069 on every file. It’s confusing to the eyes, and confusing to the eyes means confusing to debug. The service user can’t read the files it supposedly owns. podman unshare lets you interact with them properly (it enters the user namespace so the UIDs resolve), but you still have to just learn to ignore the ownership columns when browsing the filesystem normally. I lived with that for years.
The breakthrough came recently: the pod abstraction itself was the constraint. Per-container user namespace mapping on named networks solves it completely, and Podman’s built-in DNS (aardvark-dns) makes named networks a practical replacement for pods in the cases where it matters.
The problem
When Podman runs a container rootless, it creates a user namespace that maps UIDs inside the container to a range of subordinate UIDs on the host. The mapping is defined in /etc/subuid and /etc/subgid. If your service user is UID 2000 with a subordinate range starting at 200200000, the default mapping looks like this:
| Container UID | Host UID | Why |
|---|---|---|
| 0 (root) | 2000 (service user) | UID 0 maps to the real user |
| 1 | 200200000 | First subordinate UID |
| 70 (postgres) | 200200069 | 70th subordinate UID (base + 69) |
| 1000 (app) | 200200999 | 1000th subordinate UID |
One keep-id mapping rescues one container UID. Every other UID still lands on the host as a subordinate number nothing can read.
Container root maps to your service user, which is fine. Everything else maps into the subordinate range, which means files written by any non-root process inside the container end up owned by UIDs that have no meaning on the host. Your service user can’t read them. ls shows numeric UIDs instead of names. ZFS snapshots and backups faithfully preserve these meaningless ownership values.
Podman’s keep-id user namespace mode fixes this by remapping a specific container UID back to the host user. UserNS=keep-id:uid=70,gid=70 tells Podman: “map container UID 70 to my real UID on the host.” Files written by the postgres process inside the container are owned by the service user on the host. Clean ls, working backups, no translation needed.
The catch: in a Podman pod, UserNS= can only be set at the pod level. All containers in the pod share one user namespace. If your pod has PostgreSQL (UID 70) and an app server (UID 1000), you can only map one of them correctly.
Only the named network gives each container its own mapping, so every file on the host lands owned by the service user.
The experiment
I ran three phases on a CentOS Stream 10 bootc host as the auth service user (UID 2000, subordinate range 200200000-200265535). The host runs on the immutable base image I’ve been building (the deployment story has its own post).
Phase 1: Pod without UserNS overrides
The baseline. Create a pod, run postgres inside it, check what lands on disk.
podman pod create --name exp-pod
podman run -d --pod exp-pod \
-v /tmp/exp-pod/postgres:/var/lib/postgresql/data:Z \
docker.io/library/postgres:16-alpine$ ls -la /tmp/exp-pod/postgres/
total 128
drwx------ 19 200200069 200200069 4096 Mar 7 14:23 .
drwxr-xr-x 3 auth auth 4096 Mar 7 14:22 ..
drwx------ 5 200200069 200200069 4096 Mar 7 14:23 base
-rw------- 1 200200069 200200069 30 Mar 7 14:23 current_logfiles
drwx------ 2 200200069 200200069 4096 Mar 7 14:23 global
drwx------ 2 200200069 200200069 4096 Mar 7 14:23 pg_wal200200069. That’s container UID 70 (postgres) mapped through the subordinate range. The auth user owns the parent directory but cannot read anything inside postgres/ because those files are owned by a different UID entirely.
$ cat /tmp/exp-pod/postgres/current_logfiles
cat: /tmp/exp-pod/postgres/current_logfiles: Permission deniedThis is the state I lived with for years.
Phase 2: Named network with per-container keep-id
Replace the pod with a named network. Each container gets its own UserNS= mapping.
podman network create exp-net
podman run -d --network exp-net --name exp-postgres \
--userns=keep-id:uid=70,gid=70 \
-v /tmp/exp-net/postgres:/var/lib/postgresql/data:Z \
docker.io/library/postgres:16-alpine$ ls -la /tmp/exp-net/postgres/
total 128
drwx------ 19 auth auth 4096 Mar 7 14:31 .
drwxr-xr-x 3 auth auth 4096 Mar 7 14:30 ..
drwx------ 5 auth auth 4096 Mar 7 14:31 base
-rw------- 1 auth auth 30 Mar 7 14:31 current_logfiles
drwx------ 2 auth auth 4096 Mar 7 14:31 global
drwx------ 2 auth auth 4096 Mar 7 14:31 pg_walauth:auth everywhere. The service user owns everything. Backups work. ls is readable. A second container on the same network with a different UID mapping works independently:
podman run -d --network exp-net --name exp-server \
--userns=keep-id:uid=1000,gid=1000 \
-v /tmp/exp-net/server:/data:Z \
ghcr.io/goauthentik/server:latestBoth containers write files as the auth service user on the host. Both containers resolve each other by name over DNS. No conflict.
Phase 3: Pod with keep-id (the mixed-UID failure)
To confirm the pod limitation, set keep-id:uid=70,gid=70 at the pod level and try to run a second container at UID 1000 inside the same pod.
podman pod create --name exp-pod-keepid --userns=keep-id:uid=70,gid=70
podman run -d --pod exp-pod-keepid \
-v /tmp/exp-pod-keepid/postgres:/var/lib/postgresql/data:Z \
docker.io/library/postgres:16-alpinePostgres files land correctly as auth:auth. But attempting to run a process as UID 1000 inside the same pod fails. The shared user namespace only has a mapping range sufficient for the keep-id:uid=70 configuration. UID 1000 falls outside the mapped range, and operations that require it (like adduser during container startup) fail with insufficient permissions.
The pod’s single user namespace cannot serve two different non-root UIDs. This is the fundamental constraint.
Why named networks work
A Podman named network is a user-created network (as opposed to the default bridge) that uses Netavark for networking and aardvark-dns for automatic DNS resolution. Containers on the same named network resolve each other by container name with zero configuration. If your postgres container is named authentik-postgres and your app server references AUTHENTIK_POSTGRESQL__HOST=authentik-postgres, it just works.
The default podman bridge network deliberately has no DNS (Docker compatibility behavior). You must create a named network to get name resolution, which is a one-line operation.
What you lose compared to pods: shared localhost. Containers in a pod communicate over 127.0.0.1 because they share a network namespace. On a named network, containers have separate network stacks and reach each other by name over the network bridge. For most services this is transparent (you’re already using hostnames in config), but anything that hardcodes localhost for inter-container communication would need adjustment.
What you gain: each container gets its own UserNS= mapping. PostgreSQL maps keep-id:uid=70,gid=70. The app server maps keep-id:uid=1000,gid=1000. Caddy maps plain keep-id (runs as root). All files on the host ZFS datasets are owned by the service user regardless of which container wrote them. Backups work. Permissions make sense. The abstraction stops leaking.
A pod wraps every container in one user namespace, so a single keep-id map has to serve them all. A named network hands each container its own, and aardvark-dns keeps them talking.
Quadlet snippets
The named network pattern for a mixed-UID service in quadlet files:
# authentik.network
[Network]
NetworkName=authentik# authentik-postgres.container
[Container]
ContainerName=authentik-postgres
Image=docker.io/library/postgres:16-alpine
UserNS=keep-id:uid=70,gid=70
Network=authentik.network
Volume=/var/zfs/safe/auth/postgres:/var/lib/postgresql/data:Z
AutoUpdate=registry
Retry=3
RetryDelay=5s
[Service]
Restart=always
[Unit]
RequiresMountsFor=/var/zfs/safe/auth
After=zfs-import.target# authentik-server.container
[Container]
ContainerName=authentik-server
Image=ghcr.io/goauthentik/server:latest
UserNS=keep-id:uid=1000,gid=1000
Network=authentik.network
AutoUpdate=registry
Retry=3
RetryDelay=5s
[Service]
Restart=alwaysThe Network=authentik.network directive causes systemd to auto-generate Requires= and After= on the network unit. DNS resolution between containers is automatic.
For comparison, the pod pattern for a single-UID or ephemeral-data service:
# observe.pod
[Pod]
PodName=observe
PublishPort=8460:3000
# observe-grafana.container
[Container]
Pod=observe.pod
Image=docker.io/grafana/grafana:latestThe gotcha with keep-id: you need to know what UID the container image runs as. Check the image’s USER directive:
podman image inspect docker.io/library/postgres:16-alpine \
--format '{{.User}}'
# Output: 70If the image doesn’t set USER (runs as root), plain keep-id without uid/gid arguments is correct.
The omnibus exception
Some images are fundamentally incompatible with keep-id. GitLab omnibus bundles dozens of internal services each running as a different Unix user inside a single container. There is no single UID to map. The only viable approach is Podman’s default automatic subordinate UID mapping, which means the leaked UIDs land on disk.
This is acceptable when the application handles its own data management. GitLab’s gitlab-backup rake task operates inside the container’s user namespace where the UIDs make sense. For any host-side file operations (inspection, manual backup), wrap the command with podman unshare:
podman unshare ls -la /var/zfs/safe/forge/data/podman unshare enters the user’s namespace mapping so subordinate UIDs resolve to their container-internal names. It’s a workaround, not a fix, but for omnibus images it’s the only option short of decomposing the image into separate containers (which is what the named network pattern effectively gives you for services that support it).
When to use which
| Architecture | When to use | Tradeoff |
|---|---|---|
| Named network | Mixed internal UIDs, persistent data on ZFS that you back up or inspect directly | Lose shared localhost, gain correct file ownership and working backups |
| Pod | Single UID, ephemeral data, omnibus images, or services where you never touch host files | Simple setup, shared localhost, but leaked subordinate UIDs on any host-mounted volumes |
| Host network | Edge services that bind privileged ports directly | No network isolation, use sparingly |
The finding itself took an afternoon to validate across three experiment phases. The interesting part is how we got there. Setting up reproducible experiments against a live system, iterating on hypotheses, and converging on an architecture decision happened at a pace that would have taken several hours of manual iteration previously. That process, using LLMs to mediate technical experimentation, is worth its own post. The harness exploration earlier this year touched on how working with generative AI changed my velocity. This was a concrete instance of that shift applied to infrastructure rather than code.