Service-to-Service Routing in Rootless Podman

Part one was about keeping rootless podman services apart: each service its own user, its own network plane, no horizontal reach. This one is the obvious next question. If everything is walled off, how does anything talk to anything? A media server still needs to reach the identity provider to do OIDC. The answer I settled on, up front: every service addresses its peers by FQDN, that FQDN resolves to one always-available local router (Envoy), and the router decides where it actually goes. The interesting part is what it took to make an isolated, rootless container reach that router with an address the client would accept, because the obvious way is link-local, and at least one client refuses to talk to link-local on purpose.

The goal: one router, addressed by name

The doctrine is “FQDN everywhere.” A service never hardcodes a peer’s loopback port, its host-gateway, or a LAN IP. It uses the peer’s public name, dag.dunn.dev, every time. That name resolves, split-horizon, to the local Envoy on whatever host the caller is running on. Envoy reads the TLS SNI without decrypting and routes: to a 127.0.0.1 backend if the service lives on this host, or onward to another site’s Envoy over mTLS if it has moved. The only loopback hop left in the whole estate is the last one, Envoy to its own local backend.

No consumer ever learns where a service lives. It holds the name, the local Envoy holds the route, and moving the service rewrites one cluster and one DNS record.

The payoff is portability. Move a service to another box and you re-point one Envoy cluster and one DNS record. No consumer is edited, because no consumer ever knew where the service lived. It only knew the name. Certificates travel with the service, so there is no central place that has to hold every key. And there is exactly one routing model to reason about instead of a thicket of per-service host-gateway holes.

That is the design. The rest of this is the part nobody writes down: actually reaching that router from inside an isolated rootless container.

The obvious path, and the wall

Envoy runs as a rootless container with Network=host, so it binds the real host’s 0.0.0.0:443. A service in its own plane reaches the host through podman’s standard host-access path: the magic name host.containers.internal, which resolves to the link-local address 169.254.1.2. Under the hood pasta (the userspace network helper that sits beneath every rootless bridge) translates that to the host’s interface, so the connection lands on Envoy. You wire it up with an AddHost line per peer:

AddHost=dag.dunn.dev:host-gateway     # → 169.254.1.2 → host → Envoy:443

This works. Eight services on this host reach the identity provider exactly this way. Then I added a service whose OIDC client takes security seriously, and it refused.

The client was a Jellyfin OIDC plugin. Its discovery call to https://dag.dunn.dev/.well-known/openid-configuration failed before a single packet left the box, with an SSRF guard error: the host resolved to a forbidden address. The guard is genuinely well written. Reading its source, it refuses loopback, link-local (the 169.254/16 block, which includes cloud-metadata 169.254.169.254), private RFC1918, and the rest of the SSRF zoo. It has an “allow private / VPN / LAN endpoints” toggle for talking to a LAN identity provider. I turned it on. Still refused. The toggle relaxes RFC1918 and ULA, but it still rejects loopback and link-local even when enabled, by explicit design, because those are the classic SSRF pivots.

That is the wall. The one address my isolated container could reach the router on, link-local 169.254.1.2, was precisely the address class this client would never dial.

Address	Container can reach Envoy?	SSRF guard accepts?
`127.0.0.1` (loopback)	n/a	no, loopback
`169.254.1.2` (host-gateway, link-local)	yes	no, link-local
`172.22.x` / `10.89.x` (real host / bridge RFC1918)	no, firewalled / unreachable	yes

Everything reachable was rejected. Everything acceptable was unreachable.

Why not just put them on a shared network?

This was my first instinct too, and it is worth walking through why each escape hatch is closed, because the closures are the whole reason the eventual fix looks the way it does.

Just share a podman network between the two services. Part one’s correction was that podman networks are not separate namespaces within one user. The catch here is the opposite: these two services run as different rootless users, and different users get different network namespaces. A rootless container cannot join another user’s netns. That is exactly why the estate hairpins through the host in the first place.

Give the container a pasta network and a bridge at the same time. pasta can hand out a custom host-access address. But pasta is a mode, like host or none, not a network you can stack. You get one mode per container, and --network pasta plus --network mybridge errors out with selected mode pasta conflicts with bridge. You need the bridge for container-name DNS to the sidecar, so you cannot drop it.

Use a pod, so the sidecar pair shares one netns over localhost and the pod runs in pasta. This is the elegant version, and it is blocked at the kernel level. A pod shares one user namespace across all members, and you cannot give pod members different keep-id UID mappings. Trying it returns cannot set user namespace mode when joining pod with infra container, and a member with keep-id plus AddHost fails to create outright.

Pre-create a root-owned network namespace and have the containers join it. A rootless user cannot setns into a root-owned namespace without CAP_SYS_ADMIN in the owning namespace, which it does not have. The only “shared netns” available rootless is intra-user, which does not span users, which is the entire problem.

Four plausible doors, four walls. At that point the honest move is to stop guessing and go read what the people who build podman have already decided.

Each shortcut dies on a rule the kernel enforces, not a podman preference. What survived is not a shared network at all: it is an address the client agrees to dial.

Has anyone upstream hit this?

They have, and the rulings are clear enough to plan around. A few that mattered:

pasta and bridge as co-equal modes: not a thing, and unlikely to become one. For slirp4netns the answer was a flat “cannot work, by design” years ago (containers/podman#13109). For pasta the closest thread is #26114, nominally about running two pasta interfaces, where a maintainer spells out the underlying reason: two host-side network engines cannot co-manage port forwarding, and mixing pasta with the bridge’s rootlessport “would make things unmanageable,” “really not worth the effort on the podman side.” That issue is technically still open and the passt side has left a door ajar, but nothing points at co-equal modes. The direction of travel is the opposite: pasta moving under the bridge as its port forwarder, which already exists as the experimental rootless_port_forwarder="pasta" option.
Per-member pod user namespaces: rejected at the kernel level (#26848). A user namespace owns the namespaces a pod shares, so two mappings cannot coexist. The 5.8.0 change here only made the rejection fire consistently across CLI, quadlet, and API. It is a guard, not a missing feature.
A bridge container reaching a host service: this is the one that unlocked it. #28718 is almost exactly my scenario, closed not_planned, with the ruling that matters: the host loopback is deliberately not reachable, and “the --map-guest-addr option is mapped to the pasta ip which uses the default interface on the host as default. So you will need to bind your services there not just on loopback.” That last sentence is the recipe. containers/common#2136 (shipped in podman 5.3) is what wires a default --map-guest-addr into the rootless pasta in the first place.

So: the link-local default is a deliberate, collision-avoiding choice, not a bug. And the supported knob to change it is pasta’s --map-guest-addr, which maps a chosen address to the host’s real interface, where Envoy is already bound on 0.0.0.0. Nothing said this works with a custom RFC1918 on a bridge, but nothing forbade it either, and the maintainer’s own model said it should. That is a question you settle with a test, not a forum thread.

The fix: an RFC1918 the router answers on

The fix is to tell pasta to expose the host at a private address the SSRF guard will accept, instead of the link-local default. It is configuration only, scoped to the one service’s rootless user. No host networking changes, no new container, no proxy.

# ~service/.config/containers/containers.conf
[containers]
base_hosts_file = "none"
host_containers_internal_ip = "10.255.255.254"
[network]
pasta_options = ["--map-guest-addr", "10.255.255.254"]

# the quadlet stays a normal bridge member; just the AddHost target changes
AddHost=dag.dunn.dev:10.255.255.254

The client never changed and the guard never relaxed. Only the address the host was offered on changed, and that was enough.

Two things worth being precise about, because both came up as questions.

Is 10.255.255.254 magic? No. It is not tied to podman’s default subnet and it is not a value podman knows. It is a label I chose, with exactly two constraints: it has to be RFC1918 so the SSRF guard accepts it, and it must not collide with anything the container legitimately routes to, so it sits well clear of the bridge pool (10.89.0.0/16 here) and the LAN (172.22.x). --map-guest-addr is the load-bearing knob: it is what makes pasta actually NAT that address to the host. base_hosts_file = "none" is needed because the host’s own /etc/hosts maps dag.dunn.dev to 127.0.0.1; without dropping it, the container inherits that loopback line alongside the AddHost and the guard rejects on it. host_containers_internal_ip only relabels host.containers.internal, which the explicit AddHost bypasses, so it is there for consistency more than necessity.

Did we reconfigure podman at the host level? No. Every line above lives in one rootless user’s config. The host’s networking, firewall, and Envoy are untouched.

Is that 200 real?

When the discovery call finally returned HTTP 200, the right reaction was suspicion, not celebration. A 200 proves something answered, not that the right thing answered. Maybe pasta was quietly looping back to a local listener. The clean way to settle it is the TLS certificate: Envoy is SNI-passthrough, so the certificate the container receives is the identity provider’s real certificate, end to end. Nothing local could present that without the private key.

curl -v https://dag.dunn.dev/.well-known/openid-configuration
# * Connected to dag.dunn.dev (10.255.255.254) port 443   ← the synthetic RFC1918, not 127
# *  subject: CN=dag.dunn.dev
# *  issuer:  C=US; O=Let's Encrypt; CN=YE1               ← the provider's real cert
# < HTTP/2 200

Three checks, all consistent. It connected to 10.255.255.254, not loopback. The served certificate was the provider’s real Let’s Encrypt cert, which only the real backend can present. And a negative control, the same IP with a bogus SNI Envoy does not route, returned a connection failure rather than a response, which proves it is genuinely Envoy doing SNI routing and not a local catch-all answering everything.

curl --resolve bogus.dunn.dev:443:10.255.255.254 https://bogus.dunn.dev/
# → connection closed, no response   ← Envoy refuses an unknown SNI; not a local listener

The path is real: container, to 10.255.255.254, through pasta to the host’s interface, into Envoy:443, SNI-routed to the backend. The FQDN doctrine, made literal.

Where it lands

The result is a single pattern that holds the line from part one. Each service stays in its own isolated plane. It still talks to its peers only by name. The name still resolves to one local router that owns all the routing decisions. The only thing that changed is the address pasta hands out for “the host,” from a link-local that an SSRF-strict client distrusts to a private one it accepts, set per user, in config, with no new moving parts.

A few honest edges remain. The per-user containers.conf has to be declared and converged like any other artifact, and the convergence has to restart the whole user’s container set, not a single unit, or the stale-pasta trap bites. The clean single-container case can use pasta as a mode directly (Network=pasta:--map-guest-addr,...); the common multi-container case, where a service is also a sidecar’s upstream and needs the bridge, is the two-knob form above. And I have not yet minimized that form to prove host_containers_internal_ip is droppable, which is a small test for another day.

What I like about where it landed is that it never fought the platform. The maintainers’ “host access is link-local by design, point --map-guest-addr at your interface-bound service” was the answer the whole time. The work was reading it correctly, choosing an address that does not collide, and proving with a certificate that the bytes actually went where the name promised.

Epilogue: the fleet had opinions

Between drafting this and publishing it, the pattern met the rest of the fleet, and the results are worth reporting honestly.

First, seduced by uniformity, I rolled the synthetic address out to every service on the host: one host-edge shape everywhere, no special cases. The stale-pasta trap from the aside above did exactly what it says, at scale. The config landed, but five services’ long-lived pastas kept running with their original arguments, so the address those services were now dialing was mapped by nothing, and every request through them died as a 502. The rollout was reverted the same day, and the reversal is now written down as an architecture decision record: the link-local default is the default, and the synthetic address is a scoped exception for the one SSRF-strict client, enforced by a CI rule that fails the build if 10.255.255.254 appears in any other service’s quadlet. Convergence for the one exception now asserts against the running pasta’s arguments, pgrep rather than the config file, because a config file that disagrees with the live process was the entire failure mode.

Second, the exception itself is temporary, because the real fix was never the address. It was the client. I sent the plugin a configurable SSRF allowlist (JellyfinSecurity#106), so a deliberate operator can permit one specific non-RFC1918 address instead of relabeling the host. It merged within a day. Once that release converges here, the last synthetic retires and the whole estate is back on the stock link-local default.

Which is the right ending. The platform’s default survived contact with everything except one strict client, and the answer to a strict client is to make the client configurable, not to teach the whole fleet a new address. The workaround earned its keep as a bridge, and the bridge is already coming down.