Network Plane Isolation in Rootless Podman

While setting up a homelab SSO stack, I hit a problem that is easy to state and hard to debug: two containers on separate podman networks could still reach each other when they should have been blocked. The answer, up front: when netavark drives its firewall through the iptables backend (the default on Fedora 43 and earlier), --opt isolate=true silently fails, and the fix is to switch it to the nftables backend in containers.conf and recreate the networks. Fedora 44 makes nftables the default, so the problem disappears there on its own.

The threat model is ordinary. A compromised edge container (a reverse proxy) must not be able to reach an internal service’s admin API, which has no authentication. Network segmentation is the obvious tool, and for my deployment pattern podman networks are the obvious segmentation mechanism. But the naive setup quietly does nothing in rootless mode, and the failure is invisible until you probe for it.

The goal

The design: a Caddy reverse proxy and an internal service share an “edge” network so Caddy can proxy to the service’s public listener. A second “internal” network carries management traffic. The service binds its admin API to its internal-network IP rather than 0.0.0.0, and both networks get --opt isolate=true so traffic cannot cross between bridges.

The target state. Two hops stay inside a single plane and are meant to work; the third crosses planes, and isolate=true is the only thing standing in its way.

One correction to the mental model explains the design: podman networks are not separate network namespaces. A container joining two networks gets one netns with one veth per network, and in rootless mode every bridge belonging to your user lives inside the same shared rootlesskit namespace, which happily routes between them. Binding the admin listener to the internal IP handles the listener side; isolate=true is what tells netavark to install firewall rules dropping cross-bridge forwarding. Both are required.

All of that is correct. On this host none of it worked, and nothing said so.

The reproduction

Two isolated networks, a dual-homed provider answering on 0.0.0.0:8080 (deliberately sloppy, so the test exercises the network layer rather than the listener binding), and an edge-only client:

podman network create --opt isolate=true --subnet 10.89.1.0/24 edge
podman network create --opt isolate=true --subnet 10.89.2.0/24 internal

podman run -d --name provider \
  --network edge:ip=10.89.1.10 --network internal:ip=10.89.2.10 \
  busybox sh -c 'while true; do
    printf "HTTP/1.0 200 OK\r\nContent-Length:6\r\n\r\nadmin\n" | nc -l -p 8080
  done'

podman run -d --name edge-client \
  --network edge:ip=10.89.1.20 \
  busybox sleep infinity

Probing from the edge-only client:

podman exec edge-client wget -qO- -T 3 http://10.89.1.10:8080
# → admin        (shared edge bridge: expected)

podman exec edge-client wget -qO- -T 3 http://10.89.2.10:8080
# → admin        (cross-bridge: should be blocked, is not)

The client has no attachment to the internal bridge, yet it reaches the internal IP. The isolation rules that should prevent this were never installed.

Root cause

Turns out netavark, the engine behind podman networks, can drive its firewall through either an iptables backend or an nftables one. On Fedora 43 this host was using the iptables backend, which shells out to iptables-nft, and that is the path with the bug.

iptables-nft 1.8.11, the version Fedora 42, 43, and 44 all ship, has a bug in the -C (rule-exists) check: when the rule involves an interface match, the check returns a false positive whether or not the rule is present. Netavark calls -C before installing each isolation rule, sees “already exists,” and skips the install. No error, no log line, no rule. Podman issues #25438 and #26913 track it, where the maintainers confirm it is an iptables bug, not a podman one. 1.8.10 was fine; 1.8.11 introduced the regression and 1.8.12 fixed it (“nft: fix interface comparisons in -C commands,” February 2026).

One isolate=true, two backends. On iptables netavark asks whether the isolation rule is already there, 1.8.11 says yes for any rule carrying an interface match, and the install it skips never happened; on nftables netavark writes its NETAVARK-ISOLATION chains straight into nft and cross-bridge forwarding is dropped.

The silence is what makes it easy to miss. podman network create --opt isolate=true succeeds, podman network inspect shows isolate set, and nothing logs a failure. The control reports as present everywhere you would think to look; only an actual cross-bridge probe shows it was never enforced.

The fix

Two changes, both required.

1. Switch netavark to the nftables driver in ~/.config/containers/containers.conf:

[network]
firewall_driver = "nftables"

This has netavark call nft directly, bypassing the buggy -C path entirely. The driver is baked into each network at creation time, so existing networks must be recreated:

podman network rm edge internal
podman network create --opt isolate=true --subnet 10.89.1.0/24 edge
podman network create --opt isolate=true --subnet 10.89.2.0/24 internal

2. Keep isolate=true on both networks, not just the internal one. The isolation rule on the internal bridge stops traffic originating from it. Traffic from the edge client arrives via the edge bridge, so the edge bridge needs its own rule to drop cross-bridge forwarding at the source. One-sided isolation looks reasonable and still leaks.

Proof

Recreate the containers on the new networks and re-run the probes:

podman exec edge-client wget -qO- -T 3 http://10.89.1.10:8080
# → admin                     (edge to edge: expected)

podman exec edge-client wget -qO- -T 3 http://10.89.2.10:8080
# → wget: download timed out  (edge to internal: blocked)

podman run --rm --network internal:ip=10.89.2.20 busybox \
  wget -qO- -T 3 http://10.89.2.10:8080
# → admin                     (internal to internal: expected)

That matches the target state in the first diagram. To see the mechanism rather than infer it, inspect the ruleset from inside the rootless network namespace:

podman unshare --rootless-netns nft list ruleset

With the nftables backend, the netavark table contains NETAVARK-ISOLATION chains with drop rules covering forwarding for both bridges. With the iptables backend on iptables-nft 1.8.11, the rules are simply absent. Diffing this ruleset before and after the driver change is the fastest way to confirm the fix actually landed.

Where it landed

In the end the platform caught up faster than I shipped the workaround. I wrote this firewall_driver = "nftables" drop-in on Fedora 43, where netavark was still defaulting to the iptables backend. Before it merged into the image, Fedora 44 arrived with netavark defaulting to nftables on its own, and the drop-in was obsolete: the correct behavior had become the default. So if you are standing this up on Fedora 44 or later, you most likely never see the bug. On 42 or 43, the drop-in is the fix until you upgrade.

The iptables side of it closes too. Fedora 42, 43, and 44 all ship iptables-nft 1.8.11; Fedora 45 ships 1.8.13, which carries the 1.8.12 correction to the -C interface comparison. By then both paths are sound: the nftables backend was never affected, and the iptables backend is fixed at the source.