Score:1

Server

nftables rules to allow networked Docker containers on the same host to use their public URLs

alfonx

11/22/23, 4:06 PM

I am running two Docker containers on the same host living in Docker the same bridge network, and I have nftables set up to restrict more or less all traffic (rules below).

My only problem (that I know of) is that traffic from one container to the other is blocked, if a container uses the public URL of the service provided by the other container.

E.g. service A is an SMTP server which has a public DNS entry mail.acme.com and service B is an app that wants to send mails to the webserver using this URL. The SMTP server is configured s.t. it is accessible from the outside (i.e. port-forwarding like 0.0.0.0:25:25 in docker compose).

In this particular case, the input interface in nftables is not my external interface, but rather the interface of the Docker network the container is in. The traffic is not routed through FORWARD, the rules added by Docker don't apply and the traffic ends up in INPUT where it is dropped.

I don't want to set up individual rules to allow traffic from the Docker interfaces to the allowed ports.

I am looking for a solution that allows the services to use their respective URLs and which is agnostic of ports and interfaces used in the Docker setup. Is this even possible with nftables alone?

The only thing I have come up with so far is to jump from INPUT to FORWARD if the input interface is not an external interface, but that feels like I should not do it.

Here are my INPUT rules:

        chain INPUT {
                type filter hook input priority filter; policy drop;
                ct state vmap { invalid : drop, established : accept, related : accept }
                iifname "lo" accept
                iifname "externalif0" icmp type echo-request accept
                iifname "externalif0" icmpv6 type echo-request accept
        }

This is what happens when a container tries to reach another container on the same host: (I have truncated a bit of the trace output which did not seem too relevant)

trace id 196c1ae6 ip filter trace_chain packet: iif "dockerbr0" ip saddr <containerip> ip daddr <public host ip> tcp sport 33546 tcp dport 443 
trace id 196c1ae6 ip nat PREROUTING packet: iif "dockerbr0" ip saddr <containerip> ip daddr <public host ip> tcp sport 33546 tcp dport 443 

trace id e7c2ca4b ip filter INPUT packet: iif "dockerbr0" ip saddr <containerip> ip daddr <public host ip> tcp sport 33546 tcp dport 443

And here is outside traffic to the same service:

trace id 29f9da6c ip filter trace_chain packet: iif "externalif0" ip saddr <someip> ip daddr <public host ip> sport 36382 tcp dport 443 
trace id 29f9da6c ip nat PREROUTING packet: iif "externalif0" ip saddr <someip> ip daddr <public host ip> sport 36382 tcp dport 443 
trace id 29f9da6c ip nat DOCKER rule iifname != "dockerbr0" dport 443 dnat to <dockerip>:443 (verdict accept)

trace id 29f9da6c ip filter FORWARD packet: iif "externalif0" oif "dockerbr0" ip saddr <someip> daddr <containerip> sport 36382 tcp dport 443 
trace id 29f9da6c ip filter DOCKER rule iifname != "dockerbr0" oifname "dockerbr0" daddr <containerip> tcp dport 443 accept (verdict accept)

trace id 29f9da6c ip nat POSTROUTING packet: iif "externalif0" oif "dockerbr0" ip saddr <someip> ip daddr <containerip> sport 36382 tcp dport 443

Also on the host:

iptables-legacy -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT

2084

0 + 11

docker

nftables

docker-networking

Zoredache

11/23/23, 12:25 AM

You are using nftables on a docker host? Everything I had read suggested that wasn't supported, and caused problems.

alfonx

11/23/23, 9:06 AM

Dear @Zoredache, we use nftables on a host, which runs containers. We do not use it inside the containers. Do you have a link please.

A.B

11/28/23, 7:43 PM

@alfonx a few links (with answers of mine) on SF and UL SE why Docker is a pain with networking (except its *own* networking): https://serverfault.com/questions/963759/docker-breaks-libvirt-bridge-network : Docker broke unrelated VMs. https://unix.stackexchange.com/questions/572022/linux-bridge-for-virtual-machines-not-forwarding-ip-packets-but-is-forwarding-a likewise, but OP didn't even know Docker caused it. And finally a Q/A about using nftables along Docker: https://unix.stackexchange.com/questions/657545/nftables-whitelisting-docker .The main culprit is the mandatory br_netfilter module.

alfonx

12/5/23, 10:14 AM

Of your links only the third actually has some resemblance to my problem as far as I can tell. Please note that the setup and requirement here is very basic: no VMs and I'm not meddling with Dockers rules either. Also note that the official Docker documentation explicitly assumes people might want to run Docker on a host running a firewall: https://docs.docker.com/network/iptables/ These days docker even works OOB with nftables. The fact that Docker explicitly manipulates firewall rules shows that it is expected and intended to be used alongside a firewall, doesn't it?

alfonx

12/5/23, 10:19 AM

Also there are no legacy iptables rules in effect on the hosts, so I don't understand how the br_netfilter could be an issue the way you describe it

A.B

12/5/23, 10:44 AM

br_netfilter affects the same way iptables-legacy, iptables-nft or nftables (in ip, ip6 or inet families). Except nftables has no physdev helper module to cope with br_netfilter.

A.B

12/5/23, 11:03 AM

Though it should not interact with ip's input hook (seen in OP rules), only prerouting / forward / postrouting (and possibly sometimes output) hooks.

alfonx

12/5/23, 2:45 PM

Thanks for the clarification. I have appended the output of `iptables-legacy -S` to the post. I am not really wiser now. In your third link under "Important additional required adjustments" it says "The input path won't be used at all for Docker's containers" - but exactly this is happening in my case.

A.B

12/5/23, 3:07 PM

Adding a set of hooks doing ACCEPT does nothing: the resulting fate of a packet is a logical AND between all rules hooking in the same place: you can have iptables-legacy doing accept + iptables-nft doing accept, if you also have nftables doing drop, then the result is drop. I don't understand all you wrote, I'm not sure you understand all I wrote, so I can't tell much more than give links to what problems I have already seen.

alfonx

12/5/23, 5:00 PM

I posted the iptables-legacy rules specifically to show you that there appears to be no interaction between iptables and nftables of any kind, referencing your links. Could you maybe point out what I wrote that you didn't understand? I'll try to clarify my post accordingly. Likewise, I'm uncertain what you think I didn't understand - please feel free to point that out and I'll try to look into that again.

A.B

12/5/23, 5:03 PM

More probably we both understood everything correctly but there's something to be found that we didn't see and that somebody will find sooner or later.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: nftables rules to allow networked Docker containers on the same host to use their public URLs

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.