Why is firewalld allowing public traffic to my non-public ports, bound to Docker containers?

I'm trying to implement a pretty simple firewall in Fedora, where the public internet can access SSH, HTTP, HTTPS and Cockpit, but nothing else. Meanwhile, the servers run microservices via Docker that can talk to each other on ports 8000-8999.

I set this up on a fresh Fedora Server install with the following commands:

firewall-cmd --zone=public --add-service=cockpit
firewall-cmd --zone=public --add-service=http
firewall-cmd --zone=public --add-service=https

firewall-cmd --zone=internal --add-source=192.168.1.65
firewall-cmd --zone=internal --add-source=192.168.1.66

firewall-cmd --zone=internal --add-port=8000-8999/tcp

firewall-cmd --runtime-to-permanent

When I check my config with --list-all, everything looks correct:

> firewall-cmd --list-all --zone=internal
internal (active)
  target: default
  icmp-block-inversion: no
  interfaces:
  sources: 192.168.1.65 192.168.1.66
  services: dhcpv6-client ssh
  ports: 8000-8999/tcp
  protocols:
  forward: no
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

> firewall-cmd --list-all --zone=public
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: enp2s0
  sources:
  services: cockpit dhcpv6-client http https ssh
  ports:
  protocols:
  forward: no
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

However, when I test this, I'm able to hit http://192.168.1.65:8080. I can hit it from a machine on the same internal network (192.168.128.128), as well as a public request from an external machine. Since neither was listed in internal's sources, I assumed that firewalld would not allow the requests through.

I have an auto-configured docker zone with the docker0 interface, but removing that doesn't seem to change my ability to hit the internal port.

Why am I able to successfully request :8080 from sources that are not listed on internal?

I managed to resolve this issue by recreating the DOCKER-USER chain and would like to share it in case anyone else finds it useful.

This is based on a solution I found on Github (https://github.com/firewalld/firewalld/issues/869) and on some genius's blog (https://roosbertl.blogspot.com/2019/06/securing-docker-ports-with-firewalld.html) but I had to adapt/refine it a little.

# 1. Stop Docker
systemctl stop docker.socket
systemctl stop docker.service

# 2. Recreate DOCKER-USER iptables chain with firewalld. Ignore warnings, do not ignore errors
firewall-cmd --permanent --direct --remove-chain ipv4 filter DOCKER-USER
firewall-cmd --permanent --direct --remove-rules ipv4 filter DOCKER-USER
firewall-cmd --permanent --direct --add-chain ipv4 filter DOCKER-USER

# 3. Add iptables rules to DOCKER-USER chain - unrestricted outbound, restricted inbound to private IPs
firewall-cmd --permanent --direct --add-rule ipv4 filter DOCKER-USER 1 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -m comment --comment 'Allow containers to connect to the outside world'
firewall-cmd --permanent --direct --add-rule ipv4 filter DOCKER-USER 1 -j RETURN -s 127.0.0.0/8 -m comment --comment 'allow internal docker communication, loopback addresses'
firewall-cmd --permanent --direct --add-rule ipv4 filter DOCKER-USER 1 -j RETURN -s 172.16.0.0/12 -m comment --comment 'allow internal docker communication, private range'

# 3.1 optional: for wider internal networks
firewall-cmd --permanent --direct --add-rule ipv4 filter DOCKER-USER 1 -j RETURN -s 10.0.0.0/8 -m comment --comment 'allow internal docker communication, private range'
firewall-cmd --permanent --direct --add-rule ipv4 filter DOCKER-USER 1 -j RETURN -s 192.168.0.0/16 -m comment --comment 'allow internal docker communication, private range'

# 4. Block all other IPs. This rule has lowest precedence, so you can add rules before this one later.
firewall-cmd --permanent --direct --add-rule ipv4 filter DOCKER-USER 10 -j REJECT -m comment --comment 'reject all other traffic to DOCKER-USER'

# 5. Activate rules
firewall-cmd --reload

# 6. Start Docker
systemctl start docker.socket
systemctl start docker.service

Well, it turns out that the problem stems from the fact that I'm using Docker for my internal ports. In order to simplify the process of getting containers to talk to the world and each other, Docker made the choice to take a large amount of control over your firewalls/networking. This means that if you don't want your containers to be publicly accessible AND you need to control public access via the firewall on the same machine as your Docker daemon, you need to configure your firewalls slightly differently. Docker has some official documentation on how to do this. Basically, you have the following options:

• Set up a separate machine just for your firewall. This would probably be the easiest, since Docker and your firewall wouldn't have to share resources.
• Add your iptables rules to the DOCKER-USER chain (this is more of an answer for iptables users; I'm not sure how to get firewalld to replicate this approach)
• Disable the whole thing by setting iptables=false in your Docker service config. (this blog post discusses this option)

I also found a post that I thought was a nice variation on the DOCKER-USER chain option. Basically, you create an iptables.conf file that you can load without a flush using iptables-restore -n. Unfortunately, it's an iptables solution, not a firewalld solution.

One of the hard parts of diagnosing this issue was that I would run a script to modify the firewall to match what I wanted, and that would work until I restarted the machine or started up the Docker daemon. Docker overwrites the iptables config when it starts up.