Score:0

Server

Iptables - Can't get get my custom SNAT rules to work

Taha Adel

2/5/24, 9:09 AM

Simply, I have two networks one for docker and another one for Libvirt. I need to let one container from the docker network to reach out all VMs in the Libvirt network. So, I added an SNAT rule to match any packet sourced from "172.17.0.4" (container IP) and destined to "192.168.122.0/24" (libvirt network) and targeted those packets to be SNATed as follow

sudo iptables -t NAT -I POSTROUTING 1 -s 172.17.0.4 -d 192.168.122.0/24 -j SNAT --to 192.168.122.1

But, unfortunately, when I tried to send some echos, none of the packets matched the rule and I get "Destination Port Unreachable" error at the docker level and the packets never reach any of the VMs.

After investigating this, I found the below chain in the filter table created by libvirt:

Chain LIBVIRT_FWO (1 references)
target     prot opt source               destination         
ACCEPT     all  --  192.168.121.0/24     anywhere            
REJECT     all  --  anywhere             anywhere             reject-with icmp-port- 
unreachable
ACCEPT     all  --  192.168.122.0/24     anywhere            
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable

Here's the output of ip route command from the container:

default via 172.17.0.1 dev eth0 
172.17.0.0/16 dev eth0 proto kernel scope link src 172.17.0.4

any ideas?

2 + 2

nat

iptables

ubuntu-22.04

Tero Kilkanen

2/5/24, 9:15 AM

Please add output of `ip route` from inside the container to the question.

Taha Adel

2/5/24, 10:23 AM

I've edited the question with the output of ip route from the container

Score:0

Server

Taha Adel

2/5/24, 2:34 PM

For an unknown reason, I found the below entry in iptables filter table

-A LIBVIRT_FWI -o virbr1 -j REJECT --reject-with icmp-port-unreachable

After deleting it, I can access the libvirt network from the docker container without any issues.

Thanks Tero and Larsks for your help.

+ 0

Score:0

Server

larsks

2/5/24, 12:29 PM

You shouldn't need any custom NAT rules.

Traffic exiting the containers will hit the MASQUERADE rule that Docker installs for the network (e.g., -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE), so your virtual machines will see traffic originating from the libvirt bridge (192.168.122.1).

Configuration

I have a Docker container named example-container attached to docker0:

$ docker ps
[...]
0ddd3554466b   alpine                         "/bin/sh"                5 seconds ago   Up 4 seconds                                                                                             example-container

That container has the following network configuration:

/ # ip addr show eth0
2642: eth0@if2643: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fdc9:87ad:f15:5bea:0:242:ac11:2/64 scope global flags 02
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe11:2/64 scope link
       valid_lft forever preferred_lft forever

I have virtual machine named example-vm attached to the libvirt default network:

$ virsh list
 Id   Name         State
----------------------------
 2    example-vm   running

The virtual machine has the following network configuration:

[root@localhost ~]# ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:30:7b:a6 brd ff:ff:ff:ff:ff:ff
    altname enp1s0
    inet 192.168.122.70/24 brd 192.168.122.255 scope global dynamic noprefixroute eth0
       valid_lft 3548sec preferred_lft 3548sec
    inet6 fe80::2ccd:72ed:1fea:ccda/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

From the container to the vm

From inside the container, I am able to ping the virtual machine:

/ # ping -c2 192.168.122.70
PING 192.168.122.70 (192.168.122.70): 56 data bytes
64 bytes from 192.168.122.70: seq=0 ttl=63 time=0.262 ms
64 bytes from 192.168.122.70: seq=1 ttl=63 time=0.286 ms

--- 192.168.122.70 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.262/0.274/0.286 ms

If I run tcpdump inside the virtual machine while doing this, I see:

[root@localhost ~]# tcpdump -i eth0 -n icmp
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
07:23:38.661246 IP 192.168.122.1 > 192.168.122.70: ICMP echo request, id 9, seq 0, length 64
07:23:38.661277 IP 192.168.122.70 > 192.168.122.1: ICMP echo reply, id 9, seq 0, length 64
07:23:39.661247 IP 192.168.122.1 > 192.168.122.70: ICMP echo request, id 9, seq 1, length 64
07:23:39.661261 IP 192.168.122.70 > 192.168.122.1: ICMP echo reply, id 9, seq 1, length 64

From the vm to the container

Similarly, I can ping the container from the virtual machine:

[root@localhost ~]# ping -c2 172.17.0.2
PING 172.17.0.2 (172.17.0.2) 56(84) bytes of data.
64 bytes from 172.17.0.2: icmp_seq=1 ttl=63 time=0.149 ms
64 bytes from 172.17.0.2: icmp_seq=2 ttl=63 time=0.127 ms

--- 172.17.0.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1035ms
rtt min/avg/max/mdev = 0.127/0.138/0.149/0.011 ms

And tcpdump in the container shows:

/ # tcpdump -i eth0 -n
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
12:24:48.690361 IP 172.17.0.1 > 172.17.0.2: ICMP echo request, id 2, seq 1, length 64
12:24:48.690373 IP 172.17.0.2 > 172.17.0.1: ICMP echo reply, id 2, seq 1, length 64
12:24:49.723189 IP 172.17.0.1 > 172.17.0.2: ICMP echo request, id 2, seq 2, length 64
12:24:49.723204 IP 172.17.0.2 > 172.17.0.1: ICMP echo reply, id 2, seq 2, length 64

Netfilter rules

This works because Docker installs the following rule:

host# iptables -t nat -S POSTROUTING
-P POSTROUTING ACCEPT
[...]
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
[...]

When traffic exits the container, it hits the MASQUERADE rule. Since the libvirt virbr0 interface is the egress interface for that packet, it gets rewritten to the address of the bridge, which is why in the tcpdump output in the vm we see the ping requests originating from 192.168.122.1.

Similarly, libvirt installs the following rules:

host# iptables -t nat -S LIBVIRT_PRT
[...]
-A LIBVIRT_PRT -s 192.168.122.0/24 -d 224.0.0.0/24 -j RETURN
-A LIBVIRT_PRT -s 192.168.122.0/24 -d 255.255.255.255/32 -j RETURN
-A LIBVIRT_PRT -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A LIBVIRT_PRT -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A LIBVIRT_PRT -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE
[...]

These accomplish the same thing; requests from the virtual machine to the container get rewritten to the address of the docker0 bridge.

+ 2

Taha Adel

2/5/24, 2:31 PM

I know this, but for an unknown reason all packets originating from my docker container destined to libvirt network got rejected with a port unreachable icmp reply. Adding those custom rules was a give up step because I wasn't know the reason of blocking.

larsks

2/5/24, 2:47 PM

If you're getting a "port unreachable" message, that suggests either a routing problem or a firewall rule issue. Running `tcpdump` on your host -- looking at the Docker bridge and the libvirt bridge -- can help narrow down the scope of the problem. Similarly, if you have custom firewall rules configured, removing them and seeing if that impacts the behavior is a good place to start.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Iptables - Can't get get my custom SNAT rules to work

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.