I have the following network setup.
I am running a host with 2 vms. Each vm is running debian with cri-o, running some containers. The containers each have an ip address on the 10.200.0.0/24 subnet. One guest uses ips on the 10.200.0.0.0/16 subnet and other uses ips on the 10.200.1.0/16 subnet.
The host has a bridge interface, br0. The host eth0 interface is tied to br0 and the vms NICs are are also part of that bridge, so they get assigned ips on my lan subnet (192.168.1.0/24).
The host has the following sysctl values, so iptables should not affect bridge traffic.
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
I want the containers to be able to connect with each other (10.200.0.4 to/from 10.200.1.2). This works if I setup custom routes on each guest:
# on guest 01
ip route add 10.200.1.0/24 via 192.168.1.243
# on guest 02
ip route add 10.200.0.0/24 via 192.168.1.242
And then I am able to ping other containers from each container, but I don't want to setup static routes on each guest, as I would have to do that for each guest every time something changes.
If I delete these routes, and add some static routes to my host, I can ping all containers from that host:
ip route add 10.200.0.0/24 via 192.168.1.242
ip route add 10.200.1.0/24 via 192.168.1.243
# the guests routing table remains as initially with a default route via br0/eth0
# for 192.168.1.243
default via 192.168.1.1 dev enp1s0
10.200.1.0/24 dev cni0 proto kernel scope link src 10.200.1.1
192.168.1.0/24 dev enp1s0 proto kernel scope link src 192.168.1.243
But, crucially, the containers cannot reach each other. I would expect traffic to go from the containers, to br0, to eth0, which would then forward that traffic back to br0, to .242 or .243 as configured with ip route add.
I can see with tcpdump that the traffic is reaching br0 and eth0 on the host, but is not being forwarded back to br0, is just going back to my router which doesn't know how to route 10.200.0.0/16. If I add a static route to my router, 10.200.0.0/16 via 192.168.103, this actually works, as my host (.103) then knows how to forward this traffic to br0 and to .242 or .243, as per the static routes manually setup.
So, it seems that traffic is properly routed through when it comes from outside eth0/br0 but not when coming from inside the guests. I understand that bridging works on L2, so that traffic needs to be routed on L3 somewhere, but how come it's forwarded properly when it originates outside the guests but not when it originates from inside the guests?
