Score:0

Server

Linux randomly routes packets to the wrong interface

Mathis

11/13/23, 11:15 AM

I own a dedicated server with some kvm vps running for my clients. My main network is 192.168.1.0/24. The vps have a dedicated network 192.168.2.0/24 on a bridge interface brv6. The ipv6 traffic goes through my isp router as each vps have it's own ipv6 address. But for my security, the ipv4 traffic is routed through a nordvpn wireguard vpn (10.5.0.1/24). Here is the routing table for the vps:

default dev wg0 scope link
192.168.1.0/24 dev br0 scope link

Everything from brv6 use this routing table.

ip rule add iif brv6 table vps

Here is an illustration:

192.168.2.0/24 <==> 10.5.0.1/24 <==> Internet

VPS <==> NordVPN <==> Internet

As nordvpn servers don't know the network 192.168.2.0/24, i put a NAT between 192.168.2.0/24 and 10.5.0.1/24 so that every packet is NATed with my nordvpn client ip (10.5.0.2).

iptables -t nat -o wg0 -j MASQUERADE

But sometimes (randomly), some clients tell me they don't have ipv4 access. After checking, i found some packets are not going through wireguard, but are in the physical interface (brv6 physical nic).

tcpdump -nni enp3s0f1 ip

Also, conntrack -L shows connections with source address unNATed (in 192.168.2.0/24 instead of 10.5.0.2)

I really can't find a way to get this working. Maybe you guys have an idea ?

The server is running debian 11 on Linux 5.10

550

1 + 6

linux

vpn

nat

routing

Mathis

11/13/23, 1:18 PM

The routing tables are the same on all vps: A default route to 192.168.2.1

Mathis

11/13/23, 1:27 PM

ethtool is installed but does't seems to change my config. Anyway, there is my /etc/network/interfaces : [https://pastebin.com/iC3hJgYa](https://pastebin.com/iC3hJgYa)

anx

11/13/23, 2:57 PM

What program sets the mac address on the bridge and creates interfaces for the virtual machines to use? (on this site we like to get rid of comments that are *no longer needed* - whenever you think a comment asking for clarifications was useful in getting towards a clear question, don't add more comments - just use the [edit] button on the question)

Mathis

11/13/23, 3:01 PM

i use virt-install to create virtual machines, it automatically choose a random mac address

anx

11/13/23, 3:17 PM

Libvirt does a few things automatically, as might whatever is managing your wireguard setup, and whatever is setting up your physical NIC. On the host, try dumping the list of interfaces+addresses / routes / firewall rules (`ip a l` / `ip -4 r` / `iptables -vnL`), if you do not see any differences between intended and degraded status, at least it might serve to better understand the situation where you are encountering the problem.

Mathis

11/20/23, 3:38 PM

Wireguard is configured to not touch the routing table (with `Table = off` in the config file). After deleting iptables rules responsible for blocking broadcast/multicast, the issue seems to be resolved. I'll keep you informed in case this continues.

Score:0

Server

Mathis

12/4/23, 9:53 PM

After hard searching, i found the problem. Virtual machines inside the brv6 network were sending ARP broadcasts to get the brv6 gateway MAC address. brv6 and br0 are both on a switch. But the br0 interface was responding to the ARP packets even if it's not for it. So the VM got an ARP entry for brv6 with the MAC of br0 (which they can't reach).

To fix the problem, i set up arptables rules to block ARP in destination to 192.168.2.1 (brv6) from reaching br0. And vice-versa: block ARP to br0 from being received by brv6.

arptables -A INPUT -i br0 -d 192.168.2.1 -j DROP
arptables -A INPUT -i brv6 -d 192.168.1.138 -j DROP

I don't know if this is a feature or a bug in the linux kernel. Anyway, i found the solution and hope this could help anyone.

+ 2

A.B

12/10/23, 1:09 AM

This results from Linux using the Weak [Host Model](https://en.wikipedia.org/wiki/Host_model), including ARP behavior. You don't have to use arptables. Per-interface ARP settings can be used: [`arp_filter`](https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/Documentation/networking/ip-sysctl.rst?h=v5.15#n1529) or [`arp_ignore`](https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/Documentation/networking/ip-sysctl.rst?h=v5.15#n1580)

Mathis

12/10/23, 8:57 AM

Thanks for your answer, i added `net.ipv4.conf.all.arp_ignore=1`, deleted my arptables rules and the problem seems to be fixed. No bad replies from wrong interfaces.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Linux randomly routes packets to the wrong interface

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.