Wrong ARP entry on Lab setup with 2x10G connected through a switch

I have two boxes connected through a 10G switch:

• 1 "sender" box with 1x port static address 10.0.1.1
• 1 "receiver" box with 2x ports static addresses 10.0.1.2 and 10.0.1.3

My goal is to, from the "sender" box, send UDP packets to either of the two ports on the "receiver" box, individually.

I am struggling to understand why ARP on the sender box sees the two interfaces on the receiver box with the same MAC address but different IP addresses.

sender$ arp
Address                  HWtype  HWaddress           Flags Mask            Iface
10.0.1.2                 ether   00:0f:53:9a:44:b1   C                     enp0s25
10.0.1.3                 ether   00:0f:53:9a:44:b1   C                     enp0s25

You can see on the receiver, the MAC addresses end in b0 and b1, respectively.

receiver$ ifconfig
...
enp74s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.1.2  netmask 255.255.255.0  broadcast 10.0.1.255
        inet6 fe80::20f:53ff:fe9a:44b0  prefixlen 64  scopeid 0x20<link>
        ether 00:0f:53:9a:44:b0  txqueuelen 1000  (Ethernet)
        RX packets 43970  bytes 3337154 (3.3 MB)
        RX errors 0  dropped 6  overruns 0  frame 0
        TX packets 217  bytes 36639 (36.6 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 145

enp74s0f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.1.3  netmask 255.255.255.0  broadcast 10.0.1.255
        inet6 fe80::20f:53ff:fe9a:44b1  prefixlen 64  scopeid 0x20<link>
        ether 00:0f:53:9a:44:b1  txqueuelen 1000  (Ethernet)
        RX packets 43632  bytes 3205973 (3.2 MB)
        RX errors 0  dropped 6  overruns 0  frame 0
        TX packets 1375  bytes 249294 (249.2 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 151

On the receiver side, there are two nearly identical ARP entries, only with the interface difference. This actually looks "correct" in the sense that I could reach the same network interface on the sender side through both the local ports.

receiver$ arp
Address                  HWtype  HWaddress           Flags Mask            Iface
10.0.1.1                 ether   f0:de:f1:80:b4:7a   C                     enp74s0f1
10.0.1.1                 ether   f0:de:f1:80:b4:7a   C                     enp74s0f0

Perhaps my choice of instantiating all the interfaces on the same subnet is the problem? How can I achieve the goal of targeting the receivers ports distinctively?

My netplan on the receiver side is like

network:
  version: 2
  renderer: networkd
  ethernets:
    enp69s0:
      dhcp4: true
      match:
        macaddress: 18:c0:4d:8b:49:a3
    enp74s0f0:
      addresses:
        - 10.0.1.2/24
      dhcp4: no
      match:
        macaddress: 00:0f:53:9a:44:b0
    enp74s0f1:
      addresses:
        - 10.0.1.3/24
      dhcp4: no
      match:
        macaddress: 00:0f:53:9a:44:b1

Thank you!

Very easy way to fix this is to use the ARP filter. What it is - it filters out ARP packets that do not match a local interface. In this case only the respective interface will answer ARP requests, never on behalf of a twin interface on the same card - which was what was causing the problem.

Add these lines to /etc/sysctl.d/<something>.conf or if you wish to /etc/sysctl.conf itself:

net.ipv4.conf.<iface>.arp_ignore = 1
net.ipv4.conf.<iface>.arp_announce = 2

where <iface> is the interface name or all for applying to all interfaces in the box.

The full documentation is here:

http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.arp_problem.html#2_6_arp_announce