Join Log in
Score:1
user1955162 avatar Server

Linux confusing MAC addresses for its IP addresses to Cisco switch?

de flag
user1955162

I have multiple CentOS boxes (both Centos 7) with two different servers. These servers have 4 10GB interfaces each. They are connected to two Cisco Nexus switches. Each NIC has a different IP address, all in the same domain. In my example, my server has the following IP addresses on the following NICs:

2: ens5f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 68:05:ca:2d:12:84 brd ff:ff:ff:ff:ff:ff
    inet 172.20.1.203/24 brd 172.20.1.255 scope global ens5f0
       valid_lft forever preferred_lft forever
3: ens5f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 68:05:ca:2d:12:85 brd ff:ff:ff:ff:ff:ff
    inet 172.20.1.204/24 brd 172.20.1.255 scope global ens5f1
       valid_lft forever preferred_lft forever
4: ens4f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 68:05:ca:2d:06:10 brd ff:ff:ff:ff:ff:ff
    inet 172.20.1.200/24 brd 172.20.1.255 scope global ens4f0
       valid_lft forever preferred_lft forever
    inet 172.20.1.201/24 brd 172.20.1.255 scope global secondary ens4f0:1
       valid_lft forever preferred_lft forever
5: ens4f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 68:05:ca:2d:06:11 brd ff:ff:ff:ff:ff:ff
    inet 172.20.1.202/24 brd 172.20.1.255 scope global ens4f1
       valid_lft forever preferred_lft forever

That looks pretty normal to me. I have one interface with a secondary IP, but whatever.

When I check the switch, or any other server on the same VLAN as my server, pinging each of the IPs works, however all of the traffic traverses one interface, and the MAC addresses all belong to the one interface carrying traffic:

172.20.1.200    00:01:56  6805.ca2d.0610  Vlan720         
172.20.1.201    00:15:00  6805.ca2d.0610  Vlan720         
172.20.1.202    00:14:56  6805.ca2d.0610  Vlan720         
172.20.1.203    00:14:53  6805.ca2d.0610  Vlan720         
172.20.1.204    00:14:48  6805.ca2d.0610  Vlan720

When I try in the server to ping out of one of the other interfaces, I get no reply.

# ping -I ens5f1 172.20.1.1
PING 172.20.1.1 (172.20.1.1) from 172.20.1.204 ens5f1: 56(84) bytes of data.
^C
--- 172.20.1.1 ping statistics ---
51 packets transmitted, 0 received, 100% packet loss, time 51204ms

As would be obvious, none of the other interfaces move any traffic:

┌nmon─16g─────────────────────Hostname=zfs1─────────Refresh= 2secs ───19:38.03───────────────┐
│ Network I/O ───────────────────────────────────────────────────────────────────────────────│
│I/F Name Recv=KB/s Trans=KB/s packin packout insize outsize Peak->Recv Trans                │
│  ens5f1      0.0       0.0       0.5    0.0    60.0    0.0        0.0      0.0             │
│      lo      0.0       0.0       0.0    0.0     0.0    0.0        0.0      0.0             │
│  ens4f0    405.4      45.8     337.9  128.5  1228.6  365.0      511.4    144.9             │
│  ens5f0      0.0       0.0       0.5    0.0    60.0    0.0        0.0      0.0             │
│  ens4f1      0.2       0.0       3.0    0.0    61.3    0.0        0.3      0.0             │
│────────────────────────────────────────────────────────────────────────────────────────────│

This is while sending data to one of the other IPs.

I do not have LACP or any other configuration which would give me this expected result. I do not know why the MAC addresses are not being advertised correctly. Any ideas would be appreciated. Multiple servers with different hardware are experiencing this issue. They are also going to two different Cisco Nexus switches with trunk ports between them.

Thanks!

52
0 + 3
arp
Nikita Kipriyanov avatar
za flag
Nikita Kipriyanov
What you wanted to achieve by assigning several different interfaces to the same network? Linux will be confused on the *routing* level (OSI L3), and MAC addresses have nothing to do with this.
0
Reply
A.B avatar
cl flag
A.B
Linux uses Weak [Host Model](https://en.wikipedia.org/wiki/Host_model): the IPs belong to the host not to the NIC, and by default it will choose whatever NIC suits its decision algorithm to use an IP address (this can also be related to the NIC that got its ARP reply seen first etc.). You have to set up policy routing to have it behave differently. A Q/A where I made an answer about something similar: https://serverfault.com/questions/900385/ghost-ping-on-a-multi-nic-linux-system
2
Reply
user1955162 avatar
de flag
user1955162
This explains it. We moved our multiple interfaces to unique broadcast domains. Thanks!
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.