Score:0

Server

What is wrong with ESXi VMKernel Port in Active/Active configuration?

Appleoddity

6/9/23, 6:21 PM

I have the below, simplified configuration:

Essentially, I have an ESXi host with two physical network adapters. Each adapter plugs in to a different switch. Each switch is connected via a trunk port. A PC is connected to one of the switches. A vSwitch with a VMKernel port and VM ports is configured to use both physical NICs in an Active/Active configuration:

I have run esxtop and can see that the ESXi host has chosen the physical NIC connected to Switch 2 for the VMKernel port. From the PC, if I ping the management IP address of the ESXi host the pings are intermittent. They go up and down.

If I show the mac address-table on each switch, I see that Switch 2 always has the VMKernel's MAC address assigned to the switch port connected to the ESXi host. But, Switch 1 continually adds and removes the VMKernel's MAC address on it's respective physical port. Anytime Switch 1 has the VMKernel's MAC assigned to its physical port, the pings fail.

The reason for the failure is obvious. The reason why Switch 1 is routinely picking up the MAC address of the ESXi VMKernel port is the question. The ESXi host has chosen the interface connected to Switch 2 to be the active port. The interface connected to Switch 1 should be inactive. But, it would appear that it is possibly responding to ARP requests?

It's worth noting that none of the VMs on this host have this problem. They are all reachable and are present in only one MAC table at a time. This problem specifically affects the VMKernel port.

What about this configuration is wrong? I am looking for some type of documentation or explanation on top of a solution to this issue. I know that setting the VMKernel port to be Active/Standby mode will probably solve the issue. But, I can't find anything documented why this current configuration is a problem.

UPDATES:

I disabled CDP on the vSwitch thinking that it might be causing communication over the inactive NIC.
I overrode the vSwitch settings for the VMKernel port and set it to use explicit failover and Active/Standby. I also placed the standby NIC in the unused pool. None of it helped. What did solve the issue was changing the port order around. So, when the port connected to Switch 1 becomes active, I do not see the issue. The MAC address does not become active on Switch 2 at all. These are two significantly different NIC cards, and I'm wondering if this isn't some kind of driver issue.

Something has to be causing the VMKernel MAC address to be seen on Switch 1's port, but it comes and goes every several seconds.

Switch configs for STP and ports: Switch 1

!
spanning-tree mode rapid-pvst
spanning-tree portfast edge default
spanning-tree extend system-id
!
interface Port-channel1
 switchport access vlan 11
 switchport trunk encapsulation dot1q
 switchport mode trunk
!
interface GigabitEthernet1/0/7
 switchport access vlan 11
 switchport mode access
!
interface GigabitEthernet1/0/23
 switchport access vlan 11
 switchport trunk encapsulation dot1q
 switchport mode trunk
 channel-group 1 mode desirable
!
interface GigabitEthernet1/0/24
 switchport access vlan 11
 switchport trunk encapsulation dot1q
 switchport mode trunk
 channel-group 1 mode desirable

Switch 2

!
spanning-tree mode rapid-pvst
spanning-tree portfast edge default
spanning-tree extend system-id
!
interface Port-channel1
 switchport access vlan 11
 switchport trunk encapsulation dot1q
 switchport mode trunk
!
interface GigabitEthernet1/0/3
 switchport access vlan 11
 switchport mode access
!
interface GigabitEthernet1/0/23
 switchport access vlan 11
 switchport trunk encapsulation dot1q
 switchport mode trunk
 channel-group 1 mode desirable
!
interface GigabitEthernet1/0/24
 switchport access vlan 11
 switchport trunk encapsulation dot1q
 switchport mode trunk
 channel-group 1 mode desirable

260

0 + 0

vmware-esxi

cisco-catalyst

Score:3

Server

Kevin

6/11/23, 1:07 AM

The management vmk in ESXI assumes the MAC address of the Nic in the first PCI slot during the initial set-up. This is how it has worked forever. This can break things only when the physical device also starts sending packets. This normally does not happen, physical Nics do not send traffic, they pass traffic along. This behavior also needs to be paid attention to if you decide to move physical Nics from one host to another, this brings down 2 host connections when the physical switch freaks out. My guess is that this Nic started reporting CDP/LLDP traffic and this is when your switch sees the MAC duplication. The easiest solution is to rebuild the vmk through the command line. This will need to be done from a direct console access (DCUI) (KVM, ILO, IDRAC, etc...).

Here are the commands; (Adjust the IP's/subnet mask/portgroup name etc... to fit your needs.)

esxcli network ip interface remove --interface-name=vmk0

esxcli network vswitch standard portgroup add -p Management_Network -v vSwitch0

esxcli network ip interface add --interface-name=vmk0 --portgroup-name=Management_Network

esxcli network vswitch standard portgroup set -p Management_Network --vlan-id 50

esxcli network ip interface ipv4 set --interface-name=vmk0 --ipv4=192.168.50.116 --netmask=255.255.255.0 --gateway=192.168.50.1 --type=static

esxcli network ip interface tag add -i vmk0 -t Management

This will rebuild the management vmk with a VMware MAC address to eliminate this issue. However, I would recommend that you reach out to the hardware vendor/manufacturer for the process of shutting the CDP/LLDP coming from the physical card. This will resolve this one ESXi host issue, but you will end up with it happening to others if you allow the card(s) to continue to perform this function. If this was as big a problem as you had originally thought, VMware would not be a giant company, this is not very common...

0 + 0

Appleoddity

6/11/23, 2:20 AM

You hit the nail on the head. The problem is a hardware level LLDP agent sending out LLDP packets with the same MAC address. Now I need to decide the best way to solve the problem. Your solution is perfectly reasonable, it’s just i do not have easy physical access to the server. I can upgrade my iDrac license or use remote hands and eyes. But, if I do that I can also go in the BIOS and shut off the LLDP agent (apparently). What a pain. Thanks again.

0

Reply

Score:2

Server

Zac67

6/9/23, 6:29 PM

I have run an extremely similar setup for many years without any issues.

How have you configured the switch ports? You shouldn't do anything special (no (M)LAG/LACP) since ESXi takes care of everything. It's fine to stack the switches, just don't aggregate the ports, configure any link-state mirroring or similar.

Switch2 should have the VMkernel port's MAC on the ESXi-facing port and switch1 on the switch2-facing port, permanently.

The MAC flapping back and forth might be caused by another issue like frequent STP topology changes (which aren't usually visible by the ESXi but might still). Check the switches' logs for any anomalies.

The reason why Switch 1 is routinely picking up the MAC address of the ESXi VMKernel port is the question.

Without any LAG that could only happen if the host actually sent frames with the VMK port's MAC to switch1. It won't normally do that unless the link to switch2 failed.

The interface connected to Switch 1 should be inactive.

For the VMK port, yes. There may be VM traffic attached to the same port group.

But, it would appear that it is possibly responding to ARP requests?

ARP or not, frames with the VMK port MAC do not originate from the other port without reason.

0 + 0

Appleoddity

6/9/23, 7:20 PM

Thanks for the reply. The switches are standard access ports. Nothing special. I do see millions of BDPUs sent from Switch 1 to Switch 2. I'm not sure if that is related. It seems to me that Switch 1 is seeing traffic from the ESXi interface connected to it and adding the MAC to it's table temporarily. But, that adapter should be "inactive." I thought I was on to something by disabling CDP on the physical adapter, but it didn't make any difference. Nor has changing the VMKernel to Active/Standby mode. In other words, having anything on the vSwitch in Active/Active mode causes this issue.

0

Reply

Zac67

6/9/23, 7:29 PM

*millions of BDPUs sent from Switch 1 to Switch 2* doesn't seem to be normal - BPDUs are normally only sent on designated ports (=towards root ports), every 2s. Also, the adapters isn't active when you're running active-active. vNICs are attached by virtual port number (roughly randomly) and stay there. If you want active-passive you need to move one port down to *Standby adapters*. No, CDP/LLDP doesn't make a difference, I'd leave it active. The issue facing up with Active/Standby *very much* indicates an issue with the switch configs.

0

Reply

Zac67

6/9/23, 7:29 PM

Could you add the sanitzed configs (at least the parts relevant for the ports and STP) to you question? Do the switch logs show anything?

0

Reply

Appleoddity

6/9/23, 7:40 PM

Hi. Thank you for your help on this. Your updates are spot on, and we are in agreement. Packets from the inactive card should not be seen on Switch 1, yet it is. I updated my original post. If I change the order of ports - i.e. I make the NIC on Switch 1 active, the problem goes away and switch 2 does not see traffic from the inactive NIC. If I swap them back, the problem reappears. These are two significantly different NICs. I'm wondering if this is some kind of driver issue. I appreciate the focus on the switches, but all things point to frames coming from the inactive NIC for some reason???

0

Reply

Appleoddity

6/9/23, 7:50 PM

switch configs added. This is a fairly basic switch configuration. Nothing special. STP is only enabled because there is a connection to another stack of switches over 2 ethernet cables, and these two switches in the discussion are not stacked. Again, using `show mac address-table interface gi1/0/7` shows the VMKernel MAC address popping in and out. Where as on the other switch the MAC is stable, as expected.

0

Reply

Zac67

6/9/23, 8:42 PM

Do you have a well defined root bridge? The other switches are also running RPVST+ or RSTP/MSTP? Port 23&23 as port-channel are the stacking link? And 7 and 4 are the host links?

0

Reply

Score:1

Server

VFrontDe

6/21/23, 5:22 PM

The switch port config that you posted shows that you are using a port channel on the catalyst switches.

Just don't do that! With standalone ESXi hosts this is not supported. ESXi takes care of load balancing and failover on its own within software only. If you absolutely want to use external switch based port channels then this requires that you use vCenter and a distributed switch.

See https://kb.vmware.com/s/article/82609 and https://kb.vmware.com/s/article/1001938 for further details.

0 + 0

Appleoddity

6/21/23, 5:24 PM

Thanks for the reply. The port channel is not configured on any ESXi connected ports. These are trunks between Cisco switches.

0

Reply