Score:3

Can't get new NIC working

cn flag

OS: Ubuntu server 20.04 NIC: Intel XL710-QDA1 DAC cable: FS "generic" QSFP+ Switch: Brocade ICX-6610-24p

I cannot get this thing to work. Updated firmware. No change. Ifconfig shows nothing but RX errors. This is only after using a static IP address. DHCP wouldn't work. Pings both to and fro to IP addresses on the network. Can't ping "google.com" for example. It won't connect to anything, and nothing will connect to it. Tried the other port on the switch and it doesn't even make a link.

Where should I go from here? Bark of the tree from whom I bought the switch? I have no idea where to go from here.

Nikita Kipriyanov avatar
za flag
Start by looking into `dmesg`, and showing us what it says about the NIC.
timramich avatar
cn flag
[ 6.613446] i40e 0000:62:00.0 enp98s0: renamed from eth0 [ 13.018900] i40e 0000:62:00.0 enp98s0: changing MTU from 1500 to 9014 [ 13.727587] IPv6: ADDRCONF(NETDEV_CHANGE): enp98s0: link becomes ready [ 6644.528007] i40e 0000:62:00.0 enp98s0: NIC Link is Down [ 6821.241499] i40e 0000:62:00.0 enp98s0: NIC Link is Up, 40 Gbps Full Duplex, Flow Control: None [ 6875.222989] IPv6: ADDRCONF(NETDEV_CHANGE): enp98s0: link becomes ready [ 7059.501868] IPv6: ADDRCONF(NETDEV_CHANGE): enp98s0: link becomes ready
timramich avatar
cn flag
Updated the card's firmware almost right away. Downloaded Intel's driver almost right away, and it's using that. Used a tool to unlock non-Intel optics, even though it's using a DAC cable. Only one of the four 40g ports on the switch seem alive (only one lights up the NIC, the 3 other ones don't do anything).
David Kraeutmann avatar
fi flag
I ran into the exact same thing with an XL710 and ICX6610-48P. Frustrating, as CX3 is no longer supported on ESXi 8.
Score:2
kz flag

Intel XL710 is known for being one of the crappiest NICs in the whole history of the mankind. Google and you’ll dig out a ton of the horror stories. Intel just keeps chopping off the half baked in silicon features with every single firmware or driver update. Bottom line: You won’t make it work, just throw the damn thing into the trash can. Get yourself some Mellanox CX4LC and live long and happy life.

Score:0
cn flag

I never got this working. Things I did:

  • Ordered 2 spare DAC cables, one end programmed for Brocade, one end programmed for the XL710
  • Went through tons of combinations of settings under ethtool -K
  • Went through tons of combinations of settings under ethtool --set-priv-flags (saw one called legacy-rx and thought that would do it, since all my errors were FROM the switch, and they were framing errors. No CRC errors.).
  • Bought a second XL710
  • Since I had 2 cards, and have 2 servers, linked them together with the "generic" FS DAC. They connected together just fine.
  • Took the switch apart and checked out the 40-gig board and its connectors to interface with the motherboard. There was a speck of dirt (around 1.5mm big). Cleaned it out, no change.
  • Changed the mode to 4x10 and used one of the breakout ports. Each of the 4 virtual interfaces then worked fine, but that was useless to me.

I wound up buying a Mellanox MCX353A-FCBT and it works fine. So I'm guessing the XL710 is just incompatible with these switches.

I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.