Score:2

Server

Unable to establish link between Mikrotik router and Mellanox NIC

Vilican

6/23/24, 1:11 AM

I am trying to establish a 25 GbE link between Mikrotik CCR2004-1G-12S+2XS router and a server running Windows Server 2022 with Mellanox NIC (MCX512A-ADAT).

For cabling, I tried using the official Mikrotik DAC (XS+DA0003), unofficial FS.com with one side generic and one side "Mellanox-compatible", a pair of SFP28 from FS.com with SMF link and various other SFP/SFP+ modules just for testing.

Each time, the Mikrotik RouterOS shows a connected DAC/SFP incl. its serial number:

However, on the server (Mellanox) side, I only see information about the link being down or the cable unplugged:

I tried changing FEC mode, rate (low/high), flow control and auto-negotiation on Mikrotik, but the link still did not work.

What am I doing wrong?

Edit:

I have also tried setting the FEC and link speed manually:

mlxlink -d mt4121_pciconf0.1 --link_mode_force --speeds 25G
mlxlink -d mt4121_pciconf0.1 -k FC --fec_speed 25G
mlxlink -d mt4121_pciconf0.1 -a TG

176

1 + 8

networking

fiber

nic

interface

sfp

Vilican

7/6/24, 12:12 PM

@PimpJuiceIT - I tried setting speed/FEC manually, but I am still getting the same problem. Firmware of the NIC/router are the latest available.

Pimp Juice IT

7/6/24, 6:32 PM

Check out these tips just in case these help I wanted to share: https://www.excentis.com/blog/sfp-compatibility-issues-here-are-5-troubleshooting-tips/ I wish you good luck in getting this resolved. I don't have a lot of exposure to the SFP fast link configurations with Windows Server 2022.

Vilican

7/6/24, 6:37 PM

Thanks for the resource. However, the connection does not work even with a brand-new DAC cable from FS, which has one end configured for Mikrotik and the other for that Mellanox NIC. Also, both are recognized by the systems, but the link is down for some reason.

Pimp Juice IT

7/6/24, 6:47 PM

Getting close to needing a vendor/hardware manufacturer ticket to assist troubleshooting. Maybe rare but could be a defective cable. I suppose you don't have any known working DEC cable to test with but if so, perhaps that as another steps. I know it sounds weird but disabled and re-enabling the nic from the Windows side via the Windows GUI doesn't do anything. I look forward to hearing what the solution and problem was here regardless.

Vilican

7/6/24, 7:13 PM

Actually, I do have several spare DACs or transceivers, but this link does not work with them either. I tried raising a ticket to vendor support, but they bounced me with a request to purchase their overpriced support plan.

Pimp Juice IT

7/6/24, 8:57 PM

What's the hardware of the server you are working with like make and model? This may hold a clue potentially for what's listed as compatible so consider adding that to your question too. Perhaps it is a compatibility issue. If you have server hardware manufacturer support, maybe opening a ticket with them instead will help troubleshoot (or be a cheaper option to explore). For example: https://community.hpe.com/t5/proliant-servers-ml-dl-sl/mellanox-qsfp-breakout-cable-shows-disconnected-with-hpe-560-sfp/td-p/6994771 just as an example only, not specific to your hardware though

Vilican

7/9/24, 6:58 PM

It's a custom-built server mostly from Supermicro parts. The MB is H13SSL-NT, which has this NIC officially supported. Not sure whether they want to support that, unfortunately.

Pimp Juice IT

7/9/24, 7:56 PM

Start downgrading firmware then to a previous version, all the variations. Try to put in a few calls and see what you get for free call wise both vendors. BIOS firmware and the 25 gbps nic downgrade their firmware and see if you get lucky and find a compatible/bug free pair working nicely.

Score:0

Server

Zac67

7/8/24, 12:03 PM

Judging from your screenshots, both switch and NIC seem to be accepting their respective DAC module.

However, this section

seems to indicate a firmware problem with the NIC's PHY.

You should look for ways to verify and possibly update that firmware.

+ 1

Vilican

7/9/24, 7:04 PM

I have tried re-flashing the NIC firmware using this manual: [https://network.nvidia.com/support/firmware/nic/](https://network.nvidia.com/support/firmware/nic/). However, it still does not work and reports the same message.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Unable to establish link between Mikrotik router and Mellanox NIC

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.