Dell R530 Broadcom NetXtreme tg3 nic limited to 12 MBytes/sec

I have an older Dell R530:

Manufacturer    Dell Inc.
Model   PowerEdge R530
CPU 16 CPUs x Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz
Memory  63.78 GB

it has 8 NIC's, 3 of which are tg3, Broadcom NetXtreme BCM5720's:

vmnic0  0000:02:00.0  tg3     Up            Up            1000  Full    14:18:77:5e:a6:de  1500  Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic2  0000:03:00.0  tg3     Up            Up            1000  Full    14:18:77:5e:a6:e0  1500  Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic3  0000:03:00.1  tg3     Up            Down             0  Half    14:18:77:5e:a6:e1  1500  Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet

For the life of me, I can not get them to send traffic at a higher rate than ~12MBytes/second:

---------------------------------------------
Server listening on 5201
---------------------------------------------
Accepted connection from 192.168.1.117, port 50971
[  5] local 192.168.1.148 port 5201 connected to 192.168.1.117 port 50972
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec  11.2 MBytes  93.6 Mbits/sec
[  5]   1.00-2.00   sec  11.2 MBytes  93.9 Mbits/sec
[  5]   2.00-3.00   sec  11.2 MBytes  93.9 Mbits/sec
[  5]   3.00-4.00   sec  11.2 MBytes  94.1 Mbits/sec
[  5]   4.00-5.00   sec  11.2 MBytes  94.0 Mbits/sec
[  5]   5.00-6.00   sec  11.2 MBytes  94.0 Mbits/sec
[  5]   6.00-7.00   sec  11.2 MBytes  94.1 Mbits/sec
[  5]   7.00-8.00   sec  11.2 MBytes  94.1 Mbits/sec
[  5]   8.00-9.00   sec  11.2 MBytes  94.1 Mbits/sec
[  5]   9.00-10.00  sec  11.2 MBytes  94.1 Mbits/sec
[  5]  10.00-10.05  sec   573 KBytes  93.9 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.05  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.05  sec   113 MBytes  94.0 Mbits/sec                  receiver

Just for proof that it isn't the network, other servers get expected performance:

[  6] local 192.168.1.117 port 50998 connected to 192.168.1.242 port 5201
[ ID] Interval           Transfer     Bandwidth
[  6]   0.00-1.00   sec  75.9 MBytes   637 Mbits/sec
[  6]   1.00-2.00   sec   113 MBytes   944 Mbits/sec
[  6]   2.00-3.00   sec   113 MBytes   944 Mbits/sec
[  6]   3.00-4.00   sec   112 MBytes   935 Mbits/sec
[  6]   4.00-5.00   sec   113 MBytes   946 Mbits/sec
[  6]   5.00-6.00   sec   112 MBytes   939 Mbits/sec
[  6]   6.00-7.00   sec   110 MBytes   922 Mbits/sec
[  6]   7.00-8.00   sec   109 MBytes   918 Mbits/sec
[  6]   8.00-9.00   sec   102 MBytes   856 Mbits/sec
[  6]   9.00-10.00  sec   102 MBytes   854 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  6]   0.00-10.00  sec  1.04 GBytes   890 Mbits/sec                  sender
[  6]   0.00-10.00  sec  1.04 GBytes   889 Mbits/sec                  receiver

I've updated firmware/drivers as much as I can. According to https://www.broadcom.com/products/ethernet-connectivity/network-ics/bcm5720-1gbase-t-ic it looks like the latest firmware is 3.137 which I have installed:

[root@esxi:~] esxcli network nic get -n vmnic0
   Advertised Auto Negotiation: true
   Advertised Link Modes: 10baseT/Half, 10baseT/Full, 100baseT/Half, 100baseT/Full, 1000baseT/Half, 1000baseT/Full
   Auto Negotiation: true
   Cable Type: Twisted Pair
   Current Message Level: 4260095
   Driver Info:
         Bus Info: 0000:02:00.0
         Driver: tg3
         Firmware Version: FFV7.10.64 bc 5720-v1.36
         Version: 3.137l.v60.1
   Link Detected: true
   Link Status: Up
   Name: vmnic0
   PHYAddress: 1
   Pause Autonegotiate: true
   Pause RX: true
   Pause TX: true
   Supported Ports: TP
   Supports Auto Negotiation: true
   Supports Pause: true
   Supports Wakeon: true
   Transceiver: internal
   Virtual Address: 00:00...[snip]
   Wakeon: MagicPacket(tm)

All of the diagnostics/status report that it is connected at 1000 full and I've tried multiple NIC's, cables and switches all with the same result.

I've spent hours trying to figure this out, and all of the google searching and testing have gotten me nowhere. Anyone have any ideas what else I can try or check?

I've also tried setting the link speed manually, but it does auto-negotiate to 1000 already.

If all the physical links show 1000 Mbit/s pretty much the only things left are:

• the ESXi port group the VMkernel adapter(?) attaches to has a traffic limit set
• the switch the traffic runs through has some kind of QoS limit/traffic policing set up

By the way, the advertised modes of a NIC/switch port don't matter. The negotiated mode does.

Also, Broadcom NICs usually support "Ethernet@Wirespeed" which makes a 1000BASE-T NIC fall back to 100BASE-FX when one of the twisted pairs in a cable fails (standard 1000BASE-T doesn't link at all in such a case). So, even if everything is configured correctly, a bad cable might throw you back to 100 Mbit/s.

Turns out it was an issue with cabling. The hint that should have tipped me off was the 94.9 Mbits/sec limit--obviously the connection was running 100Mbps. The fact that it was set for 1G and everything was saying it was 1G, it should have tipped me off that something in the wiring was messed up and causing it to run at 100Mbps. I had the datacenter switch out the wires today and it's running at 1G as it should.

I was incorrect when stating that there was other equipment in the rack running at 1G. I installed a new switch in the rack and once I looked at the interface, I noticed the port was only reporting 100Mbps. Once I saw this, I double-checked and a NAS device I had in the rack would indeed only copy large files at <= 100Mbps.

Apologies for the misleading info in the question. I will leave it up in case some poor soul is having the same issue somewhere down the line: if you see 11.3MB/sec or 11.2MB/sec or 94.9Mb/sec consistently, DOUBLE CHECK THAT YOUR PORT ISN'T ACTUALLY RUNNING AT 100Mbps even if everything is saying it is! I didn't see it until installing a smart switch/router that reported the actual number rather than what some config was or what was auto-negotiated.