iperf3 errors and retransmissions on ESXi

When running iperf3 to test a 100G connection between a VMware ESXi box and a TrueNAS Core box, I'm seeing the following on the ESXi box:

1. The message "iperf3: getsockopt - Function not implemented" on every interval
2. A large number of retransmissions reported on the first and last intervals but not in the summary stats

Are both of these OK to ignore? Or are they indicative of actual problems?

Here are the details....

Server setup:

Supermicro H12SSL-NT, EPYC 7262, 128GB RAM
TrueNAS 13.0 U3
Mellanox ConnectX-5 100G
iperf3 v3.10.1

Client setup:

Supermicro X9SRE-F, E5-1650 v2, 16GB RAM
ESXi 6.7.0 P08
Mellanox ConnectX-5 100G
iperf3 v3.1.6

Server command:

iperf3 -s -B 192.168.25.5 -p 5101

Client command and output:

[root@esxi:/usr/lib/vmware/vsan/bin] ./iperf3.copy -B 192.168.25.6 -c 192.168.25.5 -i 1 -p 5101 -f g -t 10
Connecting to host 192.168.25.5, port 5101
[  4] local 192.168.25.6 port 27617 connected to 192.168.25.5 port 5101
iperf3: getsockopt - Function not implemented
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  3.08 GBytes  26.4 Gbits/sec  8638800   0.00 Bytes       
iperf3: getsockopt - Function not implemented
[  4]   1.00-2.00   sec  3.03 GBytes  26.0 Gbits/sec    0   0.00 Bytes       
iperf3: getsockopt - Function not implemented
[  4]   2.00-3.00   sec  3.11 GBytes  26.7 Gbits/sec    0   0.00 Bytes       
iperf3: getsockopt - Function not implemented
[  4]   3.00-4.00   sec  2.91 GBytes  25.0 Gbits/sec    0   0.00 Bytes       
iperf3: getsockopt - Function not implemented
[  4]   4.00-5.00   sec  3.01 GBytes  25.9 Gbits/sec    0   0.00 Bytes       
iperf3: getsockopt - Function not implemented
[  4]   5.00-6.00   sec  3.04 GBytes  26.1 Gbits/sec    0   0.00 Bytes       
iperf3: getsockopt - Function not implemented
[  4]   6.00-7.00   sec   883 MBytes  7.41 Gbits/sec    0   0.00 Bytes       
iperf3: getsockopt - Function not implemented
[  4]   7.00-8.00   sec  2.99 GBytes  25.7 Gbits/sec    0   0.00 Bytes       
iperf3: getsockopt - Function not implemented
[  4]   8.00-9.00   sec  1.27 GBytes  10.9 Gbits/sec    0   0.00 Bytes       
iperf3: getsockopt - Function not implemented
[  4]   9.00-10.00  sec  2.75 GBytes  23.7 Gbits/sec  4286328496   0.00 Bytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  26.0 GBytes  22.4 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  26.0 GBytes  22.4 Gbits/sec                  receiver

iperf Done.

This error message is produced due to the TCP statistics not being in the expected structure on "ESXi's Linux".

The feature check in iperf is simply "is it Linux" (https://github.com/esnet/iperf/blob/a2ce47a97acfad3acee91a6cb4da28f9f50c19df/src/tcp_info.c#L63), and expects the data to be found via a getsockopt call for TCP_INFO in a specific structure.

The overall statistics for the test run are still valid as it does not rely on the interval comparison functionality.

There are differences for FreeBSD for example, which has a different data structure being returned compared to the Linux kernel. See https://github.com/esnet/iperf/pull/244 for details.

save_tcpinfo is called to save information at each interval period, but it seems the version of iperf and ESXi's Linux don't play nicely together for interval statistics.

An example of Dell also having this problem in mainline production can be see here: https://www.dell.com/support/kbdoc/en-uk/000187640/dell-emc-vxrail-vsan-hosts-network-performance-test-does-not-exceed-10-000mb-s-bandwidth-on-25gb-or-higher-network-infrastructure.

The above Dell article may also give you some pointers on how to manipulate iperf to get the bandwidth you may have been expecting...