I had a Gigabit network, which I upgraded recently to 40Gbps.
To do so, I replace my TPLINK switch with a Cisco Nexus 9396PX switch, which has a module with 12x 40G ports. The 40G network interconnects DELL servers with ESXi7 installed.
The problem is that I am not able to achieve the throughput speed of a 40Gbps network (~1250 MB/s x 4).
- When I run
iperf
between two virtual machines, I get a bandwidth ranging from 6 Gbits/sec to 10 Gbits/sec. I never been able to reach the 40 Gbits bandwidth.
- When I copy files with
scp
I am getting a transfer speed of about 110-120 MB/s which is very far from the theoretical bandwidth of 1GB/s x 4 ~ 4G/s for a 40G network.
Recently I discovered that encryption reduce transfer speed so, I set up FTP server. Transfer speed doubled with FTP. I got a transfer rate about 240-280 MB/s.
I also updated MTU to 9000 on both hosts and VMs, but transfer rates are low.
Can anybody, please, help me to identify and solve the issue?
I want 40G Network bandwidth as configured in the Switch to be reflected in Virtual machines, and the transfer speed to reflect as close as possible transfer speed of 40G network (~1250 MB/s x 4).
This is the current architecture details:
- Cisco Nexus 9396PX Switch with 40G module
- 4 Dell PowerEdge R930 Servers (VMware ESXi7 hosts)
- NVME disks on each server
- 40 Gbps Mellanox NIC on ESXi7 hosts
- QSFP+ DAC cables to connect servers to switch
UPDATES
I just made 3 more tests. And I can not say there are improvements
FIRST TEST
- In Switch, I checked MTU on the server interface and noticed it was not set to Maximum. I tried to set it to 9000 but was denied. It can only be set either to 1500 or 9216. I chosed the highest value.
- Still in the Switch I fixed speed to 40G/s and disabled speed auto negociation
- I disabled speed auto negociation on ESXi hosts from VCenter.
Then I run the tests again. These changes did not affect the scp
copy speed transfer speed between VMs (Ubuntu 22.04).
But I noticed a light improvement when downloading from FTP server (it is installed on one of the two ubuntu VMs).
SECOND TEST
I run the following tests on VMs with Windows 10 installed.
- I added two Windows VM to the host and configured them with MTU set to 9000 too.
- I created a shared folder for copy betwnn the two Windows VMs.
When I copy files from shared folder to local, the speed is around 600MB/s. But when I copy from local to shared folder, the copy starts at around 350MB/s and the speed decreases when it reaches ~30%.
THIRD TEST
For the last test I installed Filezilla on one the the windows VMs and dowloaded files from/to FTP server (on Ubuntu VM).
- Dowload speed is around 450MB/s when the download starts and at about ~50% it decreases to ~200MB/s
- Upload speed is around 200MB/s