Score:2

I need help to achieve (as close as possible) 40G network transfer speed (~1250 MB/s x 4)

me flag

I had a Gigabit network, which I upgraded recently to 40Gbps. To do so, I replace my TPLINK switch with a Cisco Nexus 9396PX switch, which has a module with 12x 40G ports. The 40G network interconnects DELL servers with ESXi7 installed.

The problem is that I am not able to achieve the throughput speed of a 40Gbps network (~1250 MB/s x 4).

  • When I run iperf between two virtual machines, I get a bandwidth ranging from 6 Gbits/sec to 10 Gbits/sec. I never been able to reach the 40 Gbits bandwidth.
  • When I copy files with scp I am getting a transfer speed of about 110-120 MB/s which is very far from the theoretical bandwidth of 1GB/s x 4 ~ 4G/s for a 40G network.

Recently I discovered that encryption reduce transfer speed so, I set up FTP server. Transfer speed doubled with FTP. I got a transfer rate about 240-280 MB/s. I also updated MTU to 9000 on both hosts and VMs, but transfer rates are low.

Can anybody, please, help me to identify and solve the issue? I want 40G Network bandwidth as configured in the Switch to be reflected in Virtual machines, and the transfer speed to reflect as close as possible transfer speed of 40G network (~1250 MB/s x 4).

This is the current architecture details:

  • Cisco Nexus 9396PX Switch with 40G module
  • 4 Dell PowerEdge R930 Servers (VMware ESXi7 hosts)
  • NVME disks on each server
  • 40 Gbps Mellanox NIC on ESXi7 hosts
  • QSFP+ DAC cables to connect servers to switch

UPDATES

I just made 3 more tests. And I can not say there are improvements

FIRST TEST

  1. In Switch, I checked MTU on the server interface and noticed it was not set to Maximum. I tried to set it to 9000 but was denied. It can only be set either to 1500 or 9216. I chosed the highest value.
  2. Still in the Switch I fixed speed to 40G/s and disabled speed auto negociation
  3. I disabled speed auto negociation on ESXi hosts from VCenter.

Then I run the tests again. These changes did not affect the scp copy speed transfer speed between VMs (Ubuntu 22.04). But I noticed a light improvement when downloading from FTP server (it is installed on one of the two ubuntu VMs).

SECOND TEST

I run the following tests on VMs with Windows 10 installed.

  • I added two Windows VM to the host and configured them with MTU set to 9000 too.
  • I created a shared folder for copy betwnn the two Windows VMs.

When I copy files from shared folder to local, the speed is around 600MB/s. But when I copy from local to shared folder, the copy starts at around 350MB/s and the speed decreases when it reaches ~30%.

THIRD TEST

For the last test I installed Filezilla on one the the windows VMs and dowloaded files from/to FTP server (on Ubuntu VM).

  • Dowload speed is around 450MB/s when the download starts and at about ~50% it decreases to ~200MB/s
  • Upload speed is around 200MB/s
in flag
Not my area of expertise, but [this answer](https://serverfault.com/a/1127314/38644) was posted yesterday and could be helpful. Especially the second paragraph.
br flag
@yaoviametepe - Yeah, I agree with Gerald, specifically the second paragraph of my answer to that other user - 40Gbps isn't what most people think it is.
yaoviametepe avatar
me flag
@GeraldSchneider thank you for your answers. This is a valuable information you gave. May be I should focus on how to achieve 1G/s of dowload speed with the one line of 10G. Because I can not even achieve this speed.
Score:2
kz flag

40 gig is basically a bonded 4*10 gig connection. To benefit from the aggregated bandwidth you need multiple TCP sessions running at the same time. You’ll never reach combined estimated bandwidth with just one process and a single TCP session. Some good description could be read here:

https://www.starwindsoftware.com/blog/lacp-vs-mpio-on-windows-platform-which-one-is-better-in-terms-of-redundancy-and-speed-in-this-case-2

yaoviametepe avatar
me flag
Thank you for the answer. It helped me confirm that I am on 10Gbps link, not 40G. I do not know if it could be related to the virtual settings or not. The command `iperf3 -c 192.168.1.80 -P 8 -t 60` output summary is: `...[ 19] 0.00-60.02 sec 7.15 GBytes 1.02 Gbits/sec receiver [SUM] 0.00-60.00 sec 57.0 GBytes 8.16 Gbits/sec 140 sender [SUM] 0.00-60.02 sec 56.9 GBytes 8.15 Gbits/sec receiver`
Zac67 avatar
ru flag
A 40G interface is *not* the same as four bonded 10G interfaces. On the former you can get a single flow up to 40G (with some tuning) and on the latter only up to 10G.
yaoviametepe avatar
me flag
@Zac67 Can you please tell me what values needs to be tuned to achieve 40 Gbits/s throughput in my VMs?
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.