Score:3

2x10Gb network : Transfer Huge files with NFS limitied to 6 Gbps

kw flag

I'm transferring big files (78GB) through network, with NFS, but the transfer is finally limited to 6 Gbps (on a 2x10Gb NIC). I cannot find the bottleneck for now. Here my setup:

1 Server

  • MB: Supermicro X10DRH-C (pcie 3.0), only one cpu
  • NIC: 2 x 10Gb (intel X710-T2L)
  • 64GB of DDR4 (2400Mhz)
  • HBA: LSI 9311-8i => 2 mini SAS hd connected to SAS 12Gb backplane
  • 12 HDD: consumer grade, SATA. Toshiba MG09 18TB (So SAS is limited to 6 Gbps per line)

NFS /etc/exports : xxx(rw,async,fsid=0,no_subtree_check,crossmnt). 24 threads for nfsd

I tried to dd big files into all the disks, no slow down, they all write at 200MB/s

2 Clients

NFS configuration: async.

MTU: 9000.

  • threadripper 3970X, 192GB of ram, copy from a RAID0 of 4 NVME gen 3
  • 2 xeon E5, 512GB of ram, copy from a RAID0 of 4 ssd

Clients are directly connected to the server (no switch), on their own network (10.1.0.0/24 and 10.1.2.0/24).

I tried to play with ring buffer and flow control with ethtool, but it doesn't change anything. No high CPU load.

I'm generally copying 6-8 files at the same time, onto 6-8 different disks => so I'm expecting an average of 9-12Gbps transfer rate. The network spikes sometime to more than 9 Gbps on one interface, for a few seconds, but finally stabilize around 3Gbps on both interfaces. I guess it's due to caching in server ram

I need some help to find the bottleneck please.

nload

Edit: iperf result: enter image description here

iperf -w 2m result: enter image description here

us flag
How have you configured the bond interface? What is on the other end of 2x10 Gbit/s link?
Ealrann avatar
kw flag
I didn't bond : the server defines 2 ips : 10.1.0.1/24 and 10.1.2.1/24. Client1 is 10.1.0.2 and Client2 is 10.1.2.2. No switch, direct cable. Should I try bonding ?
BaronSamedi1958 avatar
kz flag
What do iperf and ttcp say?
Ealrann avatar
kw flag
iperf looks ok, I add pictures on the post. ```iperf``` only shows ~20Gbps, ```iperf -w 2m``` result is ~13Gbps. I didn't try yet with ttcp
Justin Scott avatar
lu flag
When I saw the title with the 6Gpbs limit the first thing my mind went to was the disk where the file is being copied to/from, as SAS would typically be running at about that speed. Perhaps your disks are the bottleneck here.
Ealrann avatar
kw flag
Yes probably, but since I can write more than 6Gbps in local (```dd``` big files in parallel), I'm not sure where is the problem.
Ealrann avatar
kw flag
I think now I'm limited by iowaits, but I don't know what I can do to improve that
Score:3
ca flag

Diagnosing such network issues can be tricky. Please start with the following steps (ideally done one at a time):

  • set Linux power governor to performance;

  • enable jumbo frames on both clients and server (from your post I see MTU at 9000 for clients, but it is not specified for the server);

  • try copying only the first 8 GB of a single file, at least two times in a row. If successive copies are fast, this means your limit is total disk latency/IOPs (network and NFS add some overhead, so this can impact your total delivered IOPs);

  • try switching between NFS 3 and 4.1, or consider using pNFS;

  • increase NFS rsize and wsize (see here)

de flag
and try to use multiple streams with `nconnect=N` mount option, where **N** is the number of parallel streams.
Score:1
us flag

What are you using to copy data?  I found standard tools such as cp, mv, rsync don't scale very well and will use bbcp for such transfers.  Even though the project is unfortunately abandoned, it's still better.  The best I can do is between our nfs v4.0 mounts over shared 10 Gb/s Ethernet line but at night when it's unused I've seen it saturated at 1.2 GB/s before.

bbcp is at https://www.slac.stanford.edu/~abh/bbcp/
last I checked there were forks on Github but they were abandoned as well.
Sometimes the Stanford git doesn't work but if you retry throughout the day you should be able to grab v17.12.00.00.0 from there...

If not, I keep a clean copy of the repo so I can put it on new machines. I just archived it for you with git archive -o bbcp-17.12.00.00.0.tar master then gzip'd it and put it here: https://ava.genome.arizona.edu/pub/bbcp-17.12.00.00.0.tar.gz

Best,

I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.