Score:3

Server

2x10Gb network : Transfer Huge files with NFS limitied to 6 Gbps

Ealrann

2/12/24, 1:56 PM

I'm transferring big files (78GB) through network, with NFS, but the transfer is finally limited to 6 Gbps (on a 2x10Gb NIC). I cannot find the bottleneck for now. Here my setup:

1 Server

MB: Supermicro X10DRH-C (pcie 3.0), only one cpu
NIC: 2 x 10Gb (intel X710-T2L)
64GB of DDR4 (2400Mhz)
HBA: LSI 9311-8i => 2 mini SAS hd connected to SAS 12Gb backplane
12 HDD: consumer grade, SATA. Toshiba MG09 18TB (So SAS is limited to 6 Gbps per line)

NFS /etc/exports : xxx(rw,async,fsid=0,no_subtree_check,crossmnt). 24 threads for nfsd

I tried to dd big files into all the disks, no slow down, they all write at 200MB/s

2 Clients

NFS configuration: async.

MTU: 9000.

threadripper 3970X, 192GB of ram, copy from a RAID0 of 4 NVME gen 3
2 xeon E5, 512GB of ram, copy from a RAID0 of 4 ssd

Clients are directly connected to the server (no switch), on their own network (10.1.0.0/24 and 10.1.2.0/24).

I tried to play with ring buffer and flow control with ethtool, but it doesn't change anything. No high CPU load.

I'm generally copying 6-8 files at the same time, onto 6-8 different disks => so I'm expecting an average of 9-12Gbps transfer rate. The network spikes sometime to more than 9 Gbps on one interface, for a few seconds, but finally stabilize around 3Gbps on both interfaces. I guess it's due to caching in server ram

I need some help to find the bottleneck please.

Edit: iperf result:

iperf -w 2m result:

352

2 + 7

networking

linux

nfs

sas

linux-networking

Tero Kilkanen

2/13/24, 6:55 PM

How have you configured the bond interface? What is on the other end of 2x10 Gbit/s link?

Ealrann

2/13/24, 7:49 PM

I didn't bond : the server defines 2 ips : 10.1.0.1/24 and 10.1.2.1/24. Client1 is 10.1.0.2 and Client2 is 10.1.2.2. No switch, direct cable. Should I try bonding ?

BaronSamedi1958

2/13/24, 8:39 PM

What do iperf and ttcp say?

Ealrann

2/14/24, 12:21 PM

iperf looks ok, I add pictures on the post. ```iperf``` only shows ~20Gbps, ```iperf -w 2m``` result is ~13Gbps. I didn't try yet with ttcp

Justin Scott

2/14/24, 5:24 PM

When I saw the title with the 6Gpbs limit the first thing my mind went to was the disk where the file is being copied to/from, as SAS would typically be running at about that speed. Perhaps your disks are the bottleneck here.

Ealrann

2/14/24, 5:51 PM

Yes probably, but since I can write more than 6Gbps in local (```dd``` big files in parallel), I'm not sure where is the problem.

Ealrann

2/14/24, 5:52 PM

I think now I'm limited by iowaits, but I don't know what I can do to improve that

Score:3

Server

shodanshok

2/15/24, 7:19 AM

Diagnosing such network issues can be tricky. Please start with the following steps (ideally done one at a time):

set Linux power governor to performance;
enable jumbo frames on both clients and server (from your post I see MTU at 9000 for clients, but it is not specified for the server);
try copying only the first 8 GB of a single file, at least two times in a row. If successive copies are fast, this means your limit is total disk latency/IOPs (network and NFS add some overhead, so this can impact your total delivered IOPs);
try switching between NFS 3 and 4.1, or consider using pNFS;
increase NFS rsize and wsize (see here)

+ 1

kofemann

2/20/24, 10:45 AM

and try to use multiple streams with `nconnect=N` mount option, where **N** is the number of parallel streams.

Score:1

Server

AGI-sysadmin

2/22/24, 4:56 PM

What are you using to copy data? I found standard tools such as cp, mv, rsync don't scale very well and will use bbcp for such transfers. Even though the project is unfortunately abandoned, it's still better. The best I can do is between our nfs v4.0 mounts over shared 10 Gb/s Ethernet line but at night when it's unused I've seen it saturated at 1.2 GB/s before.

bbcp is at https://www.slac.stanford.edu/~abh/bbcp/
last I checked there were forks on Github but they were abandoned as well.
Sometimes the Stanford git doesn't work but if you retry throughout the day you should be able to grab v17.12.00.00.0 from there...

If not, I keep a clean copy of the repo so I can put it on new machines. I just archived it for you with git archive -o bbcp-17.12.00.00.0.tar master then gzip'd it and put it here: https://ava.genome.arizona.edu/pub/bbcp-17.12.00.00.0.tar.gz

Best,

+ 1

Ealrann

2/23/24, 5:34 PM

Yes, it was the problem! Dumb me used nfs to just copy files. Even with a lot of configuration, it doesn't scale at all after a while. Using some TCP dispatching tool (private) changed everything

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: 2x10Gb network : Transfer Huge files with NFS limitied to 6 Gbps

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.