I'm transferring big files (78GB) through network, with NFS, but the transfer is finally limited to 6 Gbps (on a 2x10Gb NIC). I cannot find the bottleneck for now. Here my setup:
1 Server
- MB: Supermicro X10DRH-C (pcie 3.0), only one cpu
- NIC: 2 x 10Gb (intel X710-T2L)
- 64GB of DDR4 (2400Mhz)
- HBA: LSI 9311-8i => 2 mini SAS hd connected to SAS 12Gb backplane
- 12 HDD: consumer grade, SATA. Toshiba MG09 18TB (So SAS is limited to 6 Gbps per line)
NFS /etc/exports : xxx(rw,async,fsid=0,no_subtree_check,crossmnt).
24 threads for nfsd
I tried to dd
big files into all the disks, no slow down, they all write at 200MB/s
2 Clients
NFS configuration: async.
MTU: 9000.
- threadripper 3970X, 192GB of ram, copy from a RAID0 of 4 NVME gen 3
- 2 xeon E5, 512GB of ram, copy from a RAID0 of 4 ssd
Clients are directly connected to the server (no switch), on their own network (10.1.0.0/24 and 10.1.2.0/24).
I tried to play with ring buffer and flow control with ethtool
, but it doesn't change anything.
No high CPU load.
I'm generally copying 6-8 files at the same time, onto 6-8 different disks => so I'm expecting an average of 9-12Gbps transfer rate. The network spikes sometime to more than 9 Gbps on one interface, for a few seconds, but finally stabilize around 3Gbps on both interfaces. I guess it's due to caching in server ram
I need some help to find the bottleneck please.
Edit: iperf
result:
iperf -w 2m
result: