NVMe SSD poor writing performance on Debian 10

I just went for some disk performance tests on my DIY NAS and noticed some very poor performace results with my NVME disk writes !

My NAS runs Debian 10 with this config :

• ASROCK X470D4U with Ryzen 5 3600
• 16 Gb ECC RAM
• 1 NVME for OS (ADATA x8200 Pro 256MB). This is a PCIe Gen 3 x4 on a PCIe Gen 2 x2 slot, given for 3500MB/s Read & 3000MB/s Write.
• 4 SATA HDD in software RAID 6 (WD RED 10TB)
• 1 SATA HDD (WD Green 2TB)

NVME write speed : 320 MB/s !

# time ( dd if=/dev/zero of=test.img bs=1G count=10 status=progress ; sync )
10737418240 bytes (11 GB, 10 GiB) copied, 27 s, 399 MB/s
10+0 records in
10+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 33.5806 s, 320 MB/s

real    0m34.650s
user    0m0.001s
sys     0m8.943s

NVME read speed : 1,7 GB/s

# /sbin/sysctl -w vm.drop_caches=3; time ( dd if=test.img of=/dev/null bs=10G count=5 status=progress )
vm.drop_caches = 3
2147479552 bytes (2.1 GB, 2.0 GiB) copied, 1 s, 1.6 GB/s
dd: warning: partial read (2147479552 bytes); suggest iflag=fullblock
10737397760 bytes (11 GB, 10 GiB) copied, 6 s, 1.7 GB/s
0+5 records in
0+5 records out
10737397760 bytes (11 GB, 10 GiB) copied, 6.27933 s, 1.7 GB/s

real    0m6.358s
user    0m0.000s
sys     0m4.593s

For comparison, here are the results of the RAID 6 (5400 rpm) filesystem and a single HDD (5400 rpm) :

RAID 6 write speed : 275 MB/s
RAID 6 read speed : 290 MB/s
Single disk write speed : 120 MB/s
Single read speed : 125 MB/s

So, I think that the read speed is somehow normal for the NVME drive as it is plugged on a PCIe x2 slot instead of a x4 slot, so 3500 MB/s / 2 should be near the 1,7 GB/s I've found. But I can't explain the poor performance when writing...

Any idea of what is going on ?

Stéphane

Unfortunately, it's now well known that consumer SSD performance vary wildly and rarely matches expectations. There are several problems going on:

1. the first batch of a new SSD series (the batch that will be reviewed by Tom's Hardware, Anandtech, ServeTheHome etc) are generally much better and faster than the following batches. Often, after a few months the hardware changes completely (flash type and speed, sometimes even controller) while retaining the same model number!
2. write speed: there are MLC, TLC and QLC SSDs. TLC and QLC have very low write performance, and use an SLC (single cell) flash cache, generally from 1 to 4 GB in size. The top write speed is only obtained while writing to the SLC cache; any write larger than the cache will slow down dramatically.

You can check the cache effect by running dd with the status=progress and oflag=direct options. You'll see the speed be high for the first 1, 2 or 4 GB (depending upon the cache size), then drop to the TLC or QLC speed afterwards (down to 80-100MB/s on TLC SSDs, or even lower on QLC ones!).