How do I maximize nvme raid0(md) performance (in linux)?

Horatiu

10/20/22, 11:42 AM

Nvme raid0 read(and write) performance is sometimes worse than the performance of one individual underlying drive.

hdparm -t /dev/md0 :

Timing buffered disk reads: 7054 MB in  3.00       seconds = 2350.46 MB/sec

hdparm -t /dev/nvme0n1 :

Timing buffered disk reads: 7160 MB in  3.00    seconds = 2386.32 MB/sec

This is a datacenter hosted bare metal unit, the drives and the server are cooled properly so there is no thermal throttling due to high temps.
Drives are at about 45 C on average, as shown by nvme smart-log /dev/nvmeX
Something is wrong with those low array speeds and I want to understand what. These are the full specs:

Epyc 7502p [2nd gen Rome 32cores/64threads]  
128 GB DDR4 32x4 3200mhz   
3x 3.8 TB samsung nvme pcie3 enteprise SSDs   
Raid0 software created with mdadm   
xfs filesystem on top of md0 created with defaults 'mkfs.xfs /dev/md0' 
debian 10 64bit with latest updates and stock kernel (no tinkering)

What am I doing wrong ? What am I not doing at all ? I know there are diminishing returns for any raid when it comes to speed but this is bad.

PS: off the top of your head, what raid0 read speed do you expect that system to have ?

732

0 + 2

performance

xfs

raid0

mdraid

nvme

Michael Hampton

10/20/22, 12:08 PM

Use a more reasonable benchmarking tool such as `fio` to test your speeds. `hdparm` is just this side of useless.

Stuka

10/20/22, 12:46 PM

I would agree about using fio. I've never tested storage performance with hdparm. Fio examples: https://docs.oracle.com/en-us/iaas/Content/Block/References/samplefiocommandslinux.htm

Score:1

Server

Dmitry Nosachev

2/8/23, 2:22 PM

Michael Hampton and Stuka are right: use fio for benchmarking. Example:

sudo fio --filename=/dev/md0 --direct=1 --rw=read --bs=64k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 --time_based --group_reporting --name=seq_read

0 + 0

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: How do I maximize nvme raid0(md) performance (in linux)?

TH: ฉันจะเพิ่มประสิทธิภาพ nvme raid0 (md) ให้สูงสุด (ใน linux) ได้อย่างไร

RO: Cum maximizez performanța nvme raid0(md) (în Linux)?

RU: Как увеличить производительность nvme raid0(md) (в Linux)?

VI: Làm cách nào để tối đa hóa hiệu suất nvme RAID0(md) (trong linux)?

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.