Join Log in
Score:0
JackFrost avatar Server

PCI NVMe mdadm RAID1 performance too slow

ru flag
JackFrost

I know this has been discussed multiple times but I have not found any solution so far that worked so posting here hoping there is some solutions in Dec 2021…

I have a Dell R640 server with dual Xeon Gold processors and 384gb ram. The chassis is only sata/sas drive (does not support u.2) and don’t have budget for new server that supports u.2.

Note - my use case is to provide storage for VM to take advantage of NVMe speeds.

So we opted for PCI card - Dell SSD NVMe M.2 PCI-e 2x Solid State Storage Adapter Card 23PX6 NTRCY. It supports 2 NVME drives and connects via bifurcation to both as x4 PCI lane.

I have two Kingston 2TB nvme drives and I created mdadm based RAID1.

The write performance of a single nvme ssd is 1800MBps. But the RAID1 has write speed of 500MBps.

I found that the Bitmap= Internal was possible problem, and I applied

mdadm <dev> --grow --bitmap=none

Even after this the performance is nearly the same.

Any suggestions on what else I can try?

So I am not sure what happened - today when I ran the speed test again, the speed is within expectations -- Read of 1039 MBps and write of 1352MBps (using crystaldiskmark on a VM on this host)

           mdadm --detail /dev/md0
/dev/md0:
           Version : 1.2
     Creation Time : Sun Nov 28 19:08:22 2021
        Raid Level : raid1
        Array Size : 1953381440 (1862.89 GiB 2000.26 GB)
     Used Dev Size : 1953381440 (1862.89 GiB 2000.26 GB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Thu Dec  2 10:33:50 2021
             State : clean
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

Consistency Policy : bitmap

              Name : server1:0  (local to host server1)
              UUID : 69bab65f:9daa6546:687fc567:bd50164a
            Events : 26478

    Number   Major   Minor   RaidDevice State
       0     259        2        0      active sync   /dev/nvme0n1p1
       1     259        3        1      active sync   /dev/nvme1n1p1
272
0 + 0
Mike Andrews avatar
ng flag
Mike Andrews
Can you post a little more about how you're doing that write benchmark? Also, what's the aggregate performance when you run the benchmark against both SSD's directly at the same time? And, can you post the output of `mdadm --detail` for your array?
1
Reply
JackFrost avatar
ru flag
JackFrost
Too long to post as comment, so answering details below.
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.