Why is the read/write speed of two SSDs so slow when mounted as single logical volume with LVM under Ubuntu 18.04 LTS?

Tilman Hartwig

7/1/23, 6:42 AM

For scientific computing, we need fast read/write speeds. So I upgraded one of our machines with 2 SSDs. I used the command

hdparm -Tt /dev/sda

to test the read speed. As reference, our standard HDDs have a read speed of 250MB/s. Each SSD separately has a tested read speed of 2000MB/s (roughly expected).

However, when I connect the two SSDs to one logical volume (following the top-voted answer here: How to set up multiple hard drives as one volume?), then the read speed of the logical volume is only 200MB/s – much slower than the individual SSDs and even slower than the HDDs…

Is this normal? My understanding of LVM is very basic. I thought files are written to both SSDs simultaneously and therefore the read speed of the combined logical volume might even be higher than that of one single SSD. I also expect some overhead of the LVM, so I do not expect the read speed to be twice that of a single SSD. But only 10% the speed of a single SSD?

I guess my question is: is this normal or did I do something wrong? What is the expected read speed for the logical volume and how can I achieve it? As solution, we could just use the two SSDs as individual volumes, but it would be slightly less convenient. And for the expected I/O speed, I do not expect an exact numerical value, but rather if it is smaller, similar or greater than that of a single SSD.

The two SSDs are

Samsung 980 1TB PCIe Gen 3.0 x 4 NVMe M.2

and they are connected via

Ableconn PEXM2-130 Dual PCIe NVMe M.2 SSDs Carrier Adapter.

EDIT/SOLVED

I am still not sure what the original problem was, but when I followed this instruction, everything works well: https://www.theurbanpenguin.com/striped-lvm-volumes/ Now, my logical striped volume of two SSDs has a read speed that is slightly above the read speed of a single SSD (~2500MB/s compared to 2000MB/s single). I hope my experience helps other people as well.

0 + 0

ssd

lvm

18.04

FedKad

7/1/23, 8:22 AM

Can you test the two SSDs ***at the same time***, instead of each one separately? Can it be that the controller is the bottleneck?

Tilman Hartwig

7/1/23, 8:59 AM

Thank you very much for this suggestion, @FedKad. I have tried it, but the controller does not seem to be the bottleneck: When I test the read speed for both SSDs simultaneously, they both can read with about 2000MB/s.

FedKad

7/1/23, 9:07 AM

Also, can you test with a newer version of Ubuntu like 20.04 or 21.10 (live session)? Please give details on how you use LVM (mirroring, striping, etc.).

Tilman Hartwig

7/1/23, 10:47 AM

I am afraid it is difficult to test these SSDs in a live session: the machine is a working node of a cluster and can not be rebooted atm. I can test it during the next cluster maintenance. Mirroring/striping: I am new to LVM and simply followed the guide linked above. So I guess by default I did not set up mirroring/striping. However, I will look into "striping" which might be useful for our purpose. Thanks! Based on your questions, I guess you also find the small read speed of the SSDs suspicious, right?

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Why is the read/write speed of two SSDs so slow when mounted as single logical volume with LVM under Ubuntu 18.04 LTS?

TH: เหตุใดความเร็วในการอ่าน/เขียนของ SSD สองตัวจึงช้ามากเมื่อติดตั้งเป็นโลจิคัลวอลุ่มเดียวด้วย LVM ภายใต้ Ubuntu 18.04 LTS

RO: De ce este viteza de citire/scriere a două SSD-uri atât de lentă când sunt montate ca un singur volum logic cu LVM sub Ubuntu 18.04 LTS?

RU: Почему скорость чтения/записи двух твердотельных накопителей такая низкая, когда они смонтированы как один логический том с помощью LVM под Ubuntu 18.04 LTS?

VI: Tại sao tốc độ đọc/ghi của hai ổ SSD lại chậm như vậy khi được gắn thành một ổ đĩa logic duy nhất với LVM trong Ubuntu 18.04 LTS?

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.