Score:3

Server

Does it make sense to create a zpool with lots of mirror vdevs?

MechMK1

2/1/23, 9:22 AM

To prefix, I am rather new to ZFS. My goal is to create an internal NAS system, with minimal chance of losing all my data.

I've currently acquired four 4TB disks, and I plan to run them in a single zpool with two mirrored vdevs, like so:

  pool: data
 state: ONLINE
  scan: none requested
config:

    NAME                STATE     READ WRITE CKSUM
    data                ONLINE       0     0     0
      mirror-0          ONLINE       0     0     0
        <disk 0>        ONLINE       0     0     0
        <disk 1>        ONLINE       0     0     0
      mirror-1          ONLINE       0     0     0
        <disk 2>        ONLINE       0     0     0
        <disk 3>        ONLINE       0     0     0

When extending the zpool later with more disks, would it make sense to just keep adding mirror-2, mirror-3 with two paired disks each?

1020

0 + 0

zfs

zfsonlinux

Score:2

Server

shodanshok

2/3/23, 7:52 AM

Using multiple mirror vdevs is not an issue for zfs - at all. They provide much higher performance than raidz vdevs and faster resilver. An all-mirrors pool is easier to extend and, with recent zfs versions, even to shrink (ie: vdev removal is supported).

raidz has better space efficiency and, in its raidz2 and raidz3 versions, better resiliency. A raidz pool can be extended with at least 3 disks at time - leading to a new raidz1 vdev. However, such a configuration is not ideal: with big disks you should always used raidz2. This means that expanding a raidz2 vdev requires 4 disks, with no better space efficiency compared to mirrors (ie: 50%), but somewhat better resiliency (any two disks can fail without data loss).

All things considered, when needing good random IO performance I always use mirrors. I only use raidz2 when space efficiency is a bigger concern, and only with reasonably wide stripe (6+ disks).

As you plan to only use 4 disks and to expand later, I suggest you to use plain mirrors.

0 + 0

MechMK1

2/3/23, 6:49 PM

With my current setup, my motherboard can only handle 4 internal drives, so I will use 4x4TB as mirrors. If 8 TB storage won't do, then I can always replace single drives with larger ones, as they get cheaper.

drookie

2/4/23, 2:36 AM

Like SSD never existed, eh ? "Good random I/O performance". World is changed.

shodanshok

2/4/23, 11:37 AM

@drookie An all-SSD pool is less IOPS-constrained, right, but I am under the impression that OP is using HDD rather than SSD. Anyway, the points remain: a raidz vdev has the IOPS of a single drive and it is trickier to expand. Moreover, with only 4 disks a single raidz2 vdev has no space efficiency advantage over plain 2x 2-ways mirrors.

drookie

2/6/23, 6:25 AM

So don't use the raidz2, use single redundancy raidz. Didn't quite get the phrase "raidz vdev has the IOPS of a single drive". Nope, it does not.

shodanshok

2/6/23, 6:06 PM

@drookie single-parity array with big drives is not good idea. Moreover, as stated before, a raidz vdev only provides the IOPS of a *single* drive (but the throughput of multiple drives). You can check [here](https://www.reddit.com/r/zfs/comments/i9fh1x/why_does_raidz_limit_a_vdev_to_the_iops_of_the/) and [here](http://nex7.blogspot.com/2013/03/readme1st.html) for more details, or even testing you array with fio (and a cold cache).

drookie

2/7/23, 9:05 AM

These articles only state that when reading or writing data chunks, somewhere in the chain of these transfers you'll get maximum speed slowed down to the speed of the slowest disk. They don't say that the overall speed will be just it. You're misinterpreting what they say.

shodanshok

2/7/23, 11:49 AM

@drookie first link: "Why does RAIDZ limit a vdev to the **IOPS of the slowest device on it?**" - second link: "1. Virtual Devices Determine IOPS. **IOPS (I/O per second) are mostly a factor of the number of virtual devices (vdevs) in a zpool. They are not a factor of the raw number of disks in the zpool.** This is probably the single most important thing to realize and understand, and is commonly not." But hey - feel free to try yourself (short story: I already did that).

drookie

2/8/23, 8:07 PM

You still don't understand, and, what's even more important, confuse the question comprising an answer (wrong in this case) with actual answer, where the author explains things. That discovery about vdevs still stands for conventional layered raids, 10 and 50. Glad that you're exploring the IT world and you want to share. The thing is - others already did.

Score:1

Server

drookie

2/3/23, 7:06 AM

Depends on. What you are showing is in fact a raid 10. So yeah - you can continue to grow it with more mirror vdevs. Although you should really consider using raidz, because raid10 is merely a waste of disks.

0 + 0

MechMK1

2/3/23, 6:51 PM

[This article](https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz/) states that mirrors should be preferred over raidz. [This answer](https://serverfault.com/a/650120/66501) explains that raidz1 is really not a good idea, as resilvering can become an issue. And raidz2 has the same efficiency as mirroring. Due to hardware constraints, I can't use more than 4 data disks.

drookie

2/4/23, 2:35 AM

The article you referring to is alarmistic. Once there was a load of those, staing that starting from some volume space, your arrayas already contain disk errors, and you should start replacing disks faster that you're insering them in the tray. These articles jongle facts and make incorrect probability theory maths. 50% of storage efficiency is awful. Considering that with zfs you only got only 80% of total space, because after that performance degrades, 50% (which is in fact even less) is an inacceptable level of storage inefficiency. I'm using raridzs in production. For years. Still there.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Does it make sense to create a zpool with lots of mirror vdevs?

TH: มันสมเหตุสมผลไหมที่จะสร้าง zpool ด้วย mirror vdevs จำนวนมาก?

RO: Are sens să creezi un zpool cu o mulțime de vdev-uri în oglindă?

RU: Имеет ли смысл создавать zpool с большим количеством зеркальных vdev?

VI: Có hợp lý không khi tạo một zpool với nhiều vdev nhân bản?

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.