Score:0

How to properly design a 300TB array?

us flag

I'm trying to figure out good practices for building a large 300TB array. Performance is not a priority, only the 300TB usable storage target plus redundancy. And yes I will be taking backups. My planned hardware has 18 HDD slots and 12 HDD slots. Might be able to get another 12 HDD slots.

Right now I was thinking of having a 16TB x 16 raid 60 array + 2 hot spares for 192TB And 16TB x 10 raid 6 + 2 hot spares for 128TB

For 320TB usable total

I've been reading that raid arrays with large drives is not advised due to rebuild times.

paladin avatar
id flag
Better make only 4 disk RAID6 arrays and combine these as a JBOD. Better lifetime and easier to manage and to replace.
paladin avatar
id flag
What I may also recommend to you is to use btrfs RAID1. But you should carefully read about the btrfs filesystem before doing so, cause btrfs RAID1 differs a bit from the conventional RAID1, but has also many advantages (file checksum, cow, separate metafiles, integrated LVM/RAID). Whatever you do, do not use btrfs RAID5 nor btrfs RAID6, as those RAID modes are highly experimental (These experimental btrfs RAID5/RAID6 modes are currently more dangerous than RAID0!). Start to read here -> [btrfs Wiki](https://btrfs.wiki.kernel.org/index.php/Main_Page)
Score:0
br flag

RAID arrays with big disks is just fine, yes there can be delays in rebuilds but that's fine so long as you use something like R6/60 where there's still parity during rebuilds.

My biggest worry with this would be the 'all eggs in one basket' thing with just the one server - have you considered using a distributed filesystem like Ceph at all? You'd end up with more, but smaller, servers and could spread them out over racks for extra resilience - just an idea anyway.

What I'd think about for your main question would be to get a normal 2U server and use one or two external SAS disk enclosures connected to one or two PCIe SAS RAID controllers - this way you can upgrade your server - or storage - without impacting the other. It shouldn't cost too much either and there are lots of options, one big enclosure with all your disks or two, perhaps smaller, ones and then split the 0 bit of your R60 across enclosures - you can even 'dual-link' then to the RAID arrays for added resilience.

Oh and the hot-spares is a good idea too, definitely do that.

One final thing to consider - take a really close look at the actual disks you want to use - make sure they have a '100% duty cycle' (i.e. can be running 24hrs for years at a time), not all are, and make sure you don't use 'shingled' disks - they're fine for home use but their write speed is just awful in a server environment.

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.