Score:1

Simple way to dynamically increase volume size regularly

de flag

I have a pair of physical servers with 32TB of raw disk space each (4 * 8TB x 2) in a ZFS pool on Ubuntu. After formatting, I am currently at 86% used and my data volume is growing at a reasonably constant 50G per week. I've recently setup the ZFS volume on the 4 disks from just 4 separate volumes (synced to the other server for some redundancy) so I don't have to manage 4 volumes (x 2 servers) and do less balancing of each volume as it's now just 1 large volume.

I'm soon going to need to add another disk to the volume (technically 2, one per server), but I already know that you just can't dynamically add a disk to an existing ZFS volume set to increase it's overall size, you just end up with a "safer" old volume (with parity) and a completely "unsafe" new volume.

My question is: How do I regularly make the volume larger as the volume fills up? I'm also not able to just buy a pile of disks and make the volume huge and move this problem far into the future.

I'm wondering about the "steady state" of an ever growing volume, and will I just eventually end up where I was, with multiple volumes of different sizes and having to manually balance the usage across the volumes??

Update: Unfortunately the data is already de-duplicated, and will continue to grow for the foreseeable future. I suspect it will eventually stop growing, or old data could be purged, but I don't see that happening for a couple of years. (Thanks for the ideas though)

cn flag
Not an answer, but if your space is constantly growing you need to think about data retention...
djdomi avatar
za flag
and if the same data exists, consider an upgrade of ram and use deduplication. might a risk but also a chance
Romeo Ninov avatar
in flag
Check `autoextend` property: https://docs.oracle.com/cd/E19120-01/open.solaris/817-2271/githb/index.html
millebi avatar
de flag
I don't know if autoextend is available in Linux... but I'll have a look. Thanks @RomeoNinov!
Score:0
gf flag

your problem is also well describe as the scale-up vs scale-out problem.

it is an infinite source of troubles if you want to stick with ZFS because of the replacement of some vdev, or extension of zpool, however there is a (new) solution with some recent release of "TrueNas Scale" which bring HCI (hyperconvergence infrastructure) to ZFS, so one can add "node" like structure to existing storage so you are not limited to the number of disk your server will accept, you can just add servers.

From my point of view I would prefer a Ceph infrastructure that you can make grow, but if your already familiar with ZFS "TrueNAs Scale" will be easier to manipulate.

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.