In Ceph cluster, for the same total raw volume, for example 32To each in 5 nodes, how to choose between 16 ssd of 2To or 8 of 4To? Any guidance rules?

Ztevoz Milloz

10/5/23, 11:14 AM

I'm designing a ceph cluster for mixing cephFS and rbd for my company( VM et file infrastructure).

In my set I need 32To raw storage by node. I start with 5 nodes.

The seller quotes propose me to choose beetween 16 ssd of 2 teras or 8 ssd of 4 teras by node.

I mean what is the impact on the IOPS managed by ceph in theses cases, the reconstruction delay etc. May question is highly related with CEPH, not general.

I've follow many guide to drive my choices in many aspects, including ceph documentation and books. But not sure to really have the answer of this question. The only clues I've found are things like 'bigger is better'...

Wich approach I need to follow to choose beetween theses 2 options ?

Here some details if needed : The network nics dedicated to cephs Vlans are 25Gb speed, redondants etc I've considered to multiply at least 4Gb Ram for 1 tera on OSD, so 128 by node to be large. SSD disks are enterprise fitted and read intensive.

Thanks for your help

Cheers

Ztevoz

1 + 1

ceph

Ztevoz Milloz

10/5/23, 6:14 PM

Thanks Nikita, but no. I've just realized that I've no specified Ceph in the title....I've just tagged my question with ceph keyword. So my question is highly related to ceph and about his optimization. I've read the link, interesitng by the way, but global. Ceph is a distributed storage application which is not easy to configure correctly, even you are already aware with all the concepts listed in the links. Cheerz, ZtevOz

Score:0

Server

eblock

10/11/23, 12:00 PM

If those two options are your only options and both SSD models are enterprise SSDs you should be fine with both. Points to consider are scalability, performance and failure domains. The more OSDs you have the more resilient you are to disk failure because less data has to be transferred during recovery. You would also benefit from more parallelization by having more disks which could increase performance. If you don't use all slots and have only 8 disks per node instead of 16 you can add more disks later if your capacity reaches its limit. Don't let your cluster become full, that will be very difficult to get out of. Also plan your capacity to sustain the loss of an entire host so your PGs can be recovered even if an OSD host goes down. And that host failure should also not fill up your cluster, of course.

+ 2

Ztevoz Milloz

10/12/23, 5:09 PM

Hi, thanks eblock for your answer. Yes it's enterprise ssds. I totally agree the reasoning about the osds. For scalability I'll try to leave some shelves free (Dell R750) and add nodes of course. I've difficulties to understand how to compute the slack to sustain losses. Too, I need to undestrand failure domains...Ceph is attractive but can be a problem if I lack of knowledge.

eblock

10/13/23, 1:24 PM

You need to plan how many hosts or OSDs can fail but still have service availability. For example, with a default replicated pool with size 3 and 5 hosts I would use a crush rule to spread all three replicas of a PG to three different hosts (failure domain). If one entire host goes down you still have 4 hosts left, and if your capacity allows it, ceph would recover the missing replicas on the remaining 4 hosts. You could lose another host and would still have all PGs available (with a respective crush rule), but depending on the capacity you'd have a degraded but functional cluster.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: In Ceph cluster, for the same total raw volume, for example 32To each in 5 nodes, how to choose between 16 ssd of 2To or 8 of 4To? Any guidance rules?

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.