Score:1

Ceph reading and writing performance problems, fast reading and slow writing

ws flag

Ceph reading and writing performance problems, fast reading and slow writing

Hello, we need to migrate all cloud environments to Proxmox. At present, I am evaluating and testing Proxmox+Ceph+OpenStack.

But now we are facing the following difficulties:

  1. When VMware vSAN was migrated to ceph, I found that hdd+ssd performed very poorly in ceph, and the write performance was very poor. Performance is far less than vSAN
  2. The sequential writing performance of ceph in the full flash memory structure is not as good as that of a single hard disk, or even a single mechanical hard disk
  3. In using the hdd+ssd structure in bcache, the sequential write performance of ceph is far lower than that of a single hard disk

Please forgive my poor English.

Test server parameters (this is not important)

CPU:Dual Intel® Xeon® E5-2698Bv3

Memory: 8 x 16G DDR3

Dual 1 Gbit NIC:Realtek Semiconductor Co., Ltd. RTL8111/8168/8411

Disk:

1 x 500G NVME SAMSUNG MZALQ512HALU-000L1 (It is also the ssd-data Thinpool in PVE)

1 x 500G SATA WDC_WD5000AZLX-60K2TA0 (Physical machine system disk)

2 x 500G SATA WDC_WD5000AZLX-60K2TA0

1 x 1T SATA ST1000LM035-1RK172

PVE:pve-manager/7.3-4/d69b70d4 (running kernel: 5.15.74-1-pve)

Network Configure:

enp4s0 (OVS Port) -> vmbr0 (OVS Bridge) -> br0mgmt (192.168.1.3/24,192.168.1.1)

enp5s0 (OVS Port,MTU=9000) -> vmbr1 (OVS Bridge,MTU=9000)

vmbr2 (OVS Bridge,MTU=9000)

Test virtual machine parameters x 3 (three virtual machines are the same parameters)

CPU:32 (1 sockets, 32 cores) [host]

Memory:32G

Disk:

1 x local-lvm:vm-101-disk-0,iothread=1,size=32G

2 x ssd-data:vm-101-disk-0,iothread=1,size=120G

Network Device:

net0: bridge=vmbr0,firewall=1

net1: bridge=vmbr2,firewall=1,mtu=1 (Ceph Cluster/Public Network)

net2: bridge=vmbr0,firewall=1

net3: bridge=vmbr0,firewall=1

Network Configure:

ens18 (net0,OVS Port) -> vmbr0 (OVS Bridge) -> br0mgmt (10.10.1.11/24,10.10.1.1)

ens19 (net1,OVS Port,MTU=9000) -> vmbr1 (OVS Bridge,MTU=9000) -> br1ceph (192.168.10.1/24,MTU=9000)

ens20 (net2,Network Device,Active=No)

ens21 (net3,Network Device,Active=No)

Benchmarking tools

  1. fio
  2. fio-cdm (https://github.com/xlucn/fio-cdm)

For fio-cdm, if no parameters are filled in, the configuration file corresponding to fio is as follows

Use 'python fio-cdm - f -' to get

[global]
ioengine=libaio
filename=.fio_testmark
directory=/root
size=1073741824.0
direct=1
runtime=5
refill_buffers
norandommap
randrepeat=0
allrandrepeat=0
group_reporting

[seq-read-1m-q8-t1]
rw=read
bs=1m
rwmixread=0
iodepth=8
numjobs=1
loops=5
stonewall

[seq-write-1m-q8-t1]
rw=write
bs=1m
rwmixread=0
iodepth=8
numjobs=1
loops=5
stonewall

[seq-read-1m-q1-t1]
rw=read
bs=1m
rwmixread=0
iodepth=1
numjobs=1
loops=5
stonewall

[seq-write-1m-q1-t1]
rw=write
bs=1m
rwmixread=0
iodepth=1
numjobs=1
loops=5
stonewall

[rnd-read-4k-q32-t16]
rw=randread
bs=4k
rwmixread=0
iodepth=32
numjobs=16
loops=5
stonewall

[rnd-write-4k-q32-t16]
rw=randwrite
bs=4k
rwmixread=0
iodepth=32
numjobs=16
loops=5
stonewall

[rnd-read-4k-q1-t1]
rw=randread
bs=4k
rwmixread=0
iodepth=1
numjobs=1
loops=5
stonewall

[rnd-write-4k-q1-t1]
rw=randwrite
bs=4k
rwmixread=0
iodepth=1
numjobs=1
loops=5
stonewall

Environment construction steps

# prepare tools
root@pve01:~# apt update -y && apt upgrade -y
root@pve01:~# apt install fio git -y
root@pve01:~# git clone https://github.com/xlucn/fio-cdm.git

# create test block
root@pve01:~# rbd create test -s 20G
root@pve01:~# rbd map test
root@pve01:~# mkfs.xfs /dev/rbd0
root@pve01:~# mkdir /mnt/test
root@pve01:/mnt# mount /dev/rbd0 /mnt/test

# start test
root@pve01:/mnt/test# python3 ~/fio-cdm/fio-cdm

Environmental test

  1. Network Bandwidth
root@pve01:~# apt install iperf3 -y
root@pve01:~# iperf3 -s
---------------------------------------------
Server listening on 5201
---------------------------------------------
Accepted connection from 10.10.1.12, port 52968
[  5] local 10.10.1.11 port 5201 connected to 10.10.1.12 port 52972
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.87 GBytes  16.0 Gbits/sec                  
[  5]   1.00-2.00   sec  1.92 GBytes  16.5 Gbits/sec                  
[  5]   2.00-3.00   sec  1.90 GBytes  16.4 Gbits/sec                  
[  5]   3.00-4.00   sec  1.90 GBytes  16.3 Gbits/sec                  
[  5]   4.00-5.00   sec  1.85 GBytes  15.9 Gbits/sec                  
[  5]   5.00-6.00   sec  1.85 GBytes  15.9 Gbits/sec                  
[  5]   6.00-7.00   sec  1.70 GBytes  14.6 Gbits/sec                  
[  5]   7.00-8.00   sec  1.75 GBytes  15.0 Gbits/sec                  
[  5]   8.00-9.00   sec  1.89 GBytes  16.2 Gbits/sec                  
[  5]   9.00-10.00  sec  1.87 GBytes  16.0 Gbits/sec                  
[  5]  10.00-10.04  sec  79.9 MBytes  15.9 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.04  sec  18.6 GBytes  15.9 Gbits/sec                  receiver

  1. Jumbo Frames
root@pve01:~# ping -M do -s 8000 192.168.10.2
PING 192.168.10.2 (192.168.10.2) 8000(8028) bytes of data.
8008 bytes from 192.168.10.2: icmp_seq=1 ttl=64 time=1.51 ms
8008 bytes from 192.168.10.2: icmp_seq=2 ttl=64 time=0.500 ms
^C
--- 192.168.10.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 0.500/1.007/1.514/0.507 ms
root@pve01:~# 

Benchmark category

  1. Physical Disk Benchmark
  2. Single osd, single server benchmark
  3. Multiple OSDs, single server benchmarks
  4. Multiple OSDs, multiple server benchmarks

Benchmark results (Ceph and the system have not been tuned, and bcache acceleration has not been used)

1. Physical Disk Benchmark (Test sequence is 4)

step.

root@pve1:~# lsblk
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                            8:0    0 465.8G  0 disk 
├─sda1                         8:1    0  1007K  0 part 
├─sda2                         8:2    0   512M  0 part /boot/efi
└─sda3                         8:3    0 465.3G  0 part 
  ├─pve-root                 253:0    0    96G  0 lvm  /
  ├─pve-data_tmeta           253:1    0   3.5G  0 lvm  
  │ └─pve-data-tpool         253:3    0 346.2G  0 lvm  
  │   ├─pve-data             253:4    0 346.2G  1 lvm  
  │   └─pve-vm--100--disk--0 253:5    0    16G  0 lvm  
  └─pve-data_tdata           253:2    0 346.2G  0 lvm  
    └─pve-data-tpool         253:3    0 346.2G  0 lvm  
      ├─pve-data             253:4    0 346.2G  1 lvm  
      └─pve-vm--100--disk--0 253:5    0    16G  0 lvm  
sdb                            8:16   0 931.5G  0 disk 
sdc                            8:32   0 465.8G  0 disk 
sdd                            8:48   0 465.8G  0 disk 
nvme0n1                      259:0    0 476.9G  0 disk 
root@pve1:~# mkfs.xfs /dev/nvme0n1 -f
root@pve1:~# mkdir /mnt/nvme
root@pve1:~# mount /dev/nvme0n1 /mnt/nvme
root@pve1:~# cd /mnt/nvme/

result.

root@pve1:/mnt/nvme# python3 ~/fio-cdm/fio-cdm
tests: 5, size: 1.0GiB, target: /mnt/nvme 3.4GiB/476.7GiB
|Name        |  Read(MB/s)| Write(MB/s)|
|------------|------------|------------|
|SEQ1M Q8 T1 |     2361.95|     1435.48|
|SEQ1M Q1 T1 |     1629.84|     1262.63|
|RND4K Q32T16|      954.86|     1078.88|
|. IOPS      |   233119.53|   263398.08|
|. latency us|     2194.84|     1941.78|
|RND4K Q1 T1 |       55.56|      225.06|
|. IOPS      |    13565.49|    54946.21|
|. latency us|       72.76|       16.97|

2. Single osd, single server benchmark (Test sequence is 3)

Modify ceph.conf set osd_pool_default_min_size and osd_pool_default_size with 1, then systemctl restart ceph.target and fix all errors

step.

root@pve01:/mnt/test# ceph osd pool get rbd size
size: 2
root@pve01:/mnt/test# ceph config set global  mon_allow_pool_size_one true
root@pve01:/mnt/test# ceph osd pool set rbd min_size 1
set pool 2 min_size to 1
root@pve01:/mnt/test# ceph osd pool set rbd size 1 --yes-i-really-mean-it
set pool 2 size to 1

result

root@pve01:/mnt/test# ceph -s
  cluster:
    id:     1f3eacc8-2488-4e1a-94bf-7181ee7db522
    health: HEALTH_WARN
            2 pool(s) have no replicas configured
 
  services:
    mon: 3 daemons, quorum pve01,pve02,pve03 (age 17m)
    mgr: pve01(active, since 17m), standbys: pve02, pve03
    osd: 6 osds: 1 up (since 19s), 1 in (since 96s)
 
  data:
    pools:   2 pools, 33 pgs
    objects: 281 objects, 1.0 GiB
    usage:   1.1 GiB used, 119 GiB / 120 GiB avail
    pgs:     33 active+clean
 
root@pve01:/mnt/test# ceph osd tree
ID  CLASS  WEIGHT   TYPE NAME       STATUS  REWEIGHT  PRI-AFF
-1         0.70312  root default                             
-3         0.23438      host pve01                           
 0    ssd  0.11719          osd.0       up   1.00000  1.00000
 1    ssd  0.11719          osd.1     down         0  1.00000
-5         0.23438      host pve02                           
 2    ssd  0.11719          osd.2     down         0  1.00000
 3    ssd  0.11719          osd.3     down         0  1.00000
-7         0.23438      host pve03                           
 4    ssd  0.11719          osd.4     down         0  1.00000
 5    ssd  0.11719          osd.5     down         0  1.00000
root@pve01:/mnt/test# python3 ~/fio-cdm/fio-cdm
tests: 5, size: 1.0GiB, target: /mnt/test 175.8MiB/20.0GiB
|Name        |  Read(MB/s)| Write(MB/s)|
|------------|------------|------------|
|SEQ1M Q8 T1 |     1153.07|      515.29|
|SEQ1M Q1 T1 |      447.35|      142.98|
|RND4K Q32T16|       99.07|       32.19|
|. IOPS      |    24186.26|     7859.91|
|. latency us|    21148.94|    65076.23|
|RND4K Q1 T1 |        7.47|        1.48|
|. IOPS      |     1823.24|      360.98|
|. latency us|      545.98|     2765.23|
root@pve01:/mnt/test# 

3. Multiple OSDs, single server benchmarks (Test sequence is 2)

Change crushmap set step chooseleaf firstn 0 type host to step chooseleaf firstn 0 type osd

OSD tree

root@pve01:/etc/ceph# ceph osd tree
ID  CLASS  WEIGHT   TYPE NAME       STATUS  REWEIGHT  PRI-AFF
-1         0.70312  root default                             
-3         0.23438      host pve01                           
 0    ssd  0.11719          osd.0       up   1.00000  1.00000
 1    ssd  0.11719          osd.1       up   1.00000  1.00000
-5         0.23438      host pve02                           
 2    ssd  0.11719          osd.2     down         0  1.00000
 3    ssd  0.11719          osd.3     down         0  1.00000
-7         0.23438      host pve03                           
 4    ssd  0.11719          osd.4     down         0  1.00000
 5    ssd  0.11719          osd.5     down         0  1.00000

result

root@pve01:/mnt/test# python3 ~/fio-cdm/fio-cdm
tests: 5, size: 1.0GiB, target: /mnt/test 175.8MiB/20.0GiB
|Name        |  Read(MB/s)| Write(MB/s)|
|------------|------------|------------|
|SEQ1M Q8 T1 |     1376.59|      397.29|
|SEQ1M Q1 T1 |      442.74|      111.41|
|RND4K Q32T16|      114.97|       29.08|
|. IOPS      |    28068.12|     7099.90|
|. latency us|    18219.04|    72038.06|
|RND4K Q1 T1 |        6.82|        1.04|
|. IOPS      |     1665.27|      254.40|
|. latency us|      598.00|     3926.30|

4. Multiple OSDs, multiple server benchmarks (Test sequence is 1)

OSD tree

root@pve01:/etc/ceph# ceph osd tree
ID  CLASS  WEIGHT   TYPE NAME       STATUS  REWEIGHT  PRI-AFF
-1         0.70312  root default                             
-3         0.23438      host pve01                           
 0    ssd  0.11719          osd.0       up   1.00000  1.00000
 1    ssd  0.11719          osd.1       up   1.00000  1.00000
-5         0.23438      host pve02                           
 2    ssd  0.11719          osd.2       up   1.00000  1.00000
 3    ssd  0.11719          osd.3       up   1.00000  1.00000
-7         0.23438      host pve03                           
 4    ssd  0.11719          osd.4       up   1.00000  1.00000
 5    ssd  0.11719          osd.5       up   1.00000  1.00000

result

tests: 5, size: 1.0GiB, target: /mnt/test 175.8MiB/20.0GiB
|Name        |  Read(MB/s)| Write(MB/s)|
|------------|------------|------------|
|SEQ1M Q8 T1 |     1527.37|      296.25|
|SEQ1M Q1 T1 |      408.86|      106.43|
|RND4K Q32T16|      189.20|       43.00|
|. IOPS      |    46191.94|    10499.01|
|. latency us|    11068.93|    48709.85|
|RND4K Q1 T1 |        4.99|        0.95|
|. IOPS      |     1219.16|      232.37|
|. latency us|      817.51|     4299.14|

Conclusions

  1. It can be seen that the gap between the write performance of ceph (106.43MB/s) and the write performance of physical disk (1262.63MB/s) is huge, and even the RND4K Q1 T1 directly becomes a mechanical hard disk
  2. One or more OSDs and one or more machines have little impact on ceph (it may be that my number of clusters is not enough)
  3. The ceph cluster built with three nodes will cause the disk read performance to drop by half and the write performance to drop by a quarter or more

APPENDIX

1 - Some ssd benchmark results

Micron_1 100_MTFDDAK1T0TB SCSI Disk Device
G:\fio>python "E:\Programing\PycharmProjects\fio-cdm\fio-cdm"
tests: 5, size: 1.0GiB, target: G:\fio 228.2GiB/953.8GiB
|Name        |  Read(MB/s)| Write(MB/s)|
|------------|------------|------------|
|SEQ1M Q8 T1 |      363.45|      453.54|
|SEQ1M Q1 T1 |      329.47|      404.09|
|RND4K Q32T16|      196.16|      212.42|
|. IOPS      |    47890.44|    51861.48|
|. latency us|    10677.71|     9862.74|
|RND4K Q1 T1 |       20.66|       65.44|
|. IOPS      |     5044.79|    15976.40|
|. latency us|      197.04|       61.07|
SAMSUNG MZALQ512HALU-000L1
root@pve1:/mnt/test# python3 ~/fio-cdm/fio-cdm
tests: 5, size: 1.0GiB, target: /mnt/test 3.4GiB/476.7GiB
|Name        |  Read(MB/s)| Write(MB/s)|
|------------|------------|------------|
|SEQ1M Q8 T1 |     2358.84|     1476.54|
|SEQ1M Q1 T1 |     1702.19|     1291.18|
|RND4K Q32T16|      955.34|     1070.17|
|. IOPS      |   233238.46|   261273.09|
|. latency us|     2193.90|     1957.79|
|RND4K Q1 T1 |       55.04|      229.99|
|. IOPS      |    13437.11|    56149.97|
|. latency us|       73.17|       16.65|

2 - bcache

Test results of hdd+ssd mixed disk ceph architecture accelerated by bcache We can see that READ has improved significantly, but WRITE is still very poor

tests: 5, size: 1.0GiB, target: /mnt/test 104.3MiB/10.0GiB
|Name        |  Read(MB/s)| Write(MB/s)|
|------------|------------|------------|
|SEQ1M Q8 T1 |     1652.93|      242.41|
|SEQ1M Q1 T1 |      552.91|       81.16|
|RND4K Q32T16|      429.52|       31.95|
|. IOPS      |   104862.76|     7799.72|
|. latency us|     4879.87|    65618.50|
|RND4K Q1 T1 |       13.10|        0.45|
|. IOPS      |     3198.16|      110.09|
|. latency us|      310.07|     9077.11|

Even multiple osds on one disk cannot solve the WRITE problem

Detailed test data: https://www.reddit.com/r/ceph/comments/xnse2j/comment/j6qs57g/?context=3

If I use VMware vSAN, I can easily accelerate the speed of hdd to ssd, and I can hardly perceive the existence of hdd (I haven't compared it in detail, I just by feeling)

3 - Test report analysis of other disciplines

I analyzed and compared several reports, and the summary is as follows

Proxmox-VE_Ceph-Benchmark-201802.pdf

Proxmox-VE_Ceph-Benchmark-202009-rev2.pdf

Dell_R730xd_RedHat_Ceph_Performance_SizingGuide_WhitePaper.pdf

micron_9300_and_red_hat_ceph_reference_architecture.pdf

1) - pve 201802

According to the report, the test scale is 6 x Server,Each server 4 x Samsung SM863 Series, 2.5", 240 GB SSD, SATA-3 (6 Gb/s) MLC.

# Samsung SM863 Series, 2.5", 240 GB SSD
# from https://www.samsung.com/us/business/support/owners/product/sm863-series-240gb/
|Name        |  Read(MB/s)| Write(MB/s)|
|------------|------------|------------|
|SEQ?M Q? T? |      520.00|      485.00|
|RND4K Q? T? |           ?|           ?|
|. IOPS      |    97000.00|    20000.00|

Report result display

# 3 Node Cluster/ 4 x Samsung SM863 as OSD per Node
# rados bench 60 write -b 4M -t 16
# rados bench 60 read -t 16 (uses 4M from write)
|Name        |  Read(MB/s)| Write(MB/s)|
# 10 Gbit Network
|------------|------------|------------|
|SEQ4M Q? T16|     1064.42|      789.12|
# 100 Gbit Network
|------------|------------|------------|
|SEQ4M Q? T16|     3087.82|     1011.63|

We can see that the impact of network bandwidth on performance is huge. Although the performance under the 10 Gbit network is insufficient, at least the read and write performance approaches the bandwidth limit However, looking at my test results, WRITE is very bad (296.25MB/s)

2) - pve 202009

According to the report, the test scale is 3 x Server; Each server 4 x Micron 9300 Max 3.2 TB (MTFDHAL3T2TDR); 1 x 100 GbE DACs, in a full-mesh topology

# Micron 9300 Max 3.2 TB (MTFDHAL3T2TDR)
|Name        |  Read(MB/s)| Write(MB/s)|
|------------|------------|------------| 
|SEQ128KQ32T?|     3500.00|     3100.00| (MTFDHAL12T8TDR-1AT1ZABYY-Micron-LBGA-2022.pdf)
|RND4K Q512T?|     3340.00|      840.00| (Estimate according to formula, throughput ~= iops * 4k / 1000)
|. IOPS      |   835000.00|   210000.00| (MTFDHAL12T8TDR-1AT1ZABYY-Micron-LBGA-2022.pdf)
|------------|------------|------------| 
|RND4K Q1 T1 |            |      205.82| (from the report)
|. IOPS      |            |    51000.00| (from the report)
|. latency ms|            |        0.02| (from the report)

Report result display

# MULTI-VM WORKLOAD (LINUX)
# I don't understand the difference between Thread and Job, and the queue depth is not identified in the document
|Name        |  Read(MB/s)| Write(MB/s)|
|------------|------------|------------|
|SEQ4M Q? T1 |     7176.00|     2581.00| (SEQUENTIAL BANDWIDTH BY NUMBER OF JOBS)
|RND4K Q1 T1 |       86.00|       28.99| (Estimate according to formula)
|. IOPS      |    21502.00|     7248.00| (RANDOM IO/S BY NUMBER OF JOBS)

Similarly, the RND4K Q1 T1 WRITE test result is very bad, only 7k iops, and the physical disk has 51k iops, which I feel is unacceptable.

3) - Dell R730xd report

According to the report, the test scale is 5 x Storage Server; Each Server 12HDD+3SSD, 3 x replication 2 x 10GbE NIC

# Test results extracted from the report
# Figure 8  Throughput/server comparison by using different configurations
|Name        |  Read(MB/s)| Write(MB/s)|
|------------|------------|------------|
|SEQ4M Q64T1 |     1150.00|      300.00|

In this case, the WRITE in the SEQ4M Q64T1 test result is only about 300MB/s, which is about twice that of a single SAS, that is, 2 x 158.16 MB/s (4M blocks). This makes me unbelievable. It's even faster than my nvme disk. However, another important fact is that 12 * 5=60 HDDs have only 300MB/s sequential write speed. Is this performance loss too large?

4) - Micron report

According to the report, the test scale is 3 x Storage Server;Each Server 10 x micron 9300MAX 12.8T,2 x replication ,2 x 100GbE NIC

# micron 9300MAX 12.8T (MTFDHAL12T8TDR-1AT1ZABYY) Physical disk benchmark 
|Name        |  Read(MB/s)| Write(MB/s)| (? is the parameter not given)
|------------|------------|------------|
|SEQ?M Q? T? |    48360.00|           ?| (from the report)
|SEQ128KQ32T?|     3500.00|     3500.00| (MTFDHAL12T8TDR-1AT1ZABYY-Micron-LBGA-2022.pdf)
|RND4K Q512T?|     3400.00|     1240.00| (Estimate according to formula)
|. IOPS      |   850000.00|   310000.00| (MTFDHAL12T8TDR-1AT1ZABYY-Micron-LBGA-2022.pdf)
|. latency us|       86.00|       11.00| (MTFDHAL12T8TDR-1AT1ZABYY-Micron-LBGA-2022.pdf)
|------------|------------|------------|
|RND4K Q? T? |     8397.77|     1908.11| (Estimate according to formula)
|. IOPS      |  2099444.00|   477029.00| (from the report, Executive Summary)
|. latency ms|        1.50|        6.70| (from the report, Executive Summary)

Report result display

# (Test results extracted from the report)
|Name        |  Read(MB/s)| Write(MB/s)|
|------------|------------|------------|
|RND4KQ32T100|           ?|           ?|
|. IOPS      |  2099444.00|   477029.00|
|. latency ms|        1.52|        6.71|
# (I don't know if there is a problem reported on the official website. There is no performance loss here)

It has to be said that the official test platform of Micron is too high-end for our small and medium-sized enterprises to afford. From the results, WRITE is close to the performance of a single physical disk. Does it mean that if only a single node and a single disk are used, the WRITE performance will drop to 477k/30=15.9k iops If so, this will be the performance of sata ssd.

4 - More of the same questions

  1. https://forum.proxmox.com/threads/bad-rand-read-write-i-o-proxmox-ceph.68404/#post-530006
  2. https://forum.proxmox.com/threads/ceph-performance-with-simple-hardware-slow-writing.96697/#post-529524
  3. https://forum.proxmox.com/threads/bad-rand-read-write-i-o-proxmox-ceph.68404/#post-529520
  4. https://forum.proxmox.com/threads/bad-rand-read-write-i-o-proxmox-ceph.68404/#post-529486
  5. https://www.reddit.com/r/ceph/comments/xnse2j/comment/j6qobtv/?utm_source=share&utm_medium=web2x&context=3
  6. https://www.reddit.com/r/ceph/comments/kioxqx/comment/j6d3sxc/?utm_source=share&utm_medium=web2x&context=3

Finally, the question I want to know is:

  1. How to fix the write performance problem in ceph? Can ceph achieve the same performance as VMware vSAN.
  2. The results show that the performance of full flash disk is not as good as that of hdd+ssd. So if I do not use bcache, what should I do to fix the performance problem of ceph full flash disk?
  3. Is there a better solution for the hdd+ssd architecture?
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.