OS details:
PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
I had a ceph cluster based on ubuntu LTS, for proxmox storage using rbd, but vm disk speed (no matter read or write) only lower than 10MB/s, somethimes KB/s, this is background.
So I made a test env for find out why so slow, 10GB switch, 10GB netcard, 2 node, one for ceph(ubuntu 22.04, name ceph node), one for just ubuntu 22.04(name client node)
6 disk in ceph node( 3 ssd, 3 hdd), problem is I tested for ssd and hdd, almost same speed, so I tried to do everything in ceph node so I can skip the network
test prograss:
ceph node map(127.0.0.1) rbd from ceph node(127.0.0.1)
sudo rbd map speedtest/aaa --id admin --keyring /etc/ceph/ceph.client.admin.keyring --mon-host 127.0.0.1
fio to test, 2226KiB/s for write speed
direct=1
rw=write
filename=/dev/rbd0
[job]
name=test
bs=4k
iodepth=16
size=200G
runtime=60
numjobs=1
time_based
rwmixread=70
- than I remove all osd, direct to test disk /dev/sdg, 203MB/s for write speed
[global]
direct=1
rw=write
filename=/dev/sdg
[job]
name=test
bs=4k
iodepth=16
size=200G
runtime=60
numjobs=1
time_based
rwmixread=70
I guess what if ceph configuration has something wrong, so I create a nfs server for localhost, write speed but still slowwwwwwwwwww, 1028kB/s
my /etc/exportfs:
/media/speedtest *(rw,sync,all_squash,anonuid=0)
command:
sudo mount.nfs localhost:/media/speedtest /mnt/
mount disk:
/dev/sdd1 on /media type ext4 (rw,relatime)
fio config same as above
I tested for dir /media, write speed 199MB/s
enter image description here
enter image description here
I almost gonna be crazy, how is this happend ? somebody help me please, if you need something else information, tell me, I will show you everything if you need.
help me, please.