Score:0

(fio test) disk very slow with any service but normally if straight to disk

US flag

OS details:

PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

I had a ceph cluster based on ubuntu LTS, for proxmox storage using rbd, but vm disk speed (no matter read or write) only lower than 10MB/s, somethimes KB/s, this is background.

So I made a test env for find out why so slow, 10GB switch, 10GB netcard, 2 node, one for ceph(ubuntu 22.04, name ceph node), one for just ubuntu 22.04(name client node) 6 disk in ceph node( 3 ssd, 3 hdd), problem is I tested for ssd and hdd, almost same speed, so I tried to do everything in ceph node so I can skip the network test prograss:

  1. ceph node map(127.0.0.1) rbd from ceph node(127.0.0.1)

    sudo rbd map speedtest/aaa --id admin --keyring /etc/ceph/ceph.client.admin.keyring --mon-host 127.0.0.1

  2. fio to test, 2226KiB/s for write speed

    direct=1
    rw=write
    filename=/dev/rbd0

    [job]
    name=test
    bs=4k
    iodepth=16
    size=200G
    runtime=60
    numjobs=1
    time_based
    rwmixread=70
  1. than I remove all osd, direct to test disk /dev/sdg, 203MB/s for write speed
    [global]
    direct=1
    rw=write
    filename=/dev/sdg

    [job]
    name=test
    bs=4k
    iodepth=16
    size=200G
    runtime=60
    numjobs=1
    time_based
    rwmixread=70
  1. I guess what if ceph configuration has something wrong, so I create a nfs server for localhost, write speed but still slowwwwwwwwwww, 1028kB/s

    my /etc/exportfs:

        /media/speedtest *(rw,sync,all_squash,anonuid=0)
    

    command:

         sudo mount.nfs localhost:/media/speedtest /mnt/
    

    mount disk:

         /dev/sdd1 on /media type ext4 (rw,relatime)
    

    fio config same as above

  2. I tested for dir /media, write speed 199MB/s

enter image description here enter image description here

I almost gonna be crazy, how is this happend ? somebody help me please, if you need something else information, tell me, I will show you everything if you need. help me, please.

I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.