Score:0

I have a new server but IO Speed is terrible based on FIO results

tc flag

I ordered a new server because I have to move to higher performance server. However, this new server is totally awful. I tried to get the benchmark by using fio here is the result.

[root@s206716 ~]# fio --filename=/var/lib/mysql/fio.tmp --size=1m --bs=512 --name=go --rw=write --fdatasync=1
go: (g=0): rw=write, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=psync, iodepth=1
fio-3.19
Starting 1 process
^Cbs: 1 (f=1): [W(1)][39.3%][w=3KiB/s][w=7 IOPS][eta 01m:08s]   
fio: terminating on signal 2

go: (groupid=0, jobs=1): err= 0: pid=108048: Thu Apr 20 18:32:15 2023
  write: IOPS=18, BW=9401B/s (9401B/s)(397KiB/43242msec); 0 zone resets
    clat (usec): min=5, max=126657, avg=951.49, stdev=9446.81
     lat (usec): min=5, max=126658, avg=951.94, stdev=9446.80
    clat percentiles (usec):
     |  1.00th=[     6],  5.00th=[     6], 10.00th=[     9], 20.00th=[    10],
     | 30.00th=[    13], 40.00th=[    14], 50.00th=[    20], 60.00th=[    22],
     | 70.00th=[    23], 80.00th=[    25], 90.00th=[   157], 95.00th=[   297],
     | 99.00th=[  8094], 99.50th=[102237], 99.90th=[126354], 99.95th=[126354],
     | 99.99th=[126354]
   bw (  KiB/s): min=    2, max=  118, per=100.00%, avg= 9.21, stdev=20.45, samples=86
   iops        : min=    4, max=  236, avg=18.43, stdev=40.89, samples=86
  lat (usec)   : 10=24.43%, 20=30.23%, 50=32.75%, 250=6.55%, 500=3.02%
  lat (usec)   : 750=0.63%, 1000=0.76%
  lat (msec)   : 2=0.25%, 4=0.13%, 10=0.38%, 100=0.38%, 250=0.50%
  fsync/fdatasync/sync_file_range:
    sync (msec): min=2, max=233, avg=53.50, stdev=65.67
    sync percentiles (msec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    5], 40.00th=[    5], 50.00th=[    5], 60.00th=[   68],
     | 70.00th=[   97], 80.00th=[  107], 90.00th=[  132], 95.00th=[  199],
     | 99.00th=[  226], 99.50th=[  228], 99.90th=[  234], 99.95th=[  234],
     | 99.99th=[  234]
  cpu          : usr=0.02%, sys=0.09%, ctx=2445, majf=0, minf=17
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,794,0,0 short=794,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=9401B/s (9401B/s), 9401B/s-9401B/s (9401B/s-9401B/s), io=397KiB (407kB), run=43242-43242msec

Disk stats (read/write):
    dm-0: ios=104/2979, merge=0/0, ticks=971/141634, in_queue=142605, util=96.36%, aggrios=104/2973, aggrmerge=0/22, aggrticks=974/143011, aggrin_queue=143985, aggrutil=95.40%
  sda: ios=104/2973, merge=0/22, ticks=974/143011, in_queue=143985, util=95.40%

I have no idea why the speed process is really slow. This one is not a production server. I have just importing all the files and testing out. So, there is no traffic yet. I have asked the provider to do hardware testing and they say that there is no issue.

What should I do with this thing? I have done many things. Such as trying to move to another server I have (low spec server) and it has no issue. However, using this high performance server is even lowest than others.

John Mahowald avatar
cn flag
Please edit your question to add: OS distro version (CentOS?), which file system type, the rated performance of the block device (SSD model, storage array model, promised IOPS). And also add other fio option combinations to find significant differences via scientific method. --fdatasync is brutal to file systems, for example.
Wilson Hauck avatar
jp flag
Additional DB information request from new server, please. OS, Version? RAM size, # cores, any SSD or NVME devices on MySQL Host server? Post TEXT data on justpaste.it and share the links. From your SSH login root, Text results of: A) SELECT COUNT(*), sum(data_length), sum(index_length), sum(data_free) FROM information_schema.tables; B) SHOW GLOBAL STATUS; after TESTING and 4 hours UPTIME C) SHOW GLOBAL VARIABLES; D) SHOW FULL PROCESSLIST; E) STATUS; not SHOW STATUS, just STATUS; G) SHOW ENGINE INNODB STATUS; for server workload tuning analysis to provide suggestions.
rvs avatar
vn flag
rvs
Let `fio` run for longer. Add more parallelism - try `--numjobs=16`, `--iodepth=64`, increase data size (such as `--size=10G`). Do you still get same results? It is possible server is using some fancy network-attached storage which is quick but may have higher latency, so `fdatasync=1` is what's killing test results. Real databases don't behave quite this way - they try to do at least some batching.
rvs avatar
vn flag
rvs
Oh, and another thing - `--bs=512` will not be representative on modern software & hardware. Try at least 4k. With 512 byte writes you are guaranteed each of them becomes read-modify-write as SSDs use much larger block sizes.
Wilson Hauck avatar
jp flag
@James-Hartford additional consideration for rvs May 2 comment, MySQL basic unit of work is always 16384 - so use bs=16384 to be paired with MySQL typical operations.
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.