Score:0

FIO 17x slower on ZFS than raw device

in flag

I have ZFS pool RAIDZ-1 on 4 x 3.84TB SATA3 'enterprise' SSDs.

zfs create rpool/fio
zfs set primarycache=none rpool/fio

fio --ioengine=sync --filename=[X] --direct=1 --rw=read --bs=4K --numjobs=1 --iodepth=1 --runtime=10 --time_based --name=fio

Results:

--filename=/dev/sdg:        
  read: IOPS=25.1k, BW=98.2MiB/s (103MB/s)

--filename=/rpool/fio/testx
  read: IOPS=1408, BW=5634KiB/s (5769kB/s)

zpool configuration:

NAME       PROPERTY              VALUE                  SOURCE
rpool/fio  type                  filesystem             -
rpool/fio  creation              Thu Nov 18 11:04 2021  -
rpool/fio  used                  30.5G                  -
rpool/fio  available             9.69T                  -
rpool/fio  referenced            30.5G                  -
rpool/fio  compressratio         1.05x                  -
rpool/fio  mounted               yes                    -
rpool/fio  quota                 none                   default
rpool/fio  reservation           none                   default
rpool/fio  recordsize            128K                   default
rpool/fio  mountpoint            /rpool/fio             default
rpool/fio  sharenfs              off                    default
rpool/fio  checksum              on                     default
rpool/fio  compression           lz4                    local
rpool/fio  atime                 off                    inherited from rpool
rpool/fio  devices               on                     default
rpool/fio  exec                  on                     default
rpool/fio  setuid                on                     default
rpool/fio  readonly              off                    default
rpool/fio  zoned                 off                    default
rpool/fio  snapdir               hidden                 default
rpool/fio  aclmode               discard                default
rpool/fio  aclinherit            restricted             default
rpool/fio  createtxg             11836                  -
rpool/fio  canmount              on                     default
rpool/fio  xattr                 on                     default
rpool/fio  copies                1                      default
rpool/fio  version               5                      -
rpool/fio  utf8only              off                    -
rpool/fio  normalization         none                   -
rpool/fio  casesensitivity       sensitive              -
rpool/fio  vscan                 off                    default
rpool/fio  nbmand                off                    default
rpool/fio  sharesmb              off                    default
rpool/fio  refquota              none                   default
rpool/fio  refreservation        none                   default
rpool/fio  guid                  11473612800347485365   -
rpool/fio  primarycache          none                   local
rpool/fio  secondarycache        all                    default
rpool/fio  usedbysnapshots       0B                     -
rpool/fio  usedbydataset         30.5G                  -
rpool/fio  usedbychildren        0B                     -
rpool/fio  usedbyrefreservation  0B                     -
rpool/fio  logbias               latency                default
rpool/fio  objsetid              9843                   -
rpool/fio  dedup                 off                    default
rpool/fio  mlslabel              none                   default
rpool/fio  sync                  standard               inherited from rpool
rpool/fio  dnodesize             legacy                 default
rpool/fio  refcompressratio      1.05x                  -
rpool/fio  written               30.5G                  -
rpool/fio  logicalused           32.0G                  -
rpool/fio  logicalreferenced     32.0G                  -
rpool/fio  volmode               default                default
rpool/fio  filesystem_limit      none                   default
rpool/fio  snapshot_limit        none                   default
rpool/fio  filesystem_count      none                   default
rpool/fio  snapshot_count        none                   default
rpool/fio  snapdev               hidden                 default
rpool/fio  acltype               off                    default
rpool/fio  context               none                   default
rpool/fio  fscontext             none                   default
rpool/fio  defcontext            none                   default
rpool/fio  rootcontext           none                   default
rpool/fio  relatime              off                    default
rpool/fio  redundant_metadata    all                    default
rpool/fio  overlay               on                     default
rpool/fio  encryption            off                    default
rpool/fio  keylocation           none                   default
rpool/fio  keyformat             none                   default
rpool/fio  pbkdf2iters           0                      default
rpool/fio  special_small_blocks  0                      default

Is there something wrong with the test or the ZFS? Any advice/insights appreciated! Thanks.

UPDATE 1:

I've update the initial test to --bs=4K and the story broadly stays the same... Is testing on the raw device valid?

UPDATE 2:

It seems I get a different picture from zpool iostat to fio output. I've adjust the fio test to randion r/w with 64 jobs, which should reveal SSD benchmark more 'realistically'. NB the system has no other workload.

On SSD:

fio --ioengine=libaio --filename=/rpool/fio/testx --size=4G --time_based --name=fio --group_reporting --runtime=10 --direct=1 --sync=1 --iodepth=1 --rw=randrw  --bs=4K --numjobs=64

read: IOPS=4022, BW=15.7MiB/s (16.5MB/s)
write: IOPS=4042, BW=15.8MiB/s (16.6MB/s)


# zpool iostat -vy rpool 5 1
                                                        capacity     operations     bandwidth 
pool                                                  alloc   free   read  write   read  write
---------------------------------------------  -----  -----  -----  -----  -----  -----
rpool                                                  216G  27.7T  28.1K  14.5K  1.17G   706M
  raidz1                                               195G  13.8T  13.9K  7.26K   595M   358M
    ata-INTEL_SSDSC2KB038TZ_BTYI13730BAV3P8EGN-part3      -      -  3.60K  1.73K   159M  90.3M
    ata-INTEL_SSDSC2KB038TZ_BTYI13730B9Q3P8EGN-part3      -      -  3.65K  1.82K   150M  89.0M
    ata-INTEL_SSDSC2KB038TZ_BTYI13730B9G3P8EGN-part3      -      -  3.35K  1.83K   147M  90.0M
    ata-INTEL_SSDSC2KB038TZ_BTYI13730BAT3P8EGN-part3      -      -  3.34K  1.89K   139M  88.4M
  raidz1                                              21.3G  13.9T  14.2K  7.21K   604M   348M
    sde                                                   -      -  3.39K  1.81K   149M  87.5M
    sdf                                                   -      -  3.35K  1.90K   139M  86.3M
    sdg                                                   -      -  3.71K  1.70K   163M  87.8M
    sdh                                                   -      -  3.69K  1.81K   152M  86.4M
---------------------------------------------  -----  -----  -----  -----  -----  -----

On HDD:

read: IOPS=1382, BW=5531KiB/s
write: IOPS=1385, BW=5542KiB/s

$ zpool iostat -vy rpool 5 1
                                    capacity     operations     bandwidth 
pool                              alloc   free   read  write   read  write
--------------------------------  -----  -----  -----  -----  -----  -----
rpool                              160G  18.0T  3.07K  2.71K   393M   228M
  mirror                          32.2G  3.59T    624    589  78.0M  40.2M
    scsi-35000c500de5c67f7-part3      -      -    321    295  40.1M  20.4M
    scsi-35000c500de75a863-part3      -      -    303    293  37.9M  19.7M
  mirror                          31.9G  3.59T    625    551  78.2M  49.9M
    scsi-35000c500de2bd6bb-part3      -      -    313    274  39.1M  24.2M
    scsi-35000c500de5ae5a7-part3      -      -    312    277  39.0M  25.7M
  mirror                          32.2G  3.59T    648    548  81.1M  45.9M
    scsi-35000c500de5ae667-part3      -      -    320    279  40.1M  23.0M
    scsi-35000c500de2bd2d3-part3      -      -    328    268  41.0M  23.0M
  mirror                          31.6G  3.59T    612    536  76.5M  45.5M
    scsi-35000c500de5ef20f-part3      -      -    301    266  37.7M  22.7M
    scsi-35000c500de5edbfb-part3      -      -    310    269  38.9M  22.8M
  mirror                          32.0G  3.59T    629    555  78.7M  46.5M
    scsi-35000c500de5c6f7f-part3      -      -    318    283  39.8M  23.1M
    scsi-35000c500de5c6c5f-part3      -      -    311    272  38.9M  23.4M
--------------------------------  -----  -----  -----  -----  -----  -----

Per device performace looks fine on the HDDs, and the SSDs look to be roughtly 5x faster than HDDs in random r/w. Is that to be expected? I was expecting 10-100x faster...

Thanks for your attention!

Nikita Kipriyanov avatar
za flag
Are you sure you want to use `sync` I/O engine?
shodanshok avatar
ca flag
Please recreate your pool with `-o ashift=12`. Also, please try with `recordsize=16K`
simonw avatar
in flag
Thanks @shodanshok ashift is 12 and setting recordsize to 16k yields the same results.
simonw avatar
in flag
Thanks @NikitaKipriyanov same results with libaio.
Romeo Ninov avatar
in flag
You can try to remove compression of the pool
simonw avatar
in flag
Thanks @RomeoNinov I've tried that, same results!...
batistuta09 avatar
vn flag
21k IOPS on 128k blocks is not a valid result for single SATA SSD. It means approx. 2.6GB/s of throughput, what 5 times more as single SATA port delivers.
us flag
You should not use a single-parity RAID if you want to keep your data safe.
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.