Score:0

Server

Unexpected ZFS async write performance at different queue depths

Sam

2/18/24, 8:15 AM

For some reason, QD2 seems to be the best in terms of raw IOPS...

After seeing an interesting article comparing a couple different drives for SLOG usage, I got curious about the performance of my own system at different queue depths.

The system comprises an i9-13900K, 128GB DDR5-4800 system mem and two Samsung 980 PROs as single mirrored Vdev. The test is running FIO in the following config in a container on ProxMox 7:

fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=<4k/8k> --numjobs=1 --size=8g --iodepth=<1/2/4/8> --runtime=30 --time_based

Something odd I noticed was that during async writes, the best performance would consistently occur at queue depths of 2. I tried several different compression, block size and record size combinations but QD2 consistently had the best IOPS scores... I would have expected it to get better at higher queue depths or am I missing something? (I wonder if it's related to the drives having only two NAND chips per drive?)

No specific ZFS tuning has been performed other than setting ashift=12 and what's listed above.

Does anyone know why ZFS async writes seem to work best at QD2 compared to QD1, QD4 or QD8?

And now... Graphs! These are the averaged results of the 216 different combinations I tried.

(why a record size of 16k was the worst is beyond me) (A block size of 8k still results in more overall bandwidth)

Bit surprised to see Zstd beating LZ4. Perhaps I'm reaching the IO limits of these poor SSDs? The actual SSD activity was usually hitting 400-700MB/s during testing.

N.B. During all tests the CPU governor was set to "Performance", when set to "Powersave" the IOPS figures were 30-50% worse!

0 + 10

zfsonlinux

ewwhite

2/18/24, 9:39 AM

What problem do you want to solve?

Sam

2/18/24, 9:51 AM

@ewwhite I'm trying to work out why QD2 workloads seem to perform better than any other on this system. Many storage performance plots seem to get better IOPS as queue depth increases (except here). It just seems rather odd and I'm curious to understand more about what the underlying cause is.

ewwhite

2/18/24, 1:05 PM

Do you have a realistic workload that requires QD2?

John Mahowald

2/18/24, 1:54 PM

And what is that workload software? Linux async I/O APIs are a mess, there are multiple to choose from (libaio, POSIX AIO, io_uring), each with their own quirks, and with very different performance characteristics. Or the software might not attempt async I/O at all.

shodanshok

2/18/24, 5:51 PM

Can you try re-running your test using `numjobs=1/2/4/8` rather than `posixaio` (application-level async API historically was somewhat problematic under Linux, especially for ZFS)?

Sam

2/19/24, 10:01 PM

@ewwhite No, I don't have a specifically QD2 workload, I'm just trying to work out why the performance gets worse at increasing queue depth. I was expecting that increasing queue depth would have *improved* performance rather than tank it...

Sam

2/19/24, 10:03 PM

@JohnMahowald I'm running benchmarks using [the FlexibleI/O Tester](https://linux.die.net/man/1/fio) - not sure if that's the best general iops tester, but it's what I see commonly used. I'm happy to change if there's a better tool.

Sam

2/19/24, 10:05 PM

@shodanshok I'll run some more benchmarks when I get back from work later today

ewwhite

2/20/24, 1:44 AM

I understand the curiosity... but at the same time, this isn't a production issue or something to worry about outside of the synthetic benchmarks you're running.

John Mahowald

2/20/24, 2:33 PM

fio is very good, but its not your workload, its a synthetic tester, and can do storage things either smart or dumb that might not be what you see in production. What actual database or whatever are you running? What capacity planning have you done where you care if you get 50k versus 60k IOPS? Please edit your question with the details.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Unexpected ZFS async write performance at different queue depths

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.