I'm profiling a data-processing program that I wrote and I want to understand the theoretical throughput of the machine I'm on. Specifically, I want to measure what disk read speed my g4dn EC2 instance is providing my application. This particular instance has an ephemeral NVMe drive, which is what I want to benchmark.
I notice that it takes quite a few runs of hdparm until the throughput it reports stops increasing. My question is, why does hdparm take multiple runs to get full read throughput? What is it in the Linux kernel / disk driver / drive controller / actual hardware that requires multiple runs of hdparm to get accurate results?
I know the man pages say to run it a few times, but in my experience it takes much more than the 3 times recommended for the throughput to max out.
-t Perform timings of device reads for benchmark and
comparison purposes. For meaningful results, this
operation should be repeated 2-3 times on an otherwise
inactive system (no other active processes) with at least
a couple of megabytes of free memory. This displays the
speed of reading through the buffer cache to the disk
without any prior caching of data. This measurement is an
indication of how fast the drive can sustain sequential
data reads under Linux, without any filesystem overhead.
To ensure accurate measurements, the buffer cache is
flushed during the processing of -t using the BLKFLSBUF
ioctl.
I run the following to collect the read speed:
#!/usr/bin/env bash
while true; do
sudo hdparm -t /dev/nvme0n1p1;
sleep 1;
done
and get the following output:
$ while true; do sudo hdparm -t /dev/nvme0n1p1; sleep 1; done
/dev/nvme0n1p1:
Timing buffered disk reads: 470 MB in 3.09 seconds = 152.30 MB/sec
/dev/nvme0n1p1:
Timing buffered disk reads: 490 MB in 3.10 seconds = 158.21 MB/sec
/dev/nvme0n1p1:
Timing buffered disk reads: 526 MB in 3.02 seconds = 174.43 MB/sec
It takes maybe 20 runs before it settles around 330MB/sec.
Note that I am using an AMI that has the correct NVMe drivers.
