It appears that work is proceeding on parallelization of disk read/write operations for ZFS, but the work is not ready for testing.
Parameters and a bit of math to guide the responses:
Capacity per drive: 16,000,000,000,000 Bytes (not 16TB).
Sustained read/write: 270MB/s (258 MiB/s).
Mean time between failure: 285 years.
Nonrecoverable sector read errors per bit read: 1 bit error per 116,415 TB of data read.
Random Read 4K QD16 QCD: 170 IOPS.
Random Write 4K QD16 QCD: 550 IOPS.
Each 8-drive RAIDZ1 vdev is connected to an 8-channel PCIe 3.0x HBA that supports 512MB/sec sustained throughput per attached drive.
The HBA is attached to a PCI4.0 x16 slot on a 128 lane motherboard.
Running in parallel, the system supports a complete read of all 24 16TB drives in 22 hours.
My expectation is that the scrub should complete in less than 24 hours; therefore, the bottleneck is the CPU utilization for checksum verification. Given the availability of 5 computational threads/drive (this is a 128 thread/24 drive system), parallelization of checksums should solve the bottleneck problem.
Per reliability:
Stochastic theory predicts that drive failure is unlikely, given the manufacturer's MTBF of 285 years and assuming a confidence interval of six standard deviations. Nonetheless, I have 4 drives committed to error correction and disaster recovery.
Bit rot (nonrecoverable sector read errors per bit read) is a separate concern, which is why I am concerned about scrub operations. The expected error rate is 1 bit error per 116,415 TB of data read. That suggests one bit read error every 14 years IFF continuous reads at full throughput of 270MB/s are maintained 24x7 for 14 years.
This machine is part of a hot-failover 1024-node, 1 petabyte cluster.