I am experiencing poor write performance on my Windows Server 20H2 machine (ESXI 7.02). Following symptoms are observed on a docker build: Each RUN command executes successfully, though takes ~10 hours to move to the next command (committing on disk is slow). The same is about 2 minutes on physical OS, on exact same hardware.
My hardware configurations are as follows:
8 CPUs x Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz
32 GB DDR4 2933MHz
2X Seagate Barracuda 1TB ST1000DM010
Following is the output of DiskSpd tool:
Command: diskspd.exe -t16 -o32 -b4k -r4k -w50 -d120 -Sh -D -L -c10G d:\IO.dat
Output:
CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 0.25%| 0.03%| 0.22%| 99.75%
1| 0.10%| 0.03%| 0.08%| 99.90%
2| 0.07%| 0.00%| 0.07%| 99.93%
3| 0.08%| 0.01%| 0.07%| 99.92%
4| 0.12%| 0.00%| 0.12%| 99.88%
5| 0.07%| 0.01%| 0.05%| 99.93%
6| 0.05%| 0.01%| 0.04%| 99.95%
7| 0.03%| 0.00%| 0.03%| 99.97%
8| 0.07%| 0.00%| 0.07%| 99.93%
9| 0.05%| 0.00%| 0.05%| 99.95%
10| 0.08%| 0.00%| 0.08%| 99.92%
11| 0.10%| 0.01%| 0.09%| 99.90%
12| 0.07%| 0.00%| 0.07%| 99.93%
13| 0.05%| 0.00%| 0.05%| 99.95%
14| 0.04%| 0.00%| 0.04%| 99.96%
15| 0.03%| 0.00%| 0.03%| 99.97%
-------------------------------------------
avg.| 0.08%| 0.01%| 0.07%| 99.92%
Total IO
thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | IopsStdDev | LatStdDev | file
---------------------------------------------
0 | 5939200 | 1450 | 0.05 | 12.08 | 2699.677 | 2.78 | 3329.562 | d:\IO.dat (10240MiB)
1 | 5595136 | 1366 | 0.04 | 11.38 | 2872.246 | 3.42 | 4981.979 | d:\IO.dat (10240MiB)
2 | 6320128 | 1543 | 0.05 | 12.86 | 2540.342 | 2.99 | 1886.703 | d:\IO.dat (10240MiB)
3 | 5152768 | 1258 | 0.04 | 10.48 | 3142.922 | 4.99 | 4133.990 | d:\IO.dat (10240MiB)
4 | 5963776 | 1456 | 0.05 | 12.13 | 2675.156 | 2.79 | 3822.760 | d:\IO.dat (10240MiB)
5 | 4169728 | 1018 | 0.03 | 8.48 | 3890.331 | 5.69 | 7131.087 | d:\IO.dat (10240MiB)
6 | 3563520 | 870 | 0.03 | 7.25 | 4547.513 | 5.82 | 9618.899 | d:\IO.dat (10240MiB)
7 | 3121152 | 762 | 0.02 | 6.35 | 5189.994 | 6.31 | 12210.756 | d:\IO.dat (10240MiB)
8 | 5910528 | 1443 | 0.05 | 12.02 | 2723.919 | 2.72 | 3587.751 | d:\IO.dat (10240MiB)
9 | 4665344 | 1139 | 0.04 | 9.49 | 3478.946 | 5.22 | 5428.366 | d:\IO.dat (10240MiB)
10 | 5660672 | 1382 | 0.04 | 11.52 | 2817.129 | 2.91 | 4730.283 | d:\IO.dat (10240MiB)
11 | 6021120 | 1470 | 0.05 | 12.25 | 2639.039 | 4.04 | 4190.416 | d:\IO.dat (10240MiB)
12 | 7000064 | 1709 | 0.06 | 14.24 | 2236.983 | 4.61 | 565.486 | d:\IO.dat (10240MiB)
13 | 5726208 | 1398 | 0.05 | 11.65 | 2832.165 | 4.64 | 2848.993 | d:\IO.dat (10240MiB)
14 | 3055616 | 746 | 0.02 | 6.22 | 5275.084 | 4.92 | 13587.461 | d:\IO.dat (10240MiB)
15 | 7008256 | 1711 | 0.06 | 14.26 | 2237.219 | 5.18 | 577.370 | d:\IO.dat (10240MiB)
---------------------------------------------
total: 84873216 | 20721 | 0.67 | 172.67 | 3025.400 | 23.70 | 5456.260
Read IO
thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | IopsStdDev | LatStdDev | file
---------------------------------------------
0 | 2945024 | 719 | 0.02 | 5.99 | 2582.133 | 2.13 | 3254.476 | d:\IO.dat (10240MiB)
1 | 2699264 | 659 | 0.02 | 5.49 | 2971.171 | 2.16 | 5759.023 | d:\IO.dat (10240MiB)
2 | 3072000 | 750 | 0.02 | 6.25 | 2444.497 | 2.19 | 2013.276 | d:\IO.dat (10240MiB)
3 | 2580480 | 630 | 0.02 | 5.25 | 3064.072 | 3.09 | 4403.987 | d:\IO.dat (10240MiB)
4 | 3035136 | 741 | 0.02 | 6.17 | 2527.138 | 2.31 | 3417.612 | d:\IO.dat (10240MiB)
5 | 2093056 | 511 | 0.02 | 4.26 | 3597.965 | 3.27 | 6520.668 | d:\IO.dat (10240MiB)
6 | 1712128 | 418 | 0.01 | 3.48 | 3686.802 | 3.16 | 7092.978 | d:\IO.dat (10240MiB)
7 | 1585152 | 387 | 0.01 | 3.22 | 4870.749 | 3.41 | 11525.008 | d:\IO.dat (10240MiB)
8 | 3006464 | 734 | 0.02 | 6.12 | 2410.669 | 2.33 | 1546.519 | d:\IO.dat (10240MiB)
9 | 2396160 | 585 | 0.02 | 4.87 | 3121.071 | 3.17 | 4497.869 | d:\IO.dat (10240MiB)
10 | 2809856 | 686 | 0.02 | 5.72 | 3029.377 | 2.33 | 6156.560 | d:\IO.dat (10240MiB)
11 | 3117056 | 761 | 0.02 | 6.34 | 2656.653 | 2.83 | 4849.039 | d:\IO.dat (10240MiB)
12 | 3579904 | 874 | 0.03 | 7.28 | 2163.654 | 2.65 | 596.033 | d:\IO.dat (10240MiB)
13 | 2981888 | 728 | 0.02 | 6.07 | 2842.098 | 2.74 | 3119.188 | d:\IO.dat (10240MiB)
14 | 1478656 | 361 | 0.01 | 3.01 | 4678.412 | 2.57 | 12199.503 | d:\IO.dat (10240MiB)
15 | 3493888 | 853 | 0.03 | 7.11 | 2141.992 | 3.16 | 628.005 | d:\IO.dat (10240MiB)
---------------------------------------------
total: 42586112 | 10397 | 0.34 | 86.64 | 2874.556 | 15.75 | 5107.421
Write IO
thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | IopsStdDev | LatStdDev | file
---------------------------------------------
0 | 2994176 | 731 | 0.02 | 6.09 | 2815.291 | 2.08 | 3397.833 | d:\IO.dat (10240MiB)
1 | 2895872 | 707 | 0.02 | 5.89 | 2780.037 | 2.49 | 4125.883 | d:\IO.dat (10240MiB)
2 | 3248128 | 793 | 0.03 | 6.61 | 2630.991 | 2.35 | 1753.819 | d:\IO.dat (10240MiB)
3 | 2572288 | 628 | 0.02 | 5.23 | 3222.024 | 3.01 | 3842.501 | d:\IO.dat (10240MiB)
4 | 2928640 | 715 | 0.02 | 5.96 | 2828.556 | 2.06 | 4196.111 | d:\IO.dat (10240MiB)
5 | 2076672 | 507 | 0.02 | 4.22 | 4185.003 | 2.95 | 7686.252 | d:\IO.dat (10240MiB)
6 | 1851392 | 452 | 0.01 | 3.77 | 5343.481 | 3.35 | 11412.368 | d:\IO.dat (10240MiB)
7 | 1536000 | 375 | 0.01 | 3.12 | 5519.454 | 3.28 | 12871.924 | d:\IO.dat (10240MiB)
8 | 2904064 | 709 | 0.02 | 5.91 | 3048.215 | 1.92 | 4849.224 | d:\IO.dat (10240MiB)
9 | 2269184 | 554 | 0.02 | 4.62 | 3856.846 | 2.86 | 6240.374 | d:\IO.dat (10240MiB)
10 | 2850816 | 696 | 0.02 | 5.80 | 2607.930 | 2.18 | 2642.512 | d:\IO.dat (10240MiB)
11 | 2904064 | 709 | 0.02 | 5.91 | 2620.135 | 2.56 | 3341.952 | d:\IO.dat (10240MiB)
12 | 3420160 | 835 | 0.03 | 6.96 | 2313.736 | 3.16 | 520.690 | d:\IO.dat (10240MiB)
13 | 2744320 | 670 | 0.02 | 5.58 | 2821.371 | 2.90 | 2522.770 | d:\IO.dat (10240MiB)
14 | 1576960 | 385 | 0.01 | 3.21 | 5834.561 | 3.02 | 14748.978 | d:\IO.dat (10240MiB)
15 | 3514368 | 858 | 0.03 | 7.15 | 2331.892 | 3.24 | 504.676 | d:\IO.dat (10240MiB)
---------------------------------------------
total: 42287104 | 10324 | 0.34 | 86.03 | 3177.311 | 11.11 | 5782.375
ESXi Monitoring tool looks normal. It reports as low as the benchmark tool is actually writing on disk.
What can I do to find the reason and fix the performance issue?
