Poor NTFS performance

The Bic Pen

6/18/24, 8:45 PM

I'm getting very poor write performance on an external USB HDD that has an NTFS partition. I know that the sync option can slow down NTFS quite a lot, but that option isn't on for me: rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other,blksize=4096,uhelper=udisks2. As far as I know, big_writes is no longer necessary since it is on by default in the ntfs-3g driver. What can I do to improve performance?

Here's the result of a disk benchmark:

                        KDiskMark (3.1.3): https://github.com/JonMagon/KDiskMark
                    Flexible I/O Tester (fio-3.34): https://github.com/axboe/fio
---------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

[Read]
Sequential   1 MiB (Q=  8, T= 1):  5683.719 MB/s [   5550.5 IOPS] <  1417.53 us>
Sequential   1 MiB (Q=  1, T= 1):  6310.817 MB/s [   6162.9 IOPS] <   161.15 us>
    Random   4 KiB (Q= 32, T= 1):   755.180 MB/s [ 188795.1 IOPS] <   169.29 us>
    Random   4 KiB (Q=  1, T= 1):   753.670 MB/s [ 188417.6 IOPS] <     5.17 us>

[Write]
Sequential   1 MiB (Q=  8, T= 1):    11.009 MB/s [     10.8 IOPS] <  2564.91 us>
Sequential   1 MiB (Q=  1, T= 1):    10.485 MB/s [     10.2 IOPS] <   238.27 us>
    Random   4 KiB (Q= 32, T= 1):     6.108 MB/s [   1527.0 IOPS] <   377.90 us>
    Random   4 KiB (Q=  1, T= 1):     4.234 MB/s [   1058.5 IOPS] <    11.58 us>

Profile: Default
   Test: 1 GiB (x5) [Measure: 5 sec / Interval: 5 sec]
   Date: 2023-06-18 13:37:59
     OS: org.kde.Platform 5.15-21.08 [linux 5.19.0-45-generic]

~~The read performance looks very high (too high?), but the write performance is abysmal, and this speed is similar to what I was getting with rsync.~~ The above results are obviously incorrect as there is no way an HDD connected over USB3 could achieve such read speeds. Ignore the read speeds, but the write speeds seem to match what I got in practice.

The performance is normal on Windows:

---------------------------------------------
CrystalDiskMark 8.0.4 x64 (C) 2007-2021 hiyohiyo
                                  Crystal Dew World: https://crystalmark.info/
---------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

[Read]
  SEQ    1MiB (Q=  8, T= 1):    68.943 MB/s [     65.7 IOPS] <120142.86 us>
  SEQ    1MiB (Q=  1, T= 1):    65.787 MB/s [     62.7 IOPS] < 15884.81 us>
  RND    4KiB (Q= 32, T= 1):     1.068 MB/s [    260.7 IOPS] <118571.44 us>
  RND    4KiB (Q=  1, T= 1):     0.422 MB/s [    103.0 IOPS] <  9676.17 us>

[Write]
  SEQ    1MiB (Q=  8, T= 1):    93.746 MB/s [     89.4 IOPS] < 84761.63 us>
  SEQ    1MiB (Q=  1, T= 1):    69.955 MB/s [     66.7 IOPS] < 13087.88 us>
  RND    4KiB (Q= 32, T= 1):     5.369 MB/s [   1310.8 IOPS] < 24361.27 us>
  RND    4KiB (Q=  1, T= 1):     5.588 MB/s [   1364.3 IOPS] <   730.32 us>

Profile: Default
   Test: 1 GiB (x5) [E: 88% (1229/1397GiB)]
   Mode: [Admin]
   Time: Measure 5 sec / Interval 5 sec 
   Date: 2023/07/09 14:33:14
     OS: Windows 11  [10.0 Build 22621] (x64)

172

0 + 9

disk

ntfs

hard-drive

sudodus

6/18/24, 9:41 PM

My experience is that the Linux drivers for NTFS are not as good as the drivers for Linux file systems, for example ext4. On the other hand, Windows has no drivers at all for Linux file systems (so that read/write speed is 0 ;-) If it is important with a high write speed, I suggest that you use ext4 unless Windows must be able to read and write there. -- If still slow with ext4, the problem might be USB. Are you sure that the connection can use USB 3 (not only USB 2)?

The Bic Pen

6/18/24, 9:43 PM

My use case is a drive that is shared between Windows and Linux so the 0 r/w speed of ext4 on Windows would be even more of a bottleneck :(. I am using USB3, but this transfer speed is well below even USB2

sudodus

6/18/24, 9:48 PM

Well, look at USB, for example with `lsusb` and `lsusb -v` to check that there is USB 3 connection or 'full speed'. You can also check if the power supply is good enough for the HDD to work well (separate power to the drive). And maybe also check if the formatting of the file system matches the sectors of the drive (to make it work optimally).

sudodus

6/18/24, 9:53 PM

By the way, is the drive writing at much higher speed from Windows? In the same computer or in another computer? Which version of Ubuntu are you running? Can you try with a live Ubuntu system (from USB) in another computer to tell if the problem is in one particular computer or one particular installed systsem?

sudodus

6/18/24, 9:59 PM

If nothing else helps, you can try with the exFAT file system, which is probably faster (unless the problem is USB). NTFS has journaling which makes it safer against corruption, and I don't think exFAT has journaling, but if nothing else can improve speed, maybe use it anyway.

The Bic Pen

6/18/24, 10:19 PM

It's definitely USB3: `Port 7: Dev 6, If 0, Class=Mass Storage, Driver=uas, 5000M`

The Bic Pen

7/9/24, 9:35 PM

Updated with Windows performance

The Bic Pen

7/9/24, 9:48 PM

After looking at the results some more, the read speeds reported by KDiskMark are obviously incorrect as there is no way an HDD connected over USB3 could achieve such speeds. Reported the bug to the KDiskMark dev.

sudodus

7/10/24, 10:17 AM

Welcome back :-) I would still suggest that you do the tests according to my previous comments, and report back your results. That way we might exclude some dead end streets and find a way to improve your write speed ...

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Poor NTFS performance

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.