Issues with Shared NTFS HD | Ubuntu 20.04 x Windows 10

Dual boot machine with Ubuntu 20.04 and Windows 10 on seperate m.2 nvme storage devices. I have an external hard drive (14TB) set up as NTFS. On either operating system I can write to the disc. However, when I open files on the HD in Windows 10, if I generated those files using Ubuntu 20.04, they are often corrupted. For example:

D:\my\path> type myfile.mrc.tlt
The file or directory is corrupted and unreadable.

I have seen this behavior on two external hard drives (one Seagate and another WD). I had assumed the problem was with the Seagate drive. But I have now replicated it with a WD one.

Not sure where to start toubleshooting from here.

When I mount the drive while running journalctl -f I get the following:

Nov 05 17:12:21 axoneme udisksd[894]: Mounted /dev/sdd1 at /media/jared/Elements on behalf of uid 1000
Nov 05 17:12:21 axoneme dbus-daemon[1641]: [session uid=1000 pid=1641] Activating via systemd: service name='org.freedesktop.Tracker1' unit='tracker-store.service' requested by ':1.1' (uid=1000 pid=1637 comm="/usr/libexec/tracker-miner-fs " label="unconfined")
Nov 05 17:12:21 axoneme systemd[1629]: Starting Tracker metadata database store and lookup manager...
Nov 05 17:12:21 axoneme dbus-daemon[1641]: [session uid=1000 pid=1641] Activating service name='org.gnome.Shell.HotplugSniffer' requested by ':1.37' (uid=1000 pid=1860 comm="/usr/bin/gnome-shell " label="unconfined")
Nov 05 17:12:21 axoneme dbus-daemon[1641]: [session uid=1000 pid=1641] Successfully activated service 'org.gnome.Shell.HotplugSniffer'
Nov 05 17:12:21 axoneme dbus-daemon[1088]: [session uid=125 pid=1088] Successfully activated service 'org.freedesktop.Tracker1'
Nov 05 17:12:21 axoneme systemd[1072]: Started Tracker metadata database store and lookup manager.
Nov 05 17:12:21 axoneme dbus-daemon[1641]: [session uid=1000 pid=1641] Successfully activated service 'org.freedesktop.Tracker1'
Nov 05 17:12:21 axoneme systemd[1629]: Started Tracker metadata database store and lookup manager.
Nov 05 17:12:21 axoneme ntfs-3g[5491]: Trying to read non-allocated mft records (10255 > 9984): Illegal seek
Nov 05 17:12:21 axoneme ntfs-3g[5491]: Trying to read non-allocated mft records (10256 > 9984): Illegal seek
Nov 05 17:12:21 axoneme ntfs-3g[5491]: Trying to read non-allocated mft records (10164 > 9984): Illegal seek
Nov 05 17:12:21 axoneme ntfs-3g[5491]: Trying to read non-allocated mft records (10165 > 9984): Illegal seek
Nov 05 17:12:22 axoneme ntfs-3g[5491]: Trying to read non-allocated mft records (10009 > 9984): Illegal seek
Nov 05 17:12:22 axoneme ntfs-3g[5491]: Trying to read non-allocated mft records (10010 > 9984): Illegal seek
Nov 05 17:12:22 axoneme ntfs-3g[5491]: Trying to read non-allocated mft records (10030 > 9984): Illegal seek
Nov 05 17:12:22 axoneme ntfs-3g[5491]: Trying to read non-allocated mft records (10031 > 9984): Illegal seek

Similarly, if I run ls -lth in a directory on the NTFS HD with Ubuntu 20.04, I get the following in the corrupted directorys:

Nov 05 17:16:03 axoneme ntfs-3g[5491]: Trying to read non-allocated mft records (10294 > 9984): Illegal seek
Nov 05 17:16:03 axoneme ntfs-3g[5491]: Trying to read non-allocated mft records (10290 > 9984): Illegal seek
Nov 05 17:16:03 axoneme ntfs-3g[5491]: Trying to read non-allocated mft records (10360 > 9984): Illegal seek

So, discussion in comments showed that the problem starts happening once Windows gets to access the NTFS partition. So, is it a Windows problem? Looks likely, though there's a chance the ntfs-3g FUSE driver interprets something in an incorrect way compared to the Windows one, resulting in an incompatibility.

It is interesting that this problem seems to be extremely rare (I found just a few posts with the exact errors from journalctl, one from 2008 year, and another about some odd interaction with RAID). That is something to note, because that might imply you have some special configuration that causes these problems, and it would be very interesting to find out what that could be. But I'll leave that as an exercise to a reader.

In terms of a workaround what you can try is:

1. Try the new ntfs3 kernel driver (as opposed to ntfs-3g you're using), contributed into the kernel by Paragon Software since Linux 5.15. Not to be confused with older read-only ntfs kernel driver, which still haven't been removed. You will need to update to 5.15 or higher Linux kernel version. The 5.15 seems to be used by default in 22.04 (and I recommend you to upgrade 20.04 → 22.04 because by having an older software on your 20.04 you're missing out on lots of optimizations and features).

   Offhand, I don't know how to make file managers to use ntfs3 by default, but you might, for example, add a /etc/fstab entry that makes use of ntfs3 driver.

   This may or may not help with your problem. But if it doesn't, then I am 97% sure this is purely a fault of specifically yours Windows system (see also my point about the rareness). The reason for my confidence is that Paragon Software is an old company that were selling their filesystem drivers for a long time, and I am pretty sure they had enough expertise and practical experience to work out possible incompatibilities with the original Windows driver.

2. If you're using NTFS specifically to share the files, you might also consider:

   1. Using UDF filesystem instead. It is supported by both Windows and Linux.
   2. Using exfat. Since 5.7 SAMSUNG has added a driver for exfat, and they also released exfatprogs, so the proper support is in place.

P.S.: ideally, you'd also have try latest ntfs-3g, then if problem is still reproducible, report a bug. Though you might need to convince the developers that it's really a ntfs-3g problem. If the ntfs3 driver will work fine for you, then that might be an implicit evidence that the problem is in ntfs-3g driver.