ext4 errors with dm-crypt / LUKS on top of an iSCSI device

I have two oldish "NAS" devices, specifically Buffalo TeraStation Rackmount iSCSI ("TS-RIXL/R5"). Those are basically embedded computers running Linux with four SATA slots and two Ethernet ports, with a firmware that shows up as an iSCSI target.

With both devices, i did the following:

I put in four new 16TB Seagate Exos X16 drives, used Buffalo's Windows "Firmware update" tool to install the latest (1.66) firmware on them, and created a RAID5 array.

Then, on an amd64 machine running Debian Buster with standard kernel and tools, I tried to setup an encrypted filesystem:

# find the device
iscsiadm -m discovery -t st -p <ip_address>:3260
# connect it to the initiator's SCSI subsystem
iscsiadm  -m node  --targetname "iqn.2004-08.jp.buffalo:<device_id>:vol1" --portal <ip_address>:3260,1" --login
# this made /dev/sda appear

# create an encrypted device
cryptsetup luksFormat /dev/sda
cryptsetup open /dev/sda nas

# create a filesystem and mount it
mke2fs -t ext4 /dev/mapper/nas
mount /dev/mapper/nas /mnt/nas/

# make a directory
mkdir /mnt/nas/store

This appeared to work fine, but immediately made the follwoing error appear in dmesg and syslog:

EXT4-fs error (device dm-0): ext4_validate_block_bitmap:384: comm mkdir: bg 18736: bad block bitmap checksum

Copying some gigabytes of data to the device then also made errors appear such as

EXT4-fs (dm-0): Delayed block allocation failed for inode 479102523 at logical offset 8708096 with max blocks 128 with error 74
EXT4-fs (dm-0): This should not happen!! Data will be lost

This is reproducible with both NAS devices, ruling out some hardware fault.

The devices have been working flawlessly for years (with iSCSI, without encryption).

If I do the same thing now without the "cryptsetup" part, make the filesystem on /dev/sda directly and mount it, I can work with multiple gigabytes of data without errors appearing in the syslog, and without having any data corruption.

On the other hand, using the same procedure on the Debian machine, setting up an encrypted filesystem on a hard disk connected via USB (instead of iSCSI), works fine as well.

So:

ext4 => dm-crypt => iSCSI => Buffalo NAS: "This should not happen!! Data will be lost"

ext4 => iSCSI => Buffalo NAS: works

ext4 => dm-crypt => external USB disk: works

(where "works" means: i copied around 10'000 files with a size of about 48 GB, and verified using md5sum that all files were there and were not corrupted)

I then repeated the tests on a machine running Ubuntu hirsute (21.04), and got the same results.

Is there some known problem running dm-crypt on top of iSCSI? Do I maybe need to have a special configuration for the iSCSI part, for example some configuration for the caching or block size?

So, it turns out the problem was not actually with the combination of dm-crypt on top of iSCSI, but rather a limitation/bug in the TeraStation firmware.

With 4 TB disks instead of 16 TB, the problem does not manifest itself.

When exposing the 16 TB disks as individual iSCSI LUNs, and then run Linux software RAID on top of them, the problem does not manifest itself either.

I'm leaving this here in case someone googles it.

I didn't think of this, because I knew the TeraStation was running a version of Linux, and I had never, in any Linux installation, had a 16 TB disk behave different than a 4 TB disk, so I didn't think this would hit a limitation like this - and, also, I figured that if there would be a problem with the disks's size, the TeraStation would throw up some error message and not act in such a weird, somehow-data-corrupting way.