Cannot transfer data from one specific system (MTU issue?)

Summary

The default MTU value prevents data transfer for one system. Reducing it manually enables the data transfer once more, but this manual adjustment is unnecessary on an adjacent system.

Background

I have a Backups Server. I have two Raspberry Pi systems in remote locations: one in the UK and one in Albania. Alongside the Albanian Pi is a QNAP (a Linux-derived fileserver). I have an IPSec VPN from Albania to the Backups Server network, and another IPSec VPN from the UK to the Backups Server network. Everything is using IPv4. This diagram may help:

                                           |--Albania Pi
                    |<--IPSec-->|--Router--|--Albania QNAP
Central  |          |                      |--Other systems
Backups--|--Router--|
Server   |          |                      |
                    |<--IPSec-->|--Router--|--UK Pi
                                           |--Other systems

Both Pi systems and the Backups Server all have nftables without any rules, and with a default policy of ACCEPT.

The two RPi systems, the QNAP, and the Backups Server all have a default wired ethernet MTU of 1500, as would be expected.

The router's WAN-side MTU for Albania is 1442, and for the UK it's 1500. According to the routers' Path MTU Discovery option these values are correct. For the firewall managing the network containing the Backups Server it's 1500, and this is also correct.

Problem

• If I transfer a block of data from the Pi in the UK to the backups server, it works fine
• If I transfer a block of data from the QNAP in Albania to the backups server, it works fine
• If I attempt to transfer a block of data from the Pi in Albania to the backups server, it fails
• If I reduce the Pi's MTU to 1374 the transfer succeeds

More information

Here's an example of the sort of thing that's working/failing

# On the Albanian Pi
dd bs=1M count=100 if=/dev/urandom >100M.dat

# On the Backups Server
ssh albanian_pi cat 100M.dat | pv >100M.dat

# MTU adjustments on Albanian Pi
ifconfig eth0 mtu 1500    # Default before I started fiddling
ifconfig eth0 mtu 1374    # Highest value that permits data flow

The transfer used to work, but that was before an upgrade from Stretch to Buster. I'm not seeing problems to most of the other Pi systems I have, and in particular not to the UK Pi I mentioned at the beginning that is now also running Buster.

No-one in the Albania office is complaining about network issues.

I've not knowingly got a "Do Not Fragment" bit set for packets between the Albanian Pi and the Backups Server. I've not knowingly got anything blocking PMTU Discovery.

To summarise:

• Albanian QNAP -> Backups Server: all good
• Albanian Pi -> Backups Server: fails without a reduction in the MTU
• UK Pi -> Backups Server: all good

I have a workaround, but I shouldn't have to reduce the MTU on individual systems.

Question

What is actually wrong, and how can I further diagnose and resolve the cause of the problem?

Suggestions gratefully received. Thanks

It looks like Path MTU Discovery is broken between the "Albania Pi" and "Central Backups Server" systems. This is often a result of misguidedly blocking ICMP packets in routers, firewalls or iptables settings of involved systems.

In your case, since "Albania QNAP" system appears to work fine, the prime suspect is the "Albania Pi" system. Check the configuration of that system, specifically any iptables setting and ICMP and PMTUD related network settings.