Score:0

Cannot transfer data from one specific system (MTU issue?)

my flag

Summary

The default MTU value prevents data transfer for one system. Reducing it manually enables the data transfer once more, but this manual adjustment is unnecessary on an adjacent system.

Background

I have a Backups Server. I have two Raspberry Pi systems in remote locations: one in the UK and one in Albania. Alongside the Albanian Pi is a QNAP (a Linux-derived fileserver). I have an IPSec VPN from Albania to the Backups Server network, and another IPSec VPN from the UK to the Backups Server network. Everything is using IPv4. This diagram may help:

                                           |--Albania Pi
                    |<--IPSec-->|--Router--|--Albania QNAP
Central  |          |                      |--Other systems
Backups--|--Router--|
Server   |          |                      |
                    |<--IPSec-->|--Router--|--UK Pi
                                           |--Other systems

Both Pi systems and the Backups Server all have nftables without any rules, and with a default policy of ACCEPT.

The two RPi systems, the QNAP, and the Backups Server all have a default wired ethernet MTU of 1500, as would be expected.

The router's WAN-side MTU for Albania is 1442, and for the UK it's 1500. According to the routers' Path MTU Discovery option these values are correct. For the firewall managing the network containing the Backups Server it's 1500, and this is also correct.

Problem

  • If I transfer a block of data from the Pi in the UK to the backups server, it works fine
  • If I transfer a block of data from the QNAP in Albania to the backups server, it works fine
  • If I attempt to transfer a block of data from the Pi in Albania to the backups server, it fails
  • If I reduce the Pi's MTU to 1374 the transfer succeeds

More information

Here's an example of the sort of thing that's working/failing

# On the Albanian Pi
dd bs=1M count=100 if=/dev/urandom >100M.dat

# On the Backups Server
ssh albanian_pi cat 100M.dat | pv >100M.dat

# MTU adjustments on Albanian Pi
ifconfig eth0 mtu 1500    # Default before I started fiddling
ifconfig eth0 mtu 1374    # Highest value that permits data flow

The transfer used to work, but that was before an upgrade from Stretch to Buster. I'm not seeing problems to most of the other Pi systems I have, and in particular not to the UK Pi I mentioned at the beginning that is now also running Buster.

No-one in the Albania office is complaining about network issues.

I've not knowingly got a "Do Not Fragment" bit set for packets between the Albanian Pi and the Backups Server. I've not knowingly got anything blocking PMTU Discovery.

To summarise:

  • Albanian QNAP -> Backups Server: all good
  • Albanian Pi -> Backups Server: fails without a reduction in the MTU
  • UK Pi -> Backups Server: all good

I have a workaround, but I shouldn't have to reduce the MTU on individual systems.

Question

What is actually wrong, and how can I further diagnose and resolve the cause of the problem?

Suggestions gratefully received. Thanks

Score:1
bd flag

It looks like Path MTU Discovery is broken between the "Albania Pi" and "Central Backups Server" systems. This is often a result of misguidedly blocking ICMP packets in routers, firewalls or iptables settings of involved systems.

In your case, since "Albania QNAP" system appears to work fine, the prime suspect is the "Albania Pi" system. Check the configuration of that system, specifically any iptables setting and ICMP and PMTUD related network settings.

roaima avatar
my flag
Thank you. This seems to confirm my diagnosis. I have no iptables on any of the systems concerned. The remote office routers (Draytek) are the same as I use everywhere else aside from the network hosting the Backups Server. I can't find any reference to PMTUD network settings within a Linux kernel. I'm certainly not intentionally blocking ICMP but if I was it would surely be affecting both Albania devices?
Tilman Schmidt avatar
bd flag
That's why I named "Albania Pi" the prime suspect. It is of course possible to selectively block ICMP from or to a single system but I assume if someone did you would certainly know about it.
roaima avatar
my flag
Indeed I would, yes
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.