We are experiencing FTP performance issues. Looks like speed drops dramatically the more hops we are away from the server.
Cerberus FTP server, running on VMWare. 2 machines on the same VMWare host and subnet can get 25Mbyte/s, but speed drops to 2.5MB/s 10 hops away. sftp is way faster for some reason. IIS FTP has issues as well. Running on another port (2121) did not fix anything. Cisco routers uses FTP fixup. Same issue ACTIVE/PASV FTP.
wireshark shows server is using jumbo frames (offload to virtual adapter), DF flag, yet the packet is fragmented on the client. Many ACK packages to subsets of a large packet is delivered to the FTP server at the same time (say up to 10-20 ACK packets at once). Tried to disable all offload, then we get 1380 bytes payload packets maching both ends. But speed the same, or slightly faster (less than 10% speedup).
Any insights what can be the reason ? Only FTP seems impacted. Not sftp (my recommendation is to drop FTP, which has been company policy for many years). But sftp could have other things helping it.
The distance from server could indicate that Routers doing ftp fixup slows things, but the cloud servers over VPN is the slowest (3Mbps). So latency/RTT might participate. But ping says <5ms latency when the speed is dropped to 2.5Mbyte/s on LAN. Even <1ms is way too slow.
Networking people says no Cisco Firepower yet - only Cisco ASA between client and server.
One more thing. We can see on the client, that the data being sent in 8k "buckets". i.e. 5 packets of 1380 bytes + 1 packet of 1292 bytes. All buffer in Cerberus are set to 64/256k. So maybe it is the code itself that writes 8k buffers.
If we look at inter-packet timing in wireshark, we see 2 following random 1380 byte packets spaced at about the right distance to match the reported speed. Thus everything is slowed down. wireshark graph shows the same. Consistently slow speed.
