I encountered a counterintuitive phenomenon that the rtt(Round-Trip Time) of request from client to a low-network-latency server(ServerL) is higher than the one from the same client to a high-network-latency server(ServerH). The interaction between server and client is quit simple: Client just sends a large bytes stream (1~2M bytes) and server accepts all the bytes and reply a simple ok return code.
Network latency is measured by ping. Ping latency:
client <-> ServerL: 1ms
client <-> ServerH: 1.9ms
It turned out that the rtt of Client to ServerL is enlarged due to server acks. ServerL sends much more acks than ServerB do. And the time interval between two adjacent ack seen by client perspective is sometimes larger than what is measured from ServerL side.
So, what's the probably deciding factors differing the numbers of acks sent by these two server? The socket buffer configuration in these two servers are the same, consequently excluding itself from being part of reasons. Configuration are checked by the following shell command:
# sysctl -a | grep congestion
net.ipv4.tcp_allowed_congestion_control = reno cubic
net.ipv4.tcp_available_congestion_control = reno cubic
net.ipv4.tcp_congestion_control = cubic
# sysctl -a | egrep "rmem|wmem|adv_win|moderate"
net.core.rmem_default = 262144000
net.core.rmem_max = 16777216
net.core.wmem_default = 262144000
net.core.wmem_max = 16777216
net.ipv4.tcp_adv_win_scale = 1
net.ipv4.tcp_moderate_rcvbuf = 1
sysctl: net.ipv4.tcp_rmem = 4096 87380 6291456
net.ipv4.tcp_wmem = 4096 16384 4194304
net.ipv4.udp_rmem_min = 4096
net.ipv4.udp_wmem_min = 4096
vm.lowmem_reserve_ratio = 256 256 32 0 0
