Score:0

All ssh sessions via jump server to a vm slow down or even time out if ping a dead server from inside one of the session

au flag

Env: vm1 and vm2(both redhat8.2) are behind the same gateway(Ubuntu 20.04.2), and this gateway also acts as the jump server for my macbook to ssh to vm1 and vm2. This gateway is the default gateway for both vm1 and vm2.

Situation: I ssh to vm1 with 2 sessions via the jump server, and I also ssh to vm2 via the jump server also.

Problem: If I "ping" any dead server or "arping" any unreachable server from inside one of my ssh session to vm1, then both 2 sessions to vm1 become very slow even timeout. And it becomes extremely slow when I try to ssh to vm1 via the jump server again.

Observation: And the interesting thing is that if I ssh from vm2 to vm1, then everything works well, this new session is pretty fast, and I can do whatever normal operations as usual without any slow down. And if I kill the ping or arping process here before the 2 ssh sessions to vm1 timed out, then after few minutes, both of the 2 ssh sessions to vm1 become normal again.

Any idea why this happens? thanks a lot!

macbook --> gateway(jump server x.x.0.1) --> vm1,vm2 (x.x.3.101, x.x.3.102)

xu wang avatar
au flag
And I located that the slow down happens between the jump server and vm1, because if I ssh to vm1 directly from the jump server, it's also extremely slow. so looks like acting as both default gateway and jump server for vm1 caused some sort of conflicts which maybe arp related ...
xu wang avatar
au flag
Seems I found the possible root cause, because the jump server is used as the default gateway for large numbers of vms, so actually, it doesn't has the mac address of vm1 in it cache because it always sent tons of arp request asking for mac address of all vms behind it, so it always has to send out a arp broadcast to ask for vm1's mac address, but somehow the flood of arp boradcast messages asking for the mac address of the dead server greatly slow down the process for the gateway to get the mac address from vm1.
xu wang avatar
au flag
this is why all sessions to vm1 become very slow. and because vm2 already has vm1's mac address in its cache, so there is not any slow down when I ssh from vm2 to vm1. I'm thinking adding the arp cache size for the gateway to mitigate this problem...
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.