Why is SSH disconnecting with 'ssh_dispatch_run_fatal: Connection to x.x.x.x port 2020: Connection corrupted'?

I'm trying to figure out why I keep getting disconnected from my SSH session. The host is a CentOS server and the clients are MacOS.

The error seems to be sporadic - sometimes it's once an hour, other times it's several times in a minute. But what's interesting is that this doesn't just happen when I don't use the connection - I can literally be typing and the connection will go. The error is:

ssh_dispatch_run_fatal: Connection to x.x.x.x port 2020: Connection corrupted

I have also tried having multiple tabs open with an ssh connection open in each and they seem to disconnect at the same time.

Turning debug level 3 on, I have the /var/log/secure output here surrounding the time of a disconnect:

Feb 16 08:06:08 46 sshd[2864]: debug2: channel 0: request [email protected] confirm 1
Feb 16 08:06:08 46 sshd[2864]: debug3: send packet: type 98
Feb 16 08:06:08 46 sshd[2864]: debug3: receive packet: type 100
Feb 16 08:06:08 46 sshd[2864]: debug1: Got 100/121 for keepalive
Feb 16 08:06:38 46 sshd[2864]: debug2: channel 0: request [email protected] confirm 1
Feb 16 08:06:38 46 sshd[2864]: debug3: send packet: type 98
Feb 16 08:06:38 46 sshd[2864]: debug3: receive packet: type 100
Feb 16 08:06:38 46 sshd[2864]: debug1: Got 100/122 for keepalive
Feb 16 08:06:51 46 sshd[4517]: Connection closed by x.x.x.x port 62073
Feb 16 08:06:51 46 sshd[4517]: debug1: channel 0: free: server-session, nchannels 1
Feb 16 08:06:51 46 sshd[4517]: debug3: channel 0: status: The following connections are open:\r\n  #0 server-session (t4 r0 i0/0 o0/0 fd 12/8 cc -1)\r\n
Feb 16 08:06:51 46 sshd[4517]: Close session: user my_username from x.x.x.x port 62073 id 0
Feb 16 08:06:51 46 sshd[4517]: debug3: mm_request_send entering: type 30
Feb 16 08:06:51 46 sshd[4517]: debug3: session_unused: session id 0 unused
Feb 16 08:06:51 46 sshd[4517]: debug1: do_cleanup
Feb 16 08:06:51 46 sshd[4517]: debug3: PAM: sshpam_thread_cleanup entering
Feb 16 08:06:51 46 sshd[4517]: debug3: mm_request_send entering: type 122
Feb 16 08:06:51 46 sshd[4517]: debug3: mm_request_receive_expect entering: type 123
Feb 16 08:06:51 46 sshd[4517]: debug3: mm_request_receive entering
Feb 16 08:06:51 46 sshd[4514]: debug3: mm_request_receive entering
Feb 16 08:06:51 46 sshd[4514]: debug3: monitor_read: checking request 30
Feb 16 08:06:51 46 sshd[4514]: debug3: mm_answer_pty_cleanup entering
Feb 16 08:06:51 46 sshd[4514]: debug1: session_by_tty: session 0 tty /dev/pts/1
Feb 16 08:06:51 46 sshd[4514]: debug3: mm_session_close: session 0 pid 4517
Feb 16 08:06:51 46 sshd[4514]: debug3: mm_session_close: tty /dev/pts/1 ptyfd 4
Feb 16 08:06:51 46 sshd[4514]: debug1: session_pty_cleanup: session 0 release /dev/pts/1
Feb 16 08:06:51 46 sshd[4514]: debug3: session_unused: session id 0 unused
Feb 16 08:06:51 46 sshd[4514]: debug3: mm_request_receive entering
Feb 16 08:06:51 46 sshd[4514]: debug3: monitor_read: checking request 122
Feb 16 08:06:51 46 sshd[4514]: debug3: mm_request_send entering: type 123
Feb 16 08:06:51 46 sshd[4517]: debug3: mm_request_send entering: type 124
Feb 16 08:06:51 46 sshd[4517]: Transferred: sent 1503792, received 18928 bytes
Feb 16 08:06:51 46 sshd[4517]: Closing connection to x.x.x.x port 62073
Feb 16 08:06:51 46 sshd[4517]: debug3: mm_audit_event entering
Feb 16 08:06:51 46 sshd[4517]: debug3: mm_request_send entering: type 112
Feb 16 08:06:51 46 sshd[4517]: debug3: mm_request_send entering: type 50
Feb 16 08:06:51 46 sshd[4514]: debug3: mm_request_receive entering
Feb 16 08:06:51 46 sshd[4514]: debug3: monitor_read: checking request 124
Feb 16 08:06:51 46 sshd[4514]: debug3: mm_request_receive entering
Feb 16 08:06:51 46 sshd[4514]: debug3: monitor_read: checking request 112
Feb 16 08:06:51 46 sshd[4514]: debug3: mm_answer_audit_event entering
Feb 16 08:06:51 46 sshd[4514]: debug3: mm_request_receive entering
Feb 16 08:06:51 46 sshd[4514]: debug3: monitor_read: checking request 50
Feb 16 08:06:51 46 sshd[4514]: debug3: mm_answer_term: tearing down sessions
Feb 16 08:06:51 46 sshd[4514]: debug1: PAM: cleanup
Feb 16 08:06:51 46 sshd[4514]: debug1: PAM: closing session
Feb 16 08:06:51 46 sshd[4514]: pam_unix(sshd:session): session closed for user my_username
Feb 16 08:06:51 46 sshd[4514]: debug1: PAM: deleting credentials

Parts are obfuscated with x or my_username

Update

I've added a new set of logs above - less than originally but I'm hoping the applicable ones. The new line:

Feb 16 08:06:51 46 sshd[4517]: Connection closed by x.x.x.x port 62073

interests me. Would this suggest that my device is making the disconnect and it's not SSH related? I see (I think) successful keep-alive packets being sent right before the disconnect.

I've tried restarting the daemon, and using the OpenSSH (via brew) version of SSH on the Mac and that's made no difference.

Server info:

● sshd.service - OpenSSH server daemon
   Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2022-02-11 10:21:40 GMT; 1h 52min ago
     Docs: man:sshd(8)
           man:sshd_config(5)
 Main PID: 6271 (sshd)
   CGroup: /system.slice/sshd.service
           └─6271 /usr/sbin/sshd -D

The only differences in our sshd_config to the standard CentOS one is the port, log level and:

ClientAliveInterval 30
ClientAliveCountMax 5

Why am I getting disconnected? What could be causing the connection to be corrupted?

This is most likely disconnection issue, I am not sure which version you are using if you check the ssh code base it's says:

ssh will return SSH_ERR_CONN_CORRUPT mostly due to sshpkt_disconnect.

Now question is how you can keep your connection alive, we have multiple ways one way "milenao" already shared but before changing setting please read the effect first. Other way is run kind of tunnel to send any kind of traffic so service think connection is active such as you can send ping, date, watch, sleep after some frequency according to your need.

Update:

With above logs what I can say your connection is not alive and connection close has been initiated this is same what we are predicting.

   /* The connection has been terminated. */
   ssh_packet_get_bytes(ssh, &ibytes, &obytes);
   verbose("Transferred: sent %llu, received %llu bytes",
   (unsigned long long)obytes, (unsigned long long)ibytes);

   verbose("Closing connection to %.500s port %d", remote_ip, remote_port);

I had the same issue so I added the following to my .ssh/config file and I have not had seen that error yet.

Host i-* mi-*
ServerAliveInterval 300
ServerAliveCountMax 2

No guarantees that this will help but so far it has allowed my connection to stay open longer than it has.

you can try increasing the ClientAliveInterval and ClientAliveCountMax values:

ClientAliveInterval 60
ClientAliveCountMax 10

Most likely, the server stops receiving packets from clients and the connection is disconnected due to a timeout

From here: https://linux-tips.com/t/how-to-solve-ssh-disconnect-packet-corrupt-problems/258

Try running this command to disable the hardware checksum:

sudo ethtool -K eth0 tx off rx off

It sounds like it could possibly be a hardware problem. I've also read elsewhere that turning the whole server on/off or maybe doing an ip link set dev ${INTERFACE} down; ip link set dev ${INTERFACE} up might work as well.