Score:0

ubuntu upgrade went wrong: SSH connection closes after login - no access to terminal

vn flag

After an upgrade from Ubuntu 14.04 to 16.04 on my VPS done through SSH, SSH broke. do-release-upgrade always ended up with some package error, so I left it as it was. Client was left open, after ~12 hours later when I closed the connection, and wanted to reopen it, I realized that its broken: after a successful login the server terminates the connection with message Exit status 254

I did a ssh -vvv [email protected], here are the results:

$ ssh -vvv [email protected]
OpenSSH_8.1p1, OpenSSL 1.1.1d  10 Sep 2019
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: resolve_canonicalize: hostname XXX.XXX.XXX.XXX is address
debug2: ssh_connect_direct
debug1: Connecting to XXX.XXX.XXX.XXX [XXX.XXX.XXX.XXX] port 22.
debug1: Connection established.
debug1: identity file /c/Users/USER/.ssh/id_rsa type -1
debug1: identity file /c/Users/USER/.ssh/id_rsa-cert type -1
debug1: identity file /c/Users/USER/.ssh/id_dsa type -1
debug1: identity file /c/Users/USER/.ssh/id_dsa-cert type -1
debug1: identity file /c/Users/USER/.ssh/id_ecdsa type -1
debug1: identity file /c/Users/USER/.ssh/id_ecdsa-cert type -1
debug1: identity file /c/Users/USER/.ssh/id_ed25519 type -1
debug1: identity file /c/Users/USER/.ssh/id_ed25519-cert type -1
debug1: identity file /c/Users/USER/.ssh/id_xmss type -1
debug1: identity file /c/Users/USER/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.13
debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.13 pat OpenSSH_6.6.1* compat 0x04000002
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to XXX.XXX.XXX.XXX:22 as 'USERNAME'
debug3: hostkeys_foreach: reading file "/c/Users/USER/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /c/Users/USER/.ssh/known_hosts:2
debug3: load_hostkeys: loaded 1 keys from XXX.XXX.XXX.XXX
debug3: order_hostkeyalgs: prefer hostkeyalgs: [email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: [email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],[email protected],[email protected],ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],aes128-cbc,3des-cbc,aes256-cbc,aes192-cbc
debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],aes128-cbc,3des-cbc,aes256-cbc,aes192-cbc
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,[email protected],zlib
debug2: compression stoc: none,[email protected],zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: host key algorithms: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,[email protected],[email protected],[email protected],aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,[email protected],[email protected],[email protected],aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-sha1,[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-sha1,[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: compression ctos: none,[email protected]
debug2: compression stoc: none,[email protected]
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: [email protected]
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:MjRBEXmM8Lpl5zUEXnixUr7GgOxV1ka9UXI5qhB8uhE
debug3: hostkeys_foreach: reading file "/c/Users/USER/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /c/Users/USER/.ssh/known_hosts:2
debug3: load_hostkeys: loaded 1 keys from XXX.XXX.XXX.XXX
debug1: Host 'XXX.XXX.XXX.XXX' is known and matches the ECDSA host key.
debug1: Found key in /c/Users/USER/.ssh/known_hosts:2
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /c/Users/USER/.ssh/id_rsa
debug1: Will attempt key: /c/Users/USER/.ssh/id_dsa
debug1: Will attempt key: /c/Users/USER/.ssh/id_ecdsa
debug1: Will attempt key: /c/Users/USER/.ssh/id_ed25519
debug1: Will attempt key: /c/Users/USER/.ssh/id_xmss
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /c/Users/USER/.ssh/id_rsa
debug3: no such identity: /c/Users/USER/.ssh/id_rsa: No such file or directory
debug1: Trying private key: /c/Users/USER/.ssh/id_dsa
debug3: no such identity: /c/Users/USER/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /c/Users/USER/.ssh/id_ecdsa
debug3: no such identity: /c/Users/USER/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /c/Users/USER/.ssh/id_ed25519
debug3: no such identity: /c/Users/USER/.ssh/id_ed25519: No such file or directory
debug1: Trying private key: /c/Users/USER/.ssh/id_xmss
debug3: no such identity: /c/Users/USER/.ssh/id_xmss: No such file or directory
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
[email protected]'s password:
debug3: send packet: type 50
debug2: we sent a password packet, wait for reply
debug3: receive packet: type 52
debug1: Authentication succeeded (password).
Authenticated to XXX.XXX.XXX.XXX ([XXX.XXX.XXX.XXX]:22).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Requesting [email protected]
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IP_TOS 0x48
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug3: send packet: type 98
debug2: channel 0: request shell confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype [email protected] reply 0
debug2: channel 0: rcvd eow
debug2: channel 0: chan_shutdown_read (i0 o0 sock -1 wfd 4 efd 6 [write])
debug2: channel 0: input open -> closed
debug3: receive packet: type 96
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug3: receive packet: type 97
debug2: channel 0: rcvd close
debug3: channel 0: will not send data after close
Welcome to Ubuntu 16.04 LTS (GNU/Linux 3.13.0-042stab141.3 x86_64)

 * Documentation:  https://help.ubuntu.com/
New release '18.04.5 LTS' available.
Run 'do-release-upgrade' to upgrade to it.

No mail.
Last login: Wed Jun 16 21:58:00 2021 from MYHOST
debug3: channel 0: will not send data after close
debug2: channel 0: obuf empty
debug2: channel 0: chan_shutdown_write (i3 o1 sock -1 wfd 5 efd 6 [write])
debug2: channel 0: output drain -> closed
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug3: send packet: type 97
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
  #0 client-session (t4 r0 i3/0 o3/0 e[write]/0 fd -1/-1/6 sock -1 cc -1)

debug3: send packet: type 1
debug3: fd 1 is not O_NONBLOCK
Connection to XXX.XXX.XXX.XXX closed.
Transferred: sent 2236, received 2512 bytes, in 0.3 seconds
Bytes per second: sent 6692.5, received 7518.6
debug1: Exit status 254

The problem is that SSH was my only option to access the server (self-managed VPS, no possibility to access a remote console or to get a snapshot), so support recommends destroying and rebuilding the server, which is not a friendly option.

I'm trying to save the day by somehow getting SSH access back, but can't get past the above error. I don't have any server-side logs. According to the stats available on the hosting side, Memory is available (RAM usage is ~16%),

I've tried to:

  • send ^C after logging in to intercept .bashrc or .bash_profile; it didnt work, and I'm not even sure that that's causing the problem
  • ssh [email protected] bash --noprofile --norc, same outcome
  • ssh [email protected] 'echo something >&2 I see debug1: Sending command: echo something >&2 in the log, but no result on the output
  • access the database: after rebooting the system using the Server Dashboard site MySQL server never got back online
  • ask for a remote terminal or a VM snapshot from the hosting company, didnt work out

All help is appreciated

Michael Hampton avatar
cz flag
They don't provide a remote console?! That's really rather ridiculous.
vn flag
@MichaelHampton That was my reaction as well... I should've checked this before starting to upgrade, now its the harder way to recover - if possible at all
Michael Hampton avatar
cz flag
Oh, I just saw the godaddy tag. That explains everything. Do yourself a favor, get a backup of your VM and go anywhere else.
vn flag
@MichaelHampton cant even get a backup without SSH access :)
Michael Hampton avatar
cz flag
What? That's utterly insane! They should be able to provide you a complete image of your VPS.
vn flag
Just to make sure, I contacted them once again with this exact question; they've told me that it cant be done. I made it clear that I'm willing to pay for such a service, still no
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.