Score:0

Cannot connect to server after release upgrade

gw flag

I was in the process of upgrading an old 16.04 LTS, and seemingly locked myself out with no way back in. Help.

I followed the basic steps:

sudo apt-get update
sudo apt-get upgrade -y
sudo apt-get dist-upgrade
sudo do-release-upgrade

During the release-upgrade, everything went smoothly. No error messages, nothing - it's the restart that killed my access to the system. The secondary SSH process started by it on port 1022 times out completely as well.

Now, the problem is, I cannot attempt to connect as root@ip, since that was disconnected. Only connecting with an SSH key is possible.

This is the debug from SSH:

:~$ ssh -vvvvvvvvv atlas
OpenSSH_7.6p1 Ubuntu-4ubuntu0.5, OpenSSL 1.0.2n  7 Dec 2017
debug1: Reading configuration data /home/name/.ssh/config
debug1: /home/name/.ssh/config line 1: Applying options for atlas
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "(ip)" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to (ip) [(ip)] port 22.
debug1: Connection established.
debug1: identity file /home/name/.ssh/atlas_a type 0
debug1: key_load_public: No such file or directory
debug1: identity file /home/name/.ssh/atlas_a-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.5
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4ubuntu0.5
debug1: match: OpenSSH_7.6p1 Ubuntu-4ubuntu0.5 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to (ip):22 as 'name'
debug3: hostkeys_foreach: reading file "/home/name/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/name/.ssh/known_hosts:2
debug3: load_hostkeys: loaded 1 keys from (ip)
debug3: order_hostkeyalgs: prefer hostkeyalgs: [email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
Connection closed by (ip) port 22

(I edited out name & ip)

What is going on? Why can't I connect? Did I completely brick everything, or is there a chance to somehow recover from this? It's a virtual server, I do not have access to it physically and remote restarts seem to fail now as well.

Help :)

Edit: Since it seems that I won't be able to salvage this, I'll be a bit more specific: What exactly did I do wrong here? What could I have done to prevent this? The SSH debug doesn't give me any feedback on what actually caused the problem and the "reserve" port (1022) that was opened for this specific case, just doesn't work at all.

If I have to lose the whole server, at least I would like to learn from this - but it just seems to "not work", and that's it?

Edit 2: Surprisingly, I managed to access the server again in a "repair" mode offered by the host. What should I do to guarantee access and fix this, now that I'm in? (I have access to all files via VI, they are placed in a /repair/ folder)

user535733 avatar
cn flag
Is your data on that VPS backed up? You will need assistance from your VPS provider to resolve this.
Katai avatar
gw flag
@user535733 yeah, thank god I backed it up - but I don't even know why the SSH doesn't work, does the debug not say anything? I mean, is my key not recognized, is the firewall blocking me, etc etc? Why would an upgrade just suddenly lock me out? :(
Organic Marble avatar
us flag
Is there another way to get in from the vendor - for example Digital Ocean provides a console login that can be used if you lock yourself out of ssh.
Katai avatar
gw flag
@OrganicMarble doesn't look like it. Even attempts to simply remote-restart the server doesn't work; but I'll call them and ask. But in general... what the heck did I do wrong here? I mean, what could I even have done differently? I will edit my question.
ru flag
@Katai It might be totally unrelated to something you did, but without access to the system we can't confirm anything. It's possible firewall rules got messed up, or the SSH service has a legacy deprecated configuration leftover in it that does not work in the upgraded OpenSSH Server, there's no way to tell without access to the machine and the syslogs to see the errors on bootup or server side errors. Client side debug logs won't help us debug it.
Organic Marble avatar
us flag
I logged on to my vps with those verbose options; until the key exchange started, the only possibly significant difference I saw in the output was the line `key_load_public: No such file or directory` That line didn't appear in my output in the identity file area.
Katai avatar
gw flag
@ThomasWard I managed to get access to the files, via a repair function - but I can't seem to find any relevant logs. Everything I check seems unrelated - and trying to disable ufw and open up the sshd restrictions do not give any result - I still can't log into the system at all.
Organic Marble avatar
us flag
Take a look at https://serverfault.com/questions/813519/ssh-apparently-not-reading-keys-inside-ssh/813545#813545 and https://unix.stackexchange.com/questions/321968/trying-to-ssh-into-server-and-getting-key-load-public-no-such-file-or-directory/321972
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.