Score:0

SSH server stopped recieving connections after reboot and kernel upgrade

mq flag

I am running Ubuntu 22.04.1 LTS on a server, and this morning I ssh'd in to it from the Termius app on my phone to run update and upgrade as per usual, then rebooted it twice to install a kernel upgrade. After the second reboot, I was no longer able to connect via ssh. I used systemctl start ssh to start the ssh server back up, as it had apparently not restarted. However, I am still unable to connect via SSH. I can tell the server's networking is not the issue, as all the other services I'm running on it such as a website, pi-hole, and minecraft server are running fine and can be connected to, and pinging to and from the server behaves as expected. ufw has my ssh port enabled still as well.

With my SSH configuration, I require both a password and an SSH key to join, and my gut tells me the SSH key is the issue. I checked in ~/.ssh/authorized_keys and keys for all the intended devices are still there. Both my Windows and iOS SSH settings have not changed since the update yet neither work. I run the following command in Windows Powershell:

 ssh bcoffy@{server.public.ip} -p {server_ssh_port} -i "C:\Users\{path_to_ssh_key}\SSH\id_rsa0" -vvvv

and get the following:

    OpenSSH_for_Windows_8.6p1, LibreSSL 3.4.3
debug3: Failed to open file:C:/Users/{username}/.ssh/config error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_config error:2
debug2: resolve_canonicalize: hostname {server_public_port} is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> 'C:\\Users\\{username}/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> 'C:\\Users\\{username}/.ssh/known_hosts2'
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug3: ssh_connect_direct: entering
debug1: Connecting to {server.public.ip} [{server.public.ip}] port {server_ssh_port}.

EDIT: it works now, I have no idea what the issue was or how it was resolved but SSH works now, definitely everything in the answer Netbats provided helped me debug it. Thank you for the help!

Score:2
br flag

According to the error message, it looks like the config files are missing on the Windows machine (e.g. this location C:/Users/{username}/.ssh/config) or the read permission is missing.

Sometimes a problem arises when the SSH versions or the server-side and client-side versions of security algorithms mismatch.

But try following tests concerning Ubuntu server. In case all tests are OK,the problem is on Windows side.

Is the SSH daemon service running?

sudo systemctl status sshd.service

You must see a green dot at the left edge and status message Active: active (running).

Is the SSH daemon service listening on the TCP port 22?

sudo netstat -lntp | grep 22

In case the netstat command is not installed, install it:

sudo apt-get update
sudo apt-get install net-tools

Right netstat answer shoud be like this:

tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTENING  1702/sshd: /usr/sbi 
tcp6       0      0 :::22                   :::*                    LISTENING  1702/sshd: /usr/sbi 

Is there a rule on the firewall allowing access to port 22?

sudo iptables-save | grep "\--dport 22"

or second possibility how to find the TCP 22 port rule with less strict filter:

sudo iptables-save | grep "22"

Right command response example is:

-A ufw-user-input -p tcp -m tcp --dport 22 -j ACCEPT

Local SSH selftest on the Ubuntu server

Preparation: Get/verify Ubuntu server own ip address by command ip addr. Let the detected address be 192.168.1.50.

ssh -l your_username 127.0.0.1

and then

ssh -l your_username 192.168.1.50

You should connect your account back to the same server.

The w command shows 1 extra line for each SSH connection to the same server.

w

Verify connectivity to Ubuntu server from another PC

Use another network node to check the situation from outside. Replace the 192.168.1.50 by real address of your server.

ping 192.168.1.50

Verify that port 22 on the Ubuntu server is visible as open from the outside.

telnet 192.168.1.50 22

Right output should be like this:

Trying 192.168.1.50...
Connected to localhost.
Escape character is '^]'.
SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1

You can use other commands instead of the telnet to see via network the port 22 is opened. Responses are included below.

The netcat nc:

nc -4zv 192.168.1.50 22

Response:

Connection to 192.168.1.50 22 port [tcp/ssh] succeeded!

The nmap

sudo nmap -sT -p 22 192.168.1.50

Response:

Starting Nmap 7.80 ( https://nmap.org ) at 2023-01-13 21:27 CET
Nmap scan report for localhost (192.168.1.50)
Host is up (0.000067s latency).

PORT   STATE SERVICE
22/tcp open  ssh

Nmap done: 1 IP address (1 host up) scanned in 0.02 seconds


EDIT

Response to BulletproofCoffy comment

As for SSH keys, I assume you know what you're doing. Server keys must always be present and set correctly, otherwise neither remote nor local SSH connections will work.

From all your description, it looks like a problem of the local firewall on the server, unless there is another one between the two computers. For example, it is enough to have an incorrect rule order (a deny rule above an accept rule) or use the wrong profile (public/office/home) to render the rule allowing SSH is ineffective.

There is an easy test to verify whether the SSH traffic is blocked by a local firewall on the server or not. Turn off the firewall for a while.

sudo ufw disable

Then test from the other computer using the nc to see if the availability of the SSH port has changed.


EDIT 2

I'm not sure I understood the situation correctly. Even directly on the server it is not possible to detect a port as open using the nmap and nc. At the same time, the netstat command shows that the port is active and listening. Is that right?

I recommend this procedure:

Find out which application is listening on the SSH port using netstat. The last column of the statement contains the process number. Search for number in line valid for your SSH TCP port.

sudo netstat -lntp

Use the process number detected in the previous listing (e.g. 5678) to determine which application it belongs to.

ps -ef | grep 5678

Example of output:

root        1167       1  0 09:33 ?        00:00:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups

Find out if it's really your SSH daemon listening on your SSH port. Verify that the SSH port number has disappeared from the list of open ports when you stop the SSH daemon service.

sudo systemctl status sshd.service
sudo systemctl stop   sshd.service
sudo systemctl status sshd.service
sudo netstat -lntp

Now test using another application directly on Ubuntu server (e.g. nc) that the port intended for SSH can be opened and that it is visible over the network from another computer. Netcat start listening on the TCP port 22 in my example in Ubuntu first terminal window.

nc -k -l 22 > /dev/null

!!! Remember the nc listener must be started either with -k parameter or again after each test !!!

Open second terminal on the same Ubuntu server and check if the port is opened:

sudo nestat -lntp
nc -4zv localhost 22

Test the port remotely from another network node. Use some commands described above (telnet, nmap, nc).

If the port is now seen as open locally and remotely over the network, there is probably only one explanation, the faulty is the SSH daemon. Reinstall the SSH daemon.


Edit 3

The tests I designed were to help determine if the problem was with the SSH daemon application or elsewhere. The key is the answer to the question: Is the SSH TCP port visible from another computer as open when there is a different listening application than the SSH daemon in place of the daemon? If the answer is no, it means that the port is blocked somewhere and it has nothing to do with the SSH daemon. You need to figure out what is blocking the connection to the SSH port. Please try to repeat the test with the nc listening application on another completely free and not yet used port.

nc -k -l 777 > /dev/null

Turn off the UFW and try several times in a row to see if the port is visible as open locally and then from other devices on the LAN. Please use more than one device to test the port from the LAN, another PC or a smartphone with the Net Analyzer application is offered next.

If the port is available (open) from the LAN, stop listening the nc and use the proven port number for the SSH daemon. Uncomment the #Port 22 line in the file /etc/ssh/sshd_config, enter the new port number instead of 22, and restart the sshd service.

Verify with the netstat and nc that the new SSH port is open for both internal and external access.

BulletproofCoffy avatar
mq flag
The self-SSH test works as well (tho I get rejected for not having an SSH key on the server but that's expected). Interesting, so `ping` from another PC works fine as mentioned earlier, but running `nc -4zv 192.168.1.26 {ssh_port}` returns `nc: connect to 192.168.1.26 port {ssh_port} (tcp) failed: Connection refused `, while using it on other open ports i have on the server (like https) succeeded in connecting. No idea why, as `ufw` shows the SSH as open, and `lsof` does as well, plus the `iptables-save` command you recommended has the expected output
netbat avatar
br flag
See my answer edit.
BulletproofCoffy avatar
mq flag
So I did ``ufw disable`` and checked the status to be positive it was disabled. Then, using ``nmap`` from a local machine I did ``nmap -sT -p {ssh_port} 192.168.1.26``, which returns a port of ``{ssh_port}/tcp``, a state of ``closed``, and service ``unknown``. ``nc -4zv 192.168.1.26 {ssh_port}``as well returns connection refused. I also checked and netstat shows the port as listening. Anything else to try??
BulletproofCoffy avatar
mq flag
I also tried moving the SSH port, but no luck there either
netbat avatar
br flag
See my edit #2.
BulletproofCoffy avatar
mq flag
Alright, so to answer your question in the beginning, yes, the server can see via ``nc`` and ``netstat`` that the port is open as expected. However, using those commands on a machine connected to the same network, those commands do NOT show the SSH port as open. I went ahead and did your troubleshooting steps anyways, and the server can indeed see that ``nc`` is running on the SSH port when I do the steps you provided, however a different machine on the same network does not see the same thing. I think I will try to reinstall the ssh daemon in hopes that that fixes it.
BulletproofCoffy avatar
mq flag
Alright, I uninstalled ssh ``sudo apt remove openssh-server`` and ``sudo apt purge openssh-server``, then reinstalled: ``sudo apt install ssh``, then started and tried connecting from another machine on the network. It didn't work, same error as per usual: ``ssh: connect to host 192.168.1.26 port {sshport}: Connection refused``
netbat avatar
br flag
See my edit #3.
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.