I have small cloud server on Hetzner that I power on daily (using Hetzner API) from my home server at 3am and then I login there via SSH, do some work then I shut it down (it's all automatic process)
Everything was fine for months, I didn't touch neither my home server or cloud server yet today I received an email with warning
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
It was very suspicious since there was no changes to my servers so I logged in and checked /var/log/auth.log :
Dec 10 03:02:19 htznr useradd[1007]: new group: name=ubuntu, GID=1001
Dec 10 03:02:19 htznr useradd[1007]: new user: name=ubuntu, UID=1001, GID=1001, home=/home/ubuntu, shell=/bin/bash
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to group 'adm'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to group 'dialout'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to group 'cdrom'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to group 'floppy'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to group 'sudo'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to group 'audio'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to group 'dip'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to group 'video'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to group 'plugdev'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to group 'lxd'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to group 'netdev'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to shadow group 'adm'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to shadow group 'dialout'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to shadow group 'cdrom'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to shadow group 'floppy'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to shadow group 'sudo'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to shadow group 'audio'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to shadow group 'dip'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to shadow group 'video'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to shadow group 'plugdev'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to shadow group 'lxd'
Dec 10 03:02:19 htznr useradd[1007]: add 'ubuntu' to shadow group 'netdev'
Dec 10 03:02:19 htznr passwd[1014]: password for 'ubuntu' changed by 'root'
Dec 10 03:02:19 htznr systemd-logind[1057]: New seat seat0.
Dec 10 03:02:19 htznr systemd-logind[1057]: Watching system buttons on /dev/input/event0 (Power Button)
Dec 10 03:02:19 htznr systemd-logind[1057]: Watching system buttons on /dev/input/event1 (AT Translated Set 2 keyboard)
Dec 10 03:02:19 htznr sshd[1094]: Server listening on 0.0.0.0 port 222.
Dec 10 03:02:19 htznr sshd[1094]: Server listening on :: port 222.
03:02:19 is when machine was powered on. As you can see user 'ubuntu' was created with password.
I also realized all keys in /etc/ssh/ has been changed:
-rw------- 1 root root 672 Dec 10 03:02 ssh_host_dsa_key
-rw-r--r-- 1 root root 598 Dec 10 03:02 ssh_host_dsa_key.pub
-rw------- 1 root root 227 Dec 10 03:02 ssh_host_ecdsa_key
-rw-r--r-- 1 root root 170 Dec 10 03:02 ssh_host_ecdsa_key.pub
-rw------- 1 root root 399 Dec 10 03:02 ssh_host_ed25519_key
-rw-r--r-- 1 root root 90 Dec 10 03:02 ssh_host_ed25519_key.pub
-rw------- 1 root root 1.7K Dec 10 03:02 ssh_host_rsa_key
-rw-r--r-- 1 root root 390 Dec 10 03:02 ssh_host_rsa_key.pub
What can be the reason? I login to server using Hetzner's IP address.
I'm worried because I know many bots try to login with common usernames like 'centos', 'ubuntu', 'fedora' etc.
I use Ubuntu and have unattended upgrades enabled.
This is new entry in /etc/shadow
ubuntu:!:18971:0:99999:7:::
