Score:0

Suddenly SSH on all google cloud computing instance is stopped - I can't get it fixed

ky flag

I am passing a very hard time connecting VM instances on google cloud. Everything was fine, suddenly the VM is refused to connect, I can't get it to be fixed.

I have run this command in cloud shell :

gcloud beta compute ssh ceunix-ubuntu-server-instance -- -vvv 

and get the following message:

Welcome to Cloud Shell! Type "help" to get started.
Your Cloud Platform project in this session is set to ceunix-wordpress-316703.
Use “gcloud config set project [PROJECT_ID]” to change to a different project.
ceunixcorporation@cloudshell:~ (ceunix-wordpress-316703)$ gcloud beta compute ssh ceunix-ubuntu-server-instance -- -vvv
Did you mean zone [asia-southeast1-b] for instance:
[ceunix-ubuntu-server-instance] (Y/n)?  n

No zone specified. Using zone [us-central1-a] for instance: [ceunix-ubuntu-server-instance].
Writing 3 keys to /home/ceunixcorporation/.ssh/google_compute_known_hosts
Updating project ssh metadata...⠶Updated [https://www.googleapis.com/compute/beta/projects/ceunix-wordpress-316703].
Updating project ssh metadata...done.
Waiting for SSH key to propagate.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:PIrntDXiIhagDRyAki+F9hgNMxtXDhbAUy2A+VsffSE.
Please contact your system administrator.
Add correct host key in /home/ceunixcorporation/.ssh/google_compute_known_hosts to get rid of this message.
Offending RSA key in /home/ceunixcorporation/.ssh/google_compute_known_hosts:3
  remove with:
  ssh-keygen -f "/home/ceunixcorporation/.ssh/google_compute_known_hosts" -R "compute.906058796356615757"
ECDSA host key for compute.906058796356615757 has changed and you have requested strict checking.
Host key verification failed.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:PIrntDXiIhagDRyAki+F9hgNMxtXDhbAUy2A+VsffSE.
Please contact your system administrator.
Add correct host key in /home/ceunixcorporation/.ssh/google_compute_known_hosts to get rid of this message.
Offending RSA key in /home/ceunixcorporation/.ssh/google_compute_known_hosts:3
  remove with:
  ssh-keygen -f "/home/ceunixcorporation/.ssh/google_compute_known_hosts" -R "compute.906058796356615757"
ECDSA host key for compute.906058796356615757 has changed and you have requested strict checking.
Host key verification failed.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:PIrntDXiIhagDRyAki+F9hgNMxtXDhbAUy2A+VsffSE.
Please contact your system administrator.
Add correct host key in /home/ceunixcorporation/.ssh/google_compute_known_hosts to get rid of this message.
Offending RSA key in /home/ceunixcorporation/.ssh/google_compute_known_hosts:3
  remove with:
  ssh-keygen -f "/home/ceunixcorporation/.ssh/google_compute_known_hosts" -R "compute.906058796356615757"
ECDSA host key for compute.906058796356615757 has changed and you have requested strict checking.
Host key verification failed.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:PIrntDXiIhagDRyAki+F9hgNMxtXDhbAUy2A+VsffSE.
Please contact your system administrator.
Add correct host key in /home/ceunixcorporation/.ssh/google_compute_known_hosts to get rid of this message.
Offending RSA key in /home/ceunixcorporation/.ssh/google_compute_known_hosts:3
  remove with:
  ssh-keygen -f "/home/ceunixcorporation/.ssh/google_compute_known_hosts" -R "compute.906058796356615757"
ECDSA host key for compute.906058796356615757 has changed and you have requested strict checking.
Host key verification failed.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:PIrntDXiIhagDRyAki+F9hgNMxtXDhbAUy2A+VsffSE.
Please contact your system administrator.
Add correct host key in /home/ceunixcorporation/.ssh/google_compute_known_hosts to get rid of this message.
Offending RSA key in /home/ceunixcorporation/.ssh/google_compute_known_hosts:3
  remove with:
  ssh-keygen -f "/home/ceunixcorporation/.ssh/google_compute_known_hosts" -R "compute.906058796356615757"
ECDSA host key for compute.906058796356615757 has changed and you have requested strict checking.
Host key verification failed.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:PIrntDXiIhagDRyAki+F9hgNMxtXDhbAUy2A+VsffSE.
Please contact your system administrator.
Add correct host key in /home/ceunixcorporation/.ssh/google_compute_known_hosts to get rid of this message.
Offending RSA key in /home/ceunixcorporation/.ssh/google_compute_known_hosts:3
  remove with:
  ssh-keygen -f "/home/ceunixcorporation/.ssh/google_compute_known_hosts" -R "compute.906058796356615757"
ECDSA host key for compute.906058796356615757 has changed and you have requested strict checking.
Host key verification failed.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:PIrntDXiIhagDRyAki+F9hgNMxtXDhbAUy2A+VsffSE.
Please contact your system administrator.
Add correct host key in /home/ceunixcorporation/.ssh/google_compute_known_hosts to get rid of this message.
Offending RSA key in /home/ceunixcorporation/.ssh/google_compute_known_hosts:3
  remove with:
  ssh-keygen -f "/home/ceunixcorporation/.ssh/google_compute_known_hosts" -R "compute.906058796356615757"
ECDSA host key for compute.906058796356615757 has changed and you have requested strict checking.
Host key verification failed.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:PIrntDXiIhagDRyAki+F9hgNMxtXDhbAUy2A+VsffSE.
Please contact your system administrator.
Add correct host key in /home/ceunixcorporation/.ssh/google_compute_known_hosts to get rid of this message.
Offending RSA key in /home/ceunixcorporation/.ssh/google_compute_known_hosts:3
  remove with:
  ssh-keygen -f "/home/ceunixcorporation/.ssh/google_compute_known_hosts" -R "compute.906058796356615757"
ECDSA host key for compute.906058796356615757 has changed and you have requested strict checking.
Host key verification failed.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:PIrntDXiIhagDRyAki+F9hgNMxtXDhbAUy2A+VsffSE.
Please contact your system administrator.
Add correct host key in /home/ceunixcorporation/.ssh/google_compute_known_hosts to get rid of this message.
Offending RSA key in /home/ceunixcorporation/.ssh/google_compute_known_hosts:3
  remove with:
  ssh-keygen -f "/home/ceunixcorporation/.ssh/google_compute_known_hosts" -R "compute.906058796356615757"
ECDSA host key for compute.906058796356615757 has changed and you have requested strict checking.
Host key verification failed.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:PIrntDXiIhagDRyAki+F9hgNMxtXDhbAUy2A+VsffSE.
Please contact your system administrator.
Add correct host key in /home/ceunixcorporation/.ssh/google_compute_known_hosts to get rid of this message.
Offending RSA key in /home/ceunixcorporation/.ssh/google_compute_known_hosts:3
  remove with:
  ssh-keygen -f "/home/ceunixcorporation/.ssh/google_compute_known_hosts" -R "compute.906058796356615757"
ECDSA host key for compute.906058796356615757 has changed and you have requested strict checking.
Host key verification failed.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:PIrntDXiIhagDRyAki+F9hgNMxtXDhbAUy2A+VsffSE.
Please contact your system administrator.
Add correct host key in /home/ceunixcorporation/.ssh/google_compute_known_hosts to get rid of this message.
Offending RSA key in /home/ceunixcorporation/.ssh/google_compute_known_hosts:3
  remove with:
  ssh-keygen -f "/home/ceunixcorporation/.ssh/google_compute_known_hosts" -R "compute.906058796356615757"
ECDSA host key for compute.906058796356615757 has changed and you have requested strict checking.
Host key verification failed.
ERROR: (gcloud.beta.compute.ssh) Could not SSH into the instance.  It is possible that your SSH key has not propagated to the instance yet. Try running this command again.  If you still cannot connect, verify that the firewall and instance are set to accept ssh traffic.
ceunixcorporation@cloudshell:~ (ceunix-wordpress-316703)

Note: I have checked the firewall rule, it allows port 22 on all instance networks. Then, I have cleared all public keys from Computing instance > Metadata Section, then adding a new key. it not works.

Only SSH though the browser is working. If I choose to Compute Engine > VM instances > Choose a VM and SSH menu > Open in the browser window using provided private SSH key and then choose my own private ppk key also never let me allow!

Here is the error message > You cannot connect to the VM instance because of an unexpected error. Wait a few moments and then try again.

So, what I can do? I have three Ubuntu instances. All are refusing to connect. please help me.

Note: I have run nmap <my vm's external IP Address> and get this following:

Starting Nmap 7.91 ( https://nmap.org ) at 2021-07-13 08:27 Azores Standard Time

Nmap scan report for 100.142.67.34.bc.googleusercontent.com (34.67.142.100)

The host is up (0.32s latency).

Not shown: 996 filtered ports

PORT     STATE  SERVICE

80/tcp   closed http

443/tcp  closed https

3389/tcp closed ms-wbt-server

8088/tcp open   radan-http



Nmap done: 1 IP address (1 host up) scanned in 17.70 seconds

I also run this command in cloudshell: gcloud compute firewall-rules list and get the following output:

NAME                              NETWORK  DIRECTION  PRIORITY  ALLOW                         DENY  DISABLED
default-allow-http                default  INGRESS    1000      tcp:80                              False
default-allow-https               default  INGRESS    1000      tcp:443                             False
default-allow-icmp                default  INGRESS    65534     icmp                                False
default-allow-internal            default  INGRESS    65534     tcp:0-65535,udp:0-65535,icmp        False
default-allow-rdp                 default  INGRESS    65534     tcp:3389                            False
default-allow-ssh                 default  INGRESS    65534     tcp:22                              False
machinecoderguy-allow-port-7080   default  INGRESS    1000      tcp:7080,udp                        False
machnicecoderguy-allow-port-8088  default  INGRESS    1000      tcp:8088,udp                        False
jabbson avatar
sb flag
Have you tried doing what the error message asks you to do (ie removing the offending keys from the file)?
ky flag
Yes, I have tried even to create a fresh new VM instance, but it seems not working... just fresh!
ky flag
this command > ssh-keygen -f "/home/ceunixcorporation/.ssh/google_compute_known_hosts" -R "compute.906058796356615757" is not working, I have checked and found that google_compute_known_hosts file is missing. any idea to fix this?
Score:1
cn flag

The fingerprint for the VM has changed.

Stop changing items on the VM as that is not your problem unless your VM has been hacked.

The problem is that your desktop has a known_hosts file with an IP address and the host's fingerprint. Since the fingerprint has changed, you are prevented from connecting for security reasons.

If you are sure that your systems have not been hacked, delete the known_hosts file located at ~/.ssh.

Now, the important question is why has the fingerprint changed? That can be caused by a few normal reasons and some that are concerns. Do these VM's have static (not ephemeral) IP addresses? Did you perform a major upgrade to the VM's OS? Are these systems part of a managed instance group and the same IP addresses are being reused for new instances? The investigation will be left to you to perform.

ky flag
There is no file is found named known_hosts! also, I am using static IP, no load balancer or no instance group is present. I just simply go > cd /home/ceunixcorporation/.ssh/ and there is authorized_keys file is presents only!
ky flag
I am in luck: I have run ssh 34.134.51.241 -o "VerifyHostKeyDNS=yes" and it allows me to create a new known_hosts file, but it is asking for the root password, which I simply do not know.
John Hanley avatar
cn flag
@CEUNIXCorporation - The system that you are connecting **from** has the problem with the **known_hosts** file and not the system you are connecting to. Root privilege is not required to modify the known_hosts file in your account.
ky flag
Finally I have solved this problem.
Score:0
ng flag

"Host key verification failed" suggests that the host key of the remote host was changed.

SSH stores the host keys of the remote hosts in ~/.ssh/known_hosts. You can either edit that text file manually and remove the old key, or use

ssh-keygen -R hostname

From man page “-R hostname” Removes all keys belonging to hostname from a known_hosts file. This option is useful to delete hashed hosts .

You could please refer to this case for detailed information.

ky flag
Thanks for your help.
Score:-1
ky flag

Finally, I have fixed my problem. I don't know how it gonna works but It is fixed automatically.

I have made a full security scan and add a new firewall rule in Windows Defender Firewall.

Changed my router. and The problem is fixed.

Also, I have executed some commands to clear all host keys and I have missed known_hosts files in all of my VM, using this command I have managed to create a new one:

ssh <hostname or External Static IP Address> -o "VerifyHostKeyDNS=yes"

Then, run this command to verify fingerprint:

ssh-keyscan <hostname or External Static IP Address> | ssh-keygen -lf -

Then:

systemctl restart ssh

As I can understand, if you try to troubleshoot the steps introduced here: Google Cloud SSH Connectivity Check

but unable to fix your problem, you should make a virus scan and clear all known host lists by the following command:

ssh-keygen -R <hostname or External Static IP Address>

You can also run this command in cloudshell to check if google firewall is not blocking port 22

gcloud compute firewall-rules list

If you don't see port 22 not on the allowed list, it is time to add a new firewall rule to allow port 22.

Goto VPC Network > Firewall and create a new Firewall rule to allow port 22. For your help, you can look into this link: https://cloud.google.com/filestore/docs/configuring-firewall

If you are still not allowed to connect to SSH, try to check your internet provider or router or local firewall rule is not blocking you!

You can also clean all SSH public keys under Compute Engine > Metadata > SSH Keys and add a new public key for authentication.

I hope you will be able to fix your problem like me.

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.