Unable to SSH to GCP VM after reboot

The error "Permission denied (publickey)" can occur for several reasons. Follow below 2 workarounds, which may help to resolve your issue:

It's impossible to fetch the actual password from a Linux VM (it's encrypted), but if you want to fetch/reset a password for a local user on a Linux VM, it's possible to set a new password.

Workaround 1 : Resetting a password via a startup script:

This startup script can be used to set a temporary password on instance boot (replace [username] and [password] with the actual username and a temporary password of your choice:

Method-1 :

#! /bin/bash
echo [Username] : [Password] | chpasswd

If you don't have a root password, follow the steps to use a startup script to add it to your instance :

1. Go to the VM instances page in the GCP console.

2. Click on the instance for which you want to add a startup script.

3. Click the Edit button at the top of the page.

4. Click on 'Enable connecting to serial ports'

5. Under Custom metadata, you will find the 'Automation' section.

6. Set below 'Value' into the 'Startup script' box:

Method-1:

#! /bin/bash
echo root:PASSWORD | chpasswd

Method-2 :

#! /bin/bash
usermod -p $(echo "password" | openssl passwd -1 -stdin) root

7. Click Save and then click RESET on the top of the page. You might need to wait for some time for the instance to reboot.

8. Click on 'Connect to serial port' on the page.

9. In the new window, you might need to wait a bit and press 'Enter' on your keyboard once; then, you should see the login prompt.

10. Login using the root and PASSWORD you provided.

Note : Remove the custom metadata you entered once you have regained access to your VM, otherwise it will execute every time your instance reboots. Also, omitting this step can expose the root password in the console.

Workaround 2: Check your issue may be due to the host key missing on the instance, or the permission of the host key has been modified. Follow the below steps to verify this behavior and resolving the issue:

1. If the serial console can connect to the instance, then check the permission for the /etc/ssh/ssh_host_rsa_key file using ls -ltrh command in the /etc/ssh directory.

2. The permission for the file should be -rw-r---r--. If not, chmod 644 ssh_host_rsa_key command can be used to set the permissions.

3. If the serial console connection is not working, check the permissions by mounting the boot disk on the rescue instance.

4. Create a snapshot of the boot disk. See Create a Snapshot

5. Create a rescue instance and attach an additional disk of the snapshot.

6. Run lsblk -f to identify the secondary disk added.

7. Mount the disk using commands

Command to create the directory: mkdir ~/recovery Command to mount the disk: mount -o nouuid <disk_path> ~/recovery

8. Check for the ssh_host_rsa_key and ssh_host_rsa_key.pub files in the /etc/ssh/ directory and verify the permission by referring to step 2.

9. If the files are not present, then generate a key for the instance.Change the root directory by running the command: chroot ~/recovery/

Generate the host keys by running the command:

$ sudo ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key
$ sudo ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key
$ sudo ssh-keygen -t ecdsa -f /etc/ssh/ssh_host_ecdsa_key

10. Unmount the disk using umount ~/recovery from the rescue VM and detach the disk from the instance

11. Attach the disk back to the affected instance and restart the instance.