Score:0

SSH Keys not propagating correctly from instance metadata to authorized_keys: missing keys, user discrepancy, and duplicate key

gu flag

I want to add eight public keys via instance metadata to avoid adding them manually (i.e.: ssh to VMs, pasting the keys to .ssh/authorized_keys, etc.).
I added the keys in Terraform (four distinct keys for two users) using the metadata attribute of the google_compute_instance:

resource "google_compute_instance" "host" {
  count         = var.number_of_hosts
  
  // vm details...

  metadata = {
    "ssh-keys" = <<EOF
    user1:${file("${path.root}/key1.pub")}
    user1:${file("${path.root}/key2.pub")}
    user1:${file("${path.root}/key3.pub")}
    user1:${file("${path.root}/key4.pub")}
    user2:${file("${path.root}/key1.pub")}
    user2:${file("${path.root}/key2.pub")}
    user2:${file("${path.root}/key3.pub")}
    user2:${file("${path.root}/key4.pub")}
    EOF
  }

I ran terraform apply. I opened the GCP console and clicked on one of the deployed machines. In the "Details" tab, I can see all eight keys in the SSH Keys tab. Now, when I ssh from my local computer, i.e., ssh user2@EXTERNAL_IP (I deliberately started with user2, not user1 - not a typo) and then cat ~/.ssh/authorized_keys, I can only see the following:

user1 : key1
user2 : key4

Thus, I can't ssh to VM2 because the public part of the key pair that USER 2 has access to is not ~/.ssh/authorized_keys even though it is declared in the instance metadata.

On the other hand, when I do user1@EXTERNAL_IP and cat ~/.ssh/authorized_keys, I can see:

user1 : key1
user2 : key4
user1 : key1 (duplicate)

Since the private key that corresponds to user1 : key1 is there, I can ssh to VM2 successfully.

What baffles me:

  1. Why are not all keys declared in the instance metadata added to the authorized_keys?
  2. Why is there a difference in the content of the authorized_keys depending on the user?
  3. Where does the duplicate come from?

Edit - some additional information:

  1. the image used - ubuntu-minimal-2004-focal-v20230427
  2. ssh_config (only uncommented lines):
Include /etc/ssh/ssh_config.d/*.conf
Host *
SendEnv LANG LC_*
HashKnownHosts yes
GSSAPIAuthentication yes
  1. sshd_config (only uncommented lines):
Include /etc/ssh/sshd_config.d/*.conf
PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM yes
X11Forwarding yes
PrintMotd no
AcceptEnv LANG LC_*
Subsystem       sftp    /usr/lib/openssh/sftp-server


Edit 2 - relevant fragment of /var/log/syslog

@John Hanley

You were on the right track. I can see some messages of the following form: google_guest_agent[566]: ERROR non_windows_accounts.go:199 Invalid ssh key entry - unrecognized format: in the syslog. However, looking at the form of my keys, I can't entirely agree with a part of your comment: "some public keys have a username appended which will not work". Keys that I read in Terraform and that later appear in the authorized_keys look as follows:

ssh-rsa AAArsT3 username1

or

ecdsa-sha2-nistp256 AAAAE2= username1

Unfortunately, there is still some wild guessing involved. I can now see four keys in the authorized_keys (3x username1, 1x username2). It is better than 2, but I'm asking myself why the four others are still missing. I did a copy-pasting and only changed the user at the end. Is there an agreement on how the keys passed to the GCP via instance-metadata should look (presence/absence of username, newlines, etc.)?

John Hanley avatar
cn flag
My guess is that the public key files are not formatted correctly for Google Cloud metadata. They should have two fields and look like `ssh-rsa B64PUBLICKEY`. Some public keys have a username appended which will not work.
mångata avatar
gu flag
If we assume that neither `key3` nor `key4` are properly formatted (as neither of them appears in the `authorized_keys`), it still does not explain why the file only contains `user1 : key1` and `user2 : key4`, instead of `user{1,2} : key1` and `user{1,2} : key4`. Since `key1` and `key4` do appear in the file, I presume that their format is valid. The presence of a duplicate in the second case remains a mystery as well.
John Hanley avatar
cn flag
Look at the logs the OS creates (not Cloud Logging). You can see the setup from metadata. Show the relevant log entries in your post.
mångata avatar
gu flag
@JohnHanley I took a look at the `syslog` and edited my post.
John Hanley avatar
cn flag
Remove the trailing username and try again. You are getting an error regarding invalid format ...
mångata avatar
gu flag
The GCP docs is kinda misleading then. Here, in the section "Add SSH keys to instance metadata during VM creation", in the "Terraform" tab (https://cloud.google.com/compute/docs/connect/add-ssh-keys#during-vm-creation), we can see the following: ` metadata = { "ssh-keys" = <<EOT dev:ssh-ed25519 key1 dev test:ssh-ed25519 key2 test EOT } `
mångata avatar
gu flag
Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/146288/discussion-between-mangata-and-john-hanley).
John Hanley avatar
cn flag
There is a difference between updating Compute Engine metadata via the GUI or script and updating `ssh-keys` via Terraform. Use the correct format.
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.