I want to add eight public keys via instance metadata to avoid adding them manually (i.e.: ssh to VMs, pasting the keys to .ssh/authorized_keys, etc.).
I added the keys in Terraform (four distinct keys for two users) using the metadata attribute of the google_compute_instance
:
resource "google_compute_instance" "host" {
count = var.number_of_hosts
// vm details...
metadata = {
"ssh-keys" = <<EOF
user1:${file("${path.root}/key1.pub")}
user1:${file("${path.root}/key2.pub")}
user1:${file("${path.root}/key3.pub")}
user1:${file("${path.root}/key4.pub")}
user2:${file("${path.root}/key1.pub")}
user2:${file("${path.root}/key2.pub")}
user2:${file("${path.root}/key3.pub")}
user2:${file("${path.root}/key4.pub")}
EOF
}
I ran terraform apply
. I opened the GCP console and clicked on one of the deployed machines. In the "Details" tab, I can see all eight keys in the SSH Keys tab.
Now, when I ssh from my local computer, i.e., ssh user2@EXTERNAL_IP
(I deliberately started with user2, not user1 - not a typo) and then cat ~/.ssh/authorized_keys
, I can only see the following:
user1 : key1
user2 : key4
Thus, I can't ssh to VM2 because the public part of the key pair that USER 2 has access to is not ~/.ssh/authorized_keys
even though it is declared in the instance metadata.
On the other hand, when I do user1@EXTERNAL_IP
and cat ~/.ssh/authorized_keys
, I can see:
user1 : key1
user2 : key4
user1 : key1 (duplicate)
Since the private key that corresponds to user1 : key1
is there, I can ssh to VM2 successfully.
What baffles me:
- Why are not all keys declared in the instance metadata added to the
authorized_keys
?
- Why is there a difference in the content of the
authorized_keys
depending on the user?
- Where does the duplicate come from?
Edit - some additional information:
- the image used -
ubuntu-minimal-2004-focal-v20230427
ssh_config
(only uncommented lines):
Include /etc/ssh/ssh_config.d/*.conf
Host *
SendEnv LANG LC_*
HashKnownHosts yes
GSSAPIAuthentication yes
sshd_config
(only uncommented lines):
Include /etc/ssh/sshd_config.d/*.conf
PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM yes
X11Forwarding yes
PrintMotd no
AcceptEnv LANG LC_*
Subsystem sftp /usr/lib/openssh/sftp-server
Edit 2 - relevant fragment of /var/log/syslog
@John Hanley
You were on the right track. I can see some messages of the following form: google_guest_agent[566]: ERROR non_windows_accounts.go:199 Invalid ssh key entry - unrecognized format:
in the syslog
. However, looking at the form of my keys, I can't entirely agree with a part of your comment: "some public keys have a username appended which will not work". Keys that I read in Terraform and that later appear in the authorized_keys
look as follows:
ssh-rsa AAArsT3 username1
or
ecdsa-sha2-nistp256 AAAAE2= username1
Unfortunately, there is still some wild guessing involved. I can now see four keys in the authorized_keys
(3x username1, 1x username2). It is better than 2, but I'm asking myself why the four others are still missing. I did a copy-pasting and only changed the user at the end.
Is there an agreement on how the keys passed to the GCP via instance-metadata should look (presence/absence of username, newlines, etc.)?