I deployed two machines to GCP via Terraform. Let's call them control-host and target-host. I want to manage the target-host via Ansible installed on the control-host.
Unfortunately, I keep getting the following error no matter what I do:
10.128.100.3 | UNREACHABLE! => {
"changed": false,
"msg": "Failed to connect to the host via ssh: [email protected]: Permission denied (publickey).",
"unreachable": true
}
The problem perplexes me because it is possible to ssh from control-host to target-host without any problem. I would assume that if "raw" ssh is possible, so is using it via Ansible.
Here's the list of things I did and tried:
- Install Ansible on the control-host.
- Generate a key pair on the control host. Copy the public key.
- SSH on the target host. Open
./.ssh/authorized_keys. Paste the public key.
- Execute
ansible all -vvv -m ping. I got the error mentioned above.
- Open
/etc/ansible/ansible.cfg. Add the following:
[defaults]
remote_user = gcp_user
host_key_checking = False
ansible_ssh_common_args='-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null'
private_key_file = /home/gcp_user/.ssh/t_k
(t_k is the name of the key)
- Run
ansible all -vvv -m ping again. Same error.
- Run
ansible all -vvv -m ping --key-file=/home/gcp_user/.ssh/t_k. Same error.
- Open
/etc/ansible/hosts. Add the following:
10.128.100.3 ansible_ssh_private_key_file=/home/gcp_user/.ssh/t_k
Same story.
SSH logs on the server side say the following:
Connection closed by authenticating user gcp_user 10.128.100.2 port 34470 [preauth]
I followed the recommendations from the following threads and none of them helped:
- https://stackoverflow.com/questions/64681944/create-and-setup-gcp-vms-with-ansible-ssh-permission-denied-publickey
- https://stackoverflow.com/questions/55897136/ansible-failed-to-connect-to-the-host-via-ssh-permission-denied-publickey
- https://stackoverflow.com/questions/57424995/ansible-remote-user-root-ssh-permission-denied-publickey
- https://stackoverflow.com/questions/33280244/ssh-error-permission-denied-publickey-password-in-ansible
Here's the output of the command ansible all -vvv -m ping -e 'ansible_ssh_extra_args="-vvv"':
1st part:
ansible [core 2.12.10]
config file = /etc/ansible/ansible.cfg
configured module search path = ['/home/gcp_user/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python3/dist-packages/ansible
ansible collection location = /home/gcp_user/.ansible/collections:/usr/share/ansible/collections
executable location = /usr/bin/ansible
python version = 3.8.10 (default, Mar 13 2023, 10:26:41) [GCC 9.4.0]
jinja version = 2.10.1
libyaml = True
Using /etc/ansible/ansible.cfg as config file
host_list declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
script declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
auto declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Parsed /etc/ansible/hosts inventory source with ini plugin
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.
META: ran handlers
<10.128.100.3> ESTABLISH SSH CONNECTION FOR USER: gcp_user
<10.128.100.3> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/gcp_user/.ssh/t_k"' -o KbdInteractiveAuthentication=no
-o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="gcp_user"' -o ConnectTimeout=10 -vvv -o
'ControlPath="/home/gcp_user/.ansible/cp/becfdd0705"' 10.128.100.3 '/bin/sh -c '"'"'echo ~gcp_user && sleep 0'"'"''
<10.128.100.3> (255, b'', b'OpenSSH_8.2p1 Ubuntu-4ubuntu0.5, OpenSSL 1.1.1f 31 Mar 2020\r\ndebug1:
Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 19:
Including file /etc/ssh/ssh_config.d/50-cloudimg-settings.conf depth 0\r\ndebug1:
Reading configuration data /etc/ssh/ssh_config.d/50-cloudimg-settings.conf\r\ndebug1:
/etc/ssh/ssh_config line 21: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 10.128.100.3 is address\r\ndebug1:
auto-mux: Trying existing master\r\ndebug1: Control socket "/home/gcp_user/.ansible/cp/becfdd0705" does not exist\r\ndebug2:
ssh_connect_direct\r\ndebug1: Connecting to 10.128.100.3 [10.128.100.3] port 22.\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug1:
fd 3 clearing O_NONBLOCK\r\ndebug1: Connection established.\r\n
debug3: timeout: 9998 ms remain after connect\r\n
debug1: identity file /home/gcp_user/.ssh/t_k type 0\r\n
debug1: identity file /home/gcp_user/.ssh/t_k-cert type -1\r\n
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5\r\n
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.5\r\n
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.5 pat OpenSSH* compat 0x04000000\r\n
debug2: fd 3 setting O_NONBLOCK\r\ndebug1: Authenticating to 10.128.100.3:22 as \'gcp_user\'\r\n
debug3: hostkeys_foreach: reading file "/home/gcp_user/.ssh/known_hosts"\r\ndebug3: record_hostkey:
found key type RSA in file /home/gcp_user/.ssh/known_hosts:1\r\ndebug3: load_hostkeys: loaded 1 keys from 10.128.100.3\r\ndebug3
: order_hostkeyalgs: prefer hostkeyalgs: send packet: type 20\r\ndebug1: SSH2_MSG_KEXINIT sent\r\ndebug3: receive packet: type 20\r\ndebug1:
SSH2_MSG_KEXINIT received\r\ndebug2: local client KEXINIT proposal\r\ndebug2:
KEX algorithms: MACs stoc: compression ctos: [email protected],zlib,none\r\ndebug2:
compression stoc: [email protected],zlib,none\r\ndebug2: languages ctos: \r\ndebug2: languages stoc: \r\ndebug2:
first_kex_follows 0 \r\ndebug2: reserved 0 \r\ndebug2: peer server KEXINIT proposal\r\n
debug2: KEX algorithms: host key algorithms: rsa-sha2-512,: ciphers ctos: [email protected]\r\ndebug3: send packet: type 30\r\n
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY\r\ndebug3: receive packet: type 31\r\ndebug1: Server host key:
ssh-rsa SHA256:***/***\r\ndebug3: hostkeys_foreach:
reading file "/home/gcp_user/.ssh/known_hosts"\r\ndebug3: record_hostkey: found key type RSA in file /home/gcp_user/.ssh/known_hosts:1\r\n
debug3: load_hostkeys: loaded 1 keys from 10.128.100.3\r\ndebug1: Host \'10.128.100.3\' is known and matches the RSA host key.\r\n
debug1: Found key in /home/gcp_user/.ssh/known_hosts:1\r\ndebug3:
send packet: type 21\r\ndebug2: set_newkeys: mode 1\r\ndebug1: rekey out after 134217728 blocks\r\ndebug1: SSH2_MSG_NEWKEYS sent\r\ndebug1:
expecting SSH2_MSG_NEWKEYS\r\ndebug3: receive packet: type 21\r\ndebug1: SSH2_MSG_NEWKEYS received\r\ndebug2: set_newkeys: mode 0\r\ndebug1:
rekey in after 134217728 blocks\r\ndebug1: Will attempt key: normal ECDSA SHA256:*** agent\r\ndebug1:
Will attempt key: /home/gcp_user/.ssh/t_k RSA SHA256:***/*** explicit\r\ndebug2: pubkey_prepare: done\r\n
debug3: send packet: type 5\r\ndebug3: receive packet: type 7\r\ndebug1: SSH2_MSG_EXT_INFO received\r\ndebug1:
kex_input_ext_info: server-sig-algs=<s\ndebug3: receive packet: type 6\r\ndebug2: service_accept: ssh-userauth\r\ndebug1:
SSH2_MSG_SERVICE_ACCEPT received\r\ndebug3: send packet: type 50\r\ndebug3: receive packet: type 51\r\
ndebug1: Authentications that can continue: publickey\r\ndebug3: start over, passed a different list publickey\r\ndebug3:
authmethod_lookup publickey\r\ndebug3: remaining preferred: ,gssapi-keyex,hostbased,publickey\r\ndebug3: authmethod_is_enabled publickey\r\n
debug1: Next authentication method: publickey\r\ndebug1: Offering public key:
normal ECDSA SHA256:*** agent\r\n
debug3: send packet: type 50\r\ndebug2: we sent a publickey packet, wait for reply\r\ndebug3: receive packet: type 51\r\ndebug1:
Authentications that can continue: publickey\r\ndebug1: Offering public key: /home/gcp_user/.ssh/t_k RSA SHA256:***/*** explicit\r\n
debug3: send packet: type 50\r\ndebug2: we sent a publickey packet, wait for reply\r\ndebug3: receive packet: type 51\r\n
debug1: Authentications that can continue: publickey\r\ndebug2: we did not send a packet, disable method\r\n
debug1: No more authentication methods to try.\r\[email protected]: Permission denied (publickey).\r\n')
2nd part:
10.128.100.3 | UNREACHABLE! => {
"changed": false,
"msg": "Failed to connect to the host via ssh: OpenSSH_8.2p1 Ubuntu-4ubuntu0.5, OpenSSL 1.1.1f 31 Mar 2020\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3:
/etc/ssh/ssh_config line 19: Including file /etc/ssh/ssh_config.d/50-cloudimg-settings.conf depth 0\r\ndebug1:
Reading configuration data /etc/ssh/ssh_config.d/50-cloudimg-settings.conf\r\ndebug1: /etc/ssh/ssh_config line 21:
Applying options for *\r\n
debug2: resolve_canonicalize: hostname 10.128.100.3 is address \r\n
debug1: auto-mux: Trying existing master\r\ndebug1:
Control socket \"/home/gcp_user/.ansible/cp/becfdd0705\" does not exist\r\n
debug2: ssh_connect_direct\r\n
debug1: Connecting to 10.128.100.3 [10.128.100.3] port 22.\r\n
debug2: fd 3 setting O_NONBLOCK\r\n
debug1: fd 3 clearing O_NONBLOCK\r\n
debug1: Connection established.\r\n
debug3: timeout: 9998 ms remain after connect\r\n
debug1: identity file /home/gcp_user/.ssh/t_k type 0\r\n
debug1: identity file /home/gcp_user/.ssh/t_k-cert type -1\r\n
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5\r\n
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.5\r\n
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.5 pat OpenSSH* compat 0x04000000\r\n
debug2: fd 3 setting O_NONBLOCK\r\n
debug1: Authenticating to 10.128.100.3:22 as 'gcp_user'\r\n
debug3: hostkeys_foreach: reading file \"/home/gcp_user/.ssh/known_hosts\"\r\n
debug3: record_hostkey: found key type RSA in file /home/gcp_user/.ssh/known_hosts:1\r\n
debug3: load_hostkeys: loaded 1 keys from 10.128.100.3\r\n
debug3: order_hostkeyalgs: prefer hostkeyalgs:[email protected],[email protected],[email protected],rsa-sha2-512,rsa-sha2-256,ssh-rsa\r\n
debug3: send packet: type 20\r\n
debug1: SSH2_MSG_KEXINIT sent\r\n
debug3: receive packet: type 20\r\n
debug1: SSH2_MSG_KEXINIT received\r\n
debug2: local client KEXINIT proposal\r\n
debug2: KEX algorithms: [email protected],[email protected],
[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1\r\n
debug2: compression ctos: [email protected],zlib,none\r\n
debug2: compression stoc: [email protected],zlib,none\r\n
debug2: languages ctos: \r\ndebug2: languages stoc: \r\n
debug2: first_kex_follows 0 \r\ndebug2: reserved 0 \r\ndebug2: peer server-sha2-512,hmac-sha1\r\ndebug2: compression ctos: none,[email protected]\r\n
debug2: compression stoc: none,[email protected]\r\ndebug2: languages ctos: \r\ndebug2: languages stoc: \r\ndebug2: first_kex_follows 0 \r\nd
ebug2: reserved 0 \r\ndebug1: kex: algorithm: curve25519-sha256\r\ndebug1: kex: host key algorithm: rsa-sha2-512\r\ndebug1: kex: server->client cipher:
[email protected] MAC: <implicit> compression: [email protected]\r\ndebug1: kex: client->server cipher: [email protected] MAC:
<implicit> compression: [email protected]\r\ndebug3: send packet: type 30\r\ndebug1: expecting SSH2_MSG_KEX_ECDH_REPLY\r\ndebug3: receive packet: type 31\r\ndebug1:
Server host key: ssh-rsa SHA256:***\r\ndebug3: hostkeys_foreach: reading file
\"/home/gcp_user/.ssh/known_hosts\"\r\ndebug3: record_hostkey: found key type RSA in file /home/gcp_user/.ssh/known_hosts:1\r\ndebug3:
load_hostkeys: loaded 1 keys from 10.128.100.3\r\ndebug1: Host '10.128.100.3' is known and matches the RSA host key.\r\ndebug1: Found key in /home/gcp_user/.ssh/known_hosts:1\r\n
debug3: send packet: type 21\r\ndebug2: set_newkeys: mode 1\r\ndebug1: rekey out after 134217728 blocks\r\ndebug1: SSH2_MSG_NEWKEYS sent\r\ndebug1: expecting SSH2_MSG_NEWKEYS\r\ndebug3:
receive packet: type 21\r\ndebug1: SSH2_MSG_NEWKEYS received\r\ndebug2: set_newkeys: mode 0\r\ndebug1: rekey in after 134217728 blocks\r\ndebug1:
Will attempt key: normal ECDSA SHA256:*** agent\r\ndebug1: Will attempt key: /home/gcp_user/.ssh/t_k RSA SHA256:***/*** explicit\r\n
debug2: pubkey_prepare: done\r\ndebug3: send packet: type 5\r\ndebug3: receive packet: type 7\r\ndebug1: SSH2_MSG_EXT_INFO received\r\ndebug1: kex_input_ext_info: server-sig-a
debug2: service_accept: ssh-userauth\r\ndebug1: SSH2_MSG_SERVICE_ACCEPT received\r\ndebug3: send packet: type 50\r\ndebug3: receive packet: type 51\r\n
debug1: Authentications that can continue: publickey\r\ndebug3: start over, passed a different list publickey\r\ndebug3: preferred gssapi-with-mic,gssapi-keyex,hostbased,publickey\r\n
debug3: authmethod_lookup publickey\r\ndebug3: remaining preferred: ,gssapi-keyex,hostbased,publickey\r\ndebug3: authmethod_is_enabled publickey\r\n
debug1: Next authentication method: publickey\r\ndebug1: Offering public key: normal ECDSA SHA256:*** agent\r\n
debug3: send packet: type 50\r\ndebug2: we sent a publickey packet, wait for reply\r\ndebug3: receive packet: type 51\r\ndebug1: Authentications that can continue: publickey\r\n
debug1: Offering public key: /home/gcp_user/.ssh/t_k RSA SHA256:***/*** explicit\r\ndebug3: send packet: type 50\r\n
debug2: we sent a publickey packet, wait for reply\r\ndebug3: receive packet: type 51\r\ndebug1: Authentications that can continue: publickey\r\n
debug2: we did not send a packet, disable method\r\ndebug1: No more authentication methods to try.\r\[email protected]: Permission denied (publickey).",
"unreachable": true
}
