Join Log in
Score:6
mr.d avatar Server

SSH Agent forwarding not working on Ubuntu 22.04

in flag
mr.d

I am trying to connect to github over ssh on my remote server (Running Ubuntu 22.04).

On my local computer (Running Win 10), I have ~/.ssh/config file with the following:

Host remote
    HostName SERVER_IP
    port 22
    User ubuntu
    ForwardAgent yes

After connecting to the remote server, I can confirm that the ssh agent is working by typing:

echo "$SSH_AUTH_SOCK"

result: /tmp/ssh-XXXXPWEKZo/agent.1073

Also with ssh-add -l I see that the key is added

4096 SHA256:hvGuLtIuwYi2LAnQ0KdC/9IgdBUmlHZer0NyXUXd5aY C:\Users\user/.ssh/id_rsa (RSA)

In /etc/ssh/sshd_config I have allowed Agent forwarding with AllowAgentForwarding yes

But when I try to connect to github ssh -T [email protected] -vvv (the key is added to github settings) I get the following:

OpenSSH_8.9p1 Ubuntu-3, OpenSSL 3.0.2 15 Mar 2022
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/ubuntu/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/ubuntu/.ssh/known_hosts2'
debug2: resolving "github.com" port 22
debug3: resolve_host: lookup github.com:22
debug3: ssh_connect_direct: entering
debug1: Connecting to github.com [140.82.121.3] port 22.
debug3: set_sock_tos: set socket 3 IP_TOS 0x10
debug1: Connection established.
debug1: identity file /home/ubuntu/.ssh/id_rsa type -1
debug1: identity file /home/ubuntu/.ssh/id_rsa-cert type -1
debug1: identity file /home/ubuntu/.ssh/id_ecdsa type -1
debug1: identity file /home/ubuntu/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/ubuntu/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/ubuntu/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/ubuntu/.ssh/id_ed25519 type -1
debug1: identity file /home/ubuntu/.ssh/id_ed25519-cert type -1
debug1: identity file /home/ubuntu/.ssh/id_ed25519_sk type -1
debug1: identity file /home/ubuntu/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/ubuntu/.ssh/id_xmss type -1
debug1: identity file /home/ubuntu/.ssh/id_xmss-cert type -1
debug1: identity file /home/ubuntu/.ssh/id_dsa type -1
debug1: identity file /home/ubuntu/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3
debug1: Remote protocol version 2.0, remote software version babeld-2f5f2727
debug1: compat_banner: no match: babeld-2f5f2727
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to github.com:22 as 'git'
debug3: record_hostkey: found key type ED25519 in file /home/ubuntu/.ssh/known_hosts:1
debug3: load_hostkeys_file: loaded 1 keys from github.com
debug1: load_hostkeys: fopen /home/ubuntu/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug3: order_hostkeyalgs: have matching best-preference key type [email protected], using HostkeyAlgorithms verbatim
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,[email protected],diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
debug2: host key algorithms: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,[email protected],zlib
debug2: compression stoc: none,[email protected],zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256
debug2: host key algorithms: ssh-ed25519,ecdsa-sha2-nistp256,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: [email protected],[email protected],[email protected],aes256-ctr,aes192-ctr,aes128-ctr
debug2: ciphers stoc: [email protected],[email protected],[email protected],aes256-ctr,aes192-ctr,aes128-ctr
debug2: MACs ctos: [email protected],[email protected],hmac-sha2-512,hmac-sha2-256
debug2: MACs stoc: [email protected],[email protected],hmac-sha2-512,hmac-sha2-256
debug2: compression ctos: none,[email protected],zlib
debug2: compression stoc: none,[email protected],zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU
debug3: record_hostkey: found key type ED25519 in file /home/ubuntu/.ssh/known_hosts:1
debug3: load_hostkeys_file: loaded 1 keys from github.com
debug1: load_hostkeys: fopen /home/ubuntu/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'github.com' is known and matches the ED25519 host key.
debug1: Found key in /home/ubuntu/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: ssh_set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: ssh_set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
channel 1: chan_shutdown_read: shutdown() failed for fd 7 [i0 o0]: Not a socket
debug2: get_agent_identities: ssh_agent_bind_hostkey: communication with agent failed
debug1: get_agent_identities: ssh_fetch_identitylist: communication with agent failed
debug1: Will attempt key: /home/ubuntu/.ssh/id_rsa
debug1: Will attempt key: /home/ubuntu/.ssh/id_ecdsa
debug1: Will attempt key: /home/ubuntu/.ssh/id_ecdsa_sk
debug1: Will attempt key: /home/ubuntu/.ssh/id_ed25519
debug1: Will attempt key: /home/ubuntu/.ssh/id_ed25519_sk
debug1: Will attempt key: /home/ubuntu/.ssh/id_xmss
debug1: Will attempt key: /home/ubuntu/.ssh/id_dsa
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ssh-ed25519,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256,rsa-sha2-512,rsa-sha2-256,ssh-rsa>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/ubuntu/.ssh/id_rsa
debug3: no such identity: /home/ubuntu/.ssh/id_rsa: No such file or directory
debug1: Trying private key: /home/ubuntu/.ssh/id_ecdsa
debug3: no such identity: /home/ubuntu/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/ubuntu/.ssh/id_ecdsa_sk
debug3: no such identity: /home/ubuntu/.ssh/id_ecdsa_sk: No such file or directory
debug1: Trying private key: /home/ubuntu/.ssh/id_ed25519
debug3: no such identity: /home/ubuntu/.ssh/id_ed25519: No such file or directory
debug1: Trying private key: /home/ubuntu/.ssh/id_ed25519_sk
debug3: no such identity: /home/ubuntu/.ssh/id_ed25519_sk: No such file or directory
debug1: Trying private key: /home/ubuntu/.ssh/id_xmss
debug3: no such identity: /home/ubuntu/.ssh/id_xmss: No such file or directory
debug1: Trying private key: /home/ubuntu/.ssh/id_dsa
debug3: no such identity: /home/ubuntu/.ssh/id_dsa: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
[email protected]: Permission denied (publickey).

I don't have such problems on other of my servers that are running ubuntu 20.04 and 18.04.

4204
2 + 4
ssh
Ginnungagap avatar
gu flag
Ginnungagap
If you see the key with `ssh-add -l` on the remote server then your issue is not with agent forwarding. However your `ssh` logs show that it fails to communicate with the agent, have you run all these commands in order in the same terminal and shell on the remote machine? Please post the full `ssh -vvv` to have more debugging information.
0
Reply
mr.d avatar
in flag
mr.d
Hello, I have edited the question with the full -vvv output. On the remote server if I use the same key, but without forwarding (I manually upload it to the server) everything works Ok.
0
Reply
mr.d avatar
in flag
mr.d
I just noticed while testing with the same key but without forwarding, that even if I add the key to the agent on the remote server, when connecting to github it says `debug1: get_agent_identities: ssh_fetch_identitylist: communication with agent failed` and it asks for the key password, so the problem should be with the agent on the remote server. It's a clean install of Ubuntu 22.04 provided by ovh.com
0
Reply
Jakob avatar
is flag
Jakob
I have the same issue, and it seems to be connected to the ssh-client on Windows. When I try a forward with an agent running on a linux, it works fine.
1
Reply
Score:6
kaytwo avatar Server
cn flag
kaytwo

This answer helped me solve the same problem: there's some incompatibility between the ssh client shipped with Windows and the server on Ubuntu 22.04.

tldr: Installing the most recent release of the windows ssh client fixed the issue for me.

+ 2
ph flag
Dave M
While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - [From Review](/review/late-answers/536604)
0
Reply
kbridge4096 avatar
fj flag
kbridge4096
Note: do not just download the MSI file and click it, wishing everything will be done. Instead, RTFM here: https://github.com/PowerShell/Win32-OpenSSH/wiki/Install-Win32-OpenSSH-Using-MSI
0
Reply
Score:1
PFudd avatar Server
cn flag
PFudd

The answer from @kaytwo is right; updating the Windows ssh client is the cure.

Here's what strace -tt -f -v -s1024 ssh $hostname revealed:

2636504 12:05:25.582068 write(3, "\0\0\0\320", 4) = 4
2636504 12:05:25.582159 write(3, "\33\0\0\0\[email protected]\0\0\0003\0\0\0\vssh-ed25519\0\0\0.....binary-data-omitted.....", 208) = 208
2636504 12:05:25.582210 read(3, "", 4)  = 0
2636504 12:05:25.875257 getpid()        = 2636504
2636504 12:05:25.875343 write(3, "\0\0\0\1", 4) = 4
2636504 12:05:25.875532 write(3, "\v", 1) = 1
2636504 12:05:25.875691 read(3, "", 4)  = 0
2636504 12:05:25.875745 getpid()        = 2636504
2636504 12:05:25.875799 write(2, "debug1: get_agent_identities: ssh_fetch_identitylist: communication with agent failed\r\n", 87) = 87

Compare with strace -tt -f -v -s1024 ssh-add -l:

2648504 13:00:38.631807 connect(3, {sa_family=AF_UNIX, sun_path="/tmp/ssh-XXXXuWGsZN/agent.2628678"}, 110) = 0
2648504 13:00:38.632285 write(3, "\0\0\0\1", 4) = 4
2648504 13:00:38.632386 write(3, "\v", 1) = 1
2648504 13:00:38.632464 read(3, "\0\0\1\255", 4) = 4
2648504 13:00:38.920854 read(3, "\f\0\0\0\1\0\0\1\227\0\0\0\7ssh-rsa\0\0\0\3\1\0\1\0\0\1...binary-data-omitted.....", 429) = 429

Basically, sending a "[email protected]" request causes the Windows ssh-agent to stop talking; the normal "list all keys" command doesn't return anything when it's in that state. Fortunately, the state is cleared when the connection is closed, so you can try again without having to restart anything.

It looks like this configuration option (in man ssh_config) is related:

 PubkeyAuthentication
        Specifies whether to try public key authentication.  The argument to this keyword must
        be yes (the default), no, unbound or host-bound.  The final two options enable public
        key authentication while respectively disabling or enabling the OpenSSH host-bound au‐
        thentication protocol extension required for restricted ssh-agent(1) forwarding.

Note: While that paragraph says ssh -o PubkeyAuthentication=unbound ..... should disable this protocol extension, strace showed that it was still being sent, and the Windows ssh agent won't work.

This page on SSH Agent Restriction goes into total detail about the subject, from the reasons that inspired the feature to command-line options needed to use it and the byte sequences sent in the SSH Agent protocol extension.

Good luck!

+ 0
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.