(This question should be tagged with "Tailscale", but that doesn't exist yet - requires knowledge of Tailscale)
I have a standard Ubuntu 20.04 laptop, connecting via ssh to a remote Raspberry Pi (port 22). At the remote location, there is port forwarding on the router (59995->22).
Note: I have removed ~/.ssh/config during testing.
SSH always connects successfully, using:
ssh user@public_ip -i ~/.ssh/id_rsa -p 59995
However, when I use the Tailscale IP, it never connects, just hangs for 120s then times out:
ssh user@tailscale-ip -i ~/.ssh/id_rsa (does not work)
However, if I manually compile OpenSSH v8.2, and use that version, instead of the /usr/bin/ssh binary, then it works perfectly! Even though /usr/bin/ssh is also v8.2
./ssh user@tailscale-ip -i ~/.ssh/id_rsa (this works!)
Recap: the system binary works fine with "normal" ssh, but not with ssh-via-tailscale. However a manually compiled (same) version of Openssh works fine with both regular ssh, and ssh-over-tailsale. Does anyone know what might be causing this?
Additional Notes
I don't think it's any obvious problem with the tailscale network: i.e if I probe via tailscale IP then nmap scan shows correct ports are open, and I can netcat directly to tailscale_ip:22, it is receving and responding.
diff ~/openssh-8.2p1/ssh /usr/bin/ssh shows that the binary's differ. (huge size difference too, about 3MB vs 0.7MB)
/usr/bin/ssh -V : OpenSSH_8.2p1 Ubuntu-4ubuntu0.4, OpenSSL 1.1.1f 31 Mar 2020
~/openssh-8.2p1/ssh -V : OpenSSH_8.2p1, OpenSSL 1.1.1f 31 Mar 2020
Log of /usr/bin/ssh -vvv user@tailscale_ip -i ~/.ssh/id_rsa (ip redacted)
OpenSSH_8.2p1 Ubuntu-4ubuntu0.4, OpenSSL 1.1.1f 31 Mar 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolve_canonicalize: hostname 100.1.1.1 is address
debug2: ssh_connect_direct
debug1: Connecting to 100.1.1.1 [100.1.1.1] port 22.
< 2 minute hang here >
ssh: connect to host 100.1.1.1 port 22: Connection timed out
