For some reason I'm having problems connecting via SSH to my Ubuntu 22.04.2 LTS server through one of the two physical NICs in the machine with the following response:
kex_exchange_identification: read: Connection timed out
banner exchange: Connection to 10.10.100.Y port 22: Connection timed out
I can connect to that same server on the other interface without a problem.
The working interface (eno1
) is set up with the following IP address details so that it's temporarily available from my main internal network:
IP: 192.168.1.X
NM: 255.255.255.0
GW: 192.168.1.253
BC: 192.168.1.255
The interface that's not allowing me to connect (enp2s0
) is set up for communicating on a separate network:
IP: 10.10.100.Y
NM: 255.255.255.192
GW: 10.10.100.253
BC: 10.10.100.255
I can PING
the 10.10.100.Y
interface with no indication of basic connectivity issues. I'm coming from the same network as the eno1
interface (192.168.1.X
) and, as I said, I can SSH to that interface without a problem.
Pinging 10.10.100.Y with 32 bytes of data:
Reply from 10.10.100.Y: bytes=32 time<1ms TTL=64
Reply from 10.10.100.Y: bytes=32 time<1ms TTL=64
Reply from 10.10.100.Y: bytes=32 time<1ms TTL=64
Reply from 10.10.100.Y: bytes=32 time<1ms TTL=64
Ping statistics for 10.10.100.Y:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms
Just to throw another wrinkle into the mix, I also have an Ubuntu VM (20.04.6 LTS) running on an antsle Nano (basically a Rasperry Pi device) that's on the same network as the enp2s0
interface described above to which I can SSH without error. Its IP address details for that interface are:
IP: 10.10.100.Z
NM: 255.255.255.192
GW: 10.10.100.253
BC: 10.10.100.255
I've looked at several questions and answers around SE and other sites and checked a whole host of possible causes, but still can't seem to get the SSH connection to work on the enp2s0
interface. Here are just a few things I've looked at on the "problem" server:
/etc/hosts.allow
sendmail: all
# /etc/hosts.allow: list of hosts that are allowed to access the system.
# See the manual pages hosts_access(5) and hosts_options(5).
#
# Example: ALL: LOCAL @some_netgroup
# ALL: .foobar.edu EXCEPT terminalserver.foobar.edu
#
# If you're going to protect the portmapper use the name "rpcbind" for the
# daemon name. See rpcbind(8) and rpc.mountd(8) for further information.
#
sshd: ALL: aclexec /usr/local/bin/sshfilter.sh %a
This is using a GeoIP script (sshfilter.sh
) to block incoming SSH connections from "dangerous" countries (both servers are using the same script).
/etc/hosts.deny
# /etc/hosts.deny: list of hosts that are _not_ allowed to access the system.
# See the manual pages hosts_access(5) and hosts_options(5).
#
# Example: ALL: some.host.name, .some.domain
# ALL EXCEPT in.fingerd: other.host.name, .other.domain
#
# If you're going to protect the portmapper use the name "rpcbind" for the
# daemon name. See rpcbind(8) and rpc.mountd(8) for further information.
#
# The PARANOID wildcard matches any host whose name does not match its
# address.
#
# You may wish to enable this to ensure any programs that don't
# validate looked up hostnames still leave understandable logs. In past
# versions of Debian this has been the default.
# ALL: PARANOID
sshd: ALL
Using the sshfilter.sh
script in the hosts.allow
(supposedly) makes this work the way I'm expecting.
/etc/ssh/sshd_config
Include /etc/ssh/sshd_config.d/*.conf
Protocol 2
ListenAddress 192.168.1.X
ListenAddress 10.10.100.Y
LogLevel VERBOSE
LoginGraceTime 30
PermitRootLogin yes
PubkeyAuthentication yes
AuthenticationMethods publickey
HostbasedAuthentication no
IgnoreRhosts yes
PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM yes
TCPKeepAlive yes
#AllowTcpForwarding no
X11Forwarding yes
PrintMotd no
PrintLastLog no
ClientAliveInterval 150
ClientAliveCountMax 0
AcceptEnv LANG LC_*
Subsystem sftp /usr/lib/openssh/sftp-server
AllowUsers myuser anotheruser
I added the ListenAddress
lines (and restarted the sshd
service) but that didn't help.
ufw status
Status: inactive
/etc/netplan/netcfg.yaml
# This is the network config written by 'subiquity'
network:
ethernets:
enp2s0:
dhcp4: false
addresses:
- 10.10.100.Y/26
nameservers:
addresses:
- 68.105.29.16
- 68.105.28.16
- 1.1.1.1
- 9.9.9.9
- 8.8.8.8
- 10.10.100.253
routes:
- to: default
via: 10.10.100.253
metric: 100
eno1:
dhcp4: false
addresses:
- 192.168.1.X/24
nameservers:
addresses:
- 68.105.29.16
- 68.105.28.16
- 1.1.1.1
- 9.9.9.9
- 8.8.8.8
- 192.168.1.253
routes:
- to: default
via: 192.168.1.253
metric: 200
renderer: NetworkManager
version: 2
/etc/nftables.conf
flush ruleset
# `inet` applies to both IPv4 and IPv6.
table inet filter {
chain input {
type filter hook input priority 0;
# accept any localhost traffic
iif lo accept
# no ping floods:
ip protocol icmp icmp type echo-request limit rate over 10/second burst 4 packets drop
ip6 nexthdr icmpv6 icmpv6 type echo-request limit rate over 10/second burst 4 packets drop
# accept traffic originated from us
ct state established,related accept
# accept ICMP & IGMP
ip6 nexthdr icmpv6 icmpv6 type { echo-request, destination-unreachable, packet-too-big, time-exceeded, parameter-problem, mld-listener-query, mld-listener-report, mld-listener-reduction, nd-router-sol> ip protocol icmp icmp type { echo-request, destination-unreachable, router-solicitation, router-advertisement, time-exceeded, parameter-problem } accept
ip protocol igmp accept
# ssh
tcp dport 22 accept
# http, https
tcp dport 80 accept
tcp dport 443 accept
# smtp, submission, smtps
tcp dport 25 accept
tcp dport 587 accept
tcp dport 465 accept
# pop3, pop3s
tcp dport 110 accept
tcp dport 995 accept
# imap, imaps
tcp dport 143 accept
tcp dport 993 accept
# count and drop any other traffic
counter drop
}
chain output {
type filter hook output priority 0;
policy accept;
}
chain forward {
type filter hook forward priority 0;
policy drop;
}
}
iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy DROP)
target prot opt source destination
DOCKER-USER all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain DOCKER (1 references)
target prot opt source destination
ACCEPT tcp -- anywhere 172.D.0.2 tcp dpt:5000
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
RETURN all -- anywhere anywhere
Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target prot opt source destination
DROP all -- anywhere anywhere
RETURN all -- anywhere anywhere
Chain DOCKER-USER (1 references)
target prot opt source destination
RETURN all -- anywhere anywhere
route -n
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.10.100.253 0.0.0.0 UG 100 0 0 enp2s0
0.0.0.0 192.168.1.253 0.0.0.0 UG 200 0 0 eno1
172.D.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
10.10.100.192 0.0.0.0 255.255.255.192 U 101 0 0 enp2s0
192.168.1.0 0.0.0.0 255.255.255.0 U 100 0 0 eno1
netstat -tulpn | grep :22
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 192.168.1.X:22 0.0.0.0:* LISTEN -
tcp 0 0 10.10.100.Y:22 0.0.0.0:* LISTEN -
ip r
default via 10.10.100.253 dev enp2s0 proto static metric 100
default via 192.168.1.253 dev eno1 proto static metric 200
10.10.100.192/26 dev enp2s0 proto kernel scope link src 10.10.100.Y metric 101
172.D.0.0/16 dev docker0 proto kernel scope link src 172.D.0.1
192.168.1.0/24 dev eno1 proto kernel scope link src 192.168.1.X metric 100
ifconfig
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.D.0.1 netmask 255.255.0.0 broadcast 172.D.255.255
inet6 fe80::42:d3ff:fe5e:1111 prefixlen 64 scopeid 0x20<link>
ether 02:42:d3:5e:46:11 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 2 overruns 0 frame 0
TX packets 78 bytes 11868 (11.8 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.1.X netmask 255.255.255.0 broadcast 192.168.1.255
inet6 fe80::1a03:73ff:feb8:1111 prefixlen 64 scopeid 0x20<link>
ether 18:03:73:b8:82:11 txqueuelen 1000 (Ethernet)
RX packets 203434 bytes 14342810 (14.3 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 6615 bytes 1573720 (1.5 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 20 memory 0xe1b00000-e1b20000
enp2s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.10.100.Y netmask 255.255.255.192 broadcast 10.10.100.255
inet6 fe80::6a05:caff:fec2:1111 prefixlen 64 scopeid 0x20<link>
ether 68:05:ca:c2:b0:11 txqueuelen 1000 (Ethernet)
RX packets 29985 bytes 11846128 (11.8 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 18305 bytes 2268768 (2.2 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 18 memory 0xe10c0000-e10e0000
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 891535 bytes 248668330 (248.6 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 891535 bytes 248668330 (248.6 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
vethb9d0ba4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::c76:3cff:fe82:1111 prefixlen 64 scopeid 0x20<link>
ether 0e:76:3c:82:f4:11 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
ssh-keyscan 10.10.100.Y
read (10.10.100.Y): Connection timed out
read (10.10.100.Y): Connection timed out
read (10.10.100.Y): Connection timed out
read (10.10.100.Y): Connection timed out
read (10.10.100.Y): Connection timed out
ssh-keyscan 192.168.1.X
# 192.168.1.X:22 SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1
192.168.1.X ssh-rsa <key here>
# 192.168.1.X:22 SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1
192.168.1.X ecdsa-sha2-nistp256 <key here>
# 192.168.1.X:22 SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1
192.168.1.X ssh-ed25519 <key here>
# 192.168.1.X:22 SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1
# 192.168.1.X:22 SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1
I've compared all of this information with the same information from the antsle VM to try and do as much of an "apples-to-apples" comparison as possible and try to find anything that looks obviously wrong, but nothing is jumping out at me. Where else should/could I look to find the problem here?
TROUBLESHOOTING (from comments)
- I'm able to SSH to the
enp2s0
interface from the RasPi device that's on the same 10.10
network without a problem.
- I've removed the script from my
hosts.allow
and set it to sshd : ALL
, but that didn't help.
- I've not yet tried disabling public key authentication and trying to log in with a password, but I'll see about testing that later.
- According to
sysctl net.ipv4.ip_forward
, it looks like IP forwarding is enabled, although it doesn't appear to be running as a service (Unit sysctl.service could not be found.
)
- I tried updating the
/etc/nftables.conf
FORWARD policy from drop
to allow
, but that didn't help.
- I've been looking through my network firewall appliance for anything "different" about the connection to that IP address (
10.10.100.Y
), but I keep coming up empty. The connection to the RasPi device on that same network is working just fine so the firewall knows how to route between the two subnets. If the firewall is the problem, it would seem that it has to be a rule explicitly defined for that IP address or something.
Per Request
Here is a "simplified" diagram of the network topology in play here:
- A Fortigate firewall is the "edge" device for our network.
- Interface 1 is the main LAN (
192.168
) and interface 3 is the "secondary" LAN (10.10
).
- Interface 1 goes to a 24-port PoE switch that feeds the rest of our network here. Interface 3 goes to a small switch "dedicated" to the
10.10
network.
I mentioned earlier that I can access the "problem" interface from the public IP address, but only as long as I'm doing so from another network (e.g., home). Trying to SSH there from the 192.168
network here times out (no message about kex_exchange_identification
).
Is this all tangled up in everyone's head now, or is it just me?
UPDATE (still not working):
After digging around a while, I remembered that the 22.04 server (10.10.100.Y
) has nftables
running as the firewall (instead of ufw
). I've posted the nftables.conf
details above with the other firewall information.
Another interesting piece of this puzzle: My firewall has a "virtual IP" set up for routing traffic to the public IP address I've assigned on that server (24.249.N.N
) to the internal private IP on the enp2s0
interface (10.10.100.Y
). I tried to SSH into the server from a remote network to the public IP address and it worked without a problem, so that seems to rule out a lot of the server configuration solutions and leads me to believe the issue may possibly be at the firewall level, even though I can PING that interface and even connect to another device on that same network without any problems.
Just to try and cover all my bases, I checked for the IP address of my client device in my fail2ban
SSH jail, and it doesn't appear there.
Additionally, even though I don't have a clue how to actually read them, I ran some packet captures from end to end to see if anything stuck out. On the server, I ran tcpdump -i enp2s0 port 22 and src 192.168.1.C
while I was trying to connect, which resulted in the following:
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on enp2s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
11:28:04.935646 IP 192.168.1.C.50719 > 10.10.100.Y.ssh: Flags [S], seq 3110886807, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
1 packet captured
1 packet received by filter
0 packets dropped by kernel
It appears that only one packet made it to that interface for whatever reason. The other packet captures (from my client and from the firewall itself) add more information but, as I said, I'm not sure how to read it all, or even how to provide the detail here short of screenshots and/or uploading the .pcap
files, so here are some pics:
failed SSH negotiation resulting in the Connection timed out
message
this connection only failed because I didn't use my public key for the authentication
Obviously there's a lot more active communication going on between my workstation and the "working" server.
I don't know if I'm more or less confused now, but I still don't really know where to look next.
UPDATE #2:
I'm still unable to SSH to the enp2s0
interface but, since I'm able to connect on the eno1
interface, I've been splitting my time between working on that and trying to set up another package - Remotely. The only reason I bring this up is because I've been unable to connect to that service except for a single brief "glimmer" of time where I was able to get the login/admin page to come up (on the eno1
interface, but still). I don't know if one thing has anything to do with the other, but the fact that it randomly started working and then stopped again makes me think there's something very strange going on.
While I'm still open to the possibility that there's something in our network firewall appliance causing the issue, there are two things that make me think that that's not the issue:
- I can connect to another device on that
10.10
network (the RasPi)
- I can connect to the server from an outside connection via the public IP address I have assigned and routed to the
enp2s0
interface.
Of course, #2 also makes me think that the problem is less likely to be a configuration issue on the server, so I'm just totally confused.
Just to provide as much information as I can, I've also added my netplan configuration information. I had wondered if the problem was that the connection was "defaulting" to the eno1
interface and that was causing some "confusion" in the routing, but I don't think the configuration supports that theory.
UPDATE #3 (sorta "fixed" (?)):
Well, interesting development... Thankfully, I have console access to the machine, so I completely disabled the eno1
network card completely and tried to SSH to the 10.10.100.Y
and it connected successfully. I even tried with the public IP 24.249.N.B
and it worked this time as well. I re-enabled the eno1
interface and tried again and it was back to getting the kex_exchange_identification
error.
Thinking that, perhaps the issue has something to do with the two different subnets "freaking out" the network on the machine, I went ahead and reconfigured the eno1
interface to use another IP from the same subnet range (10.10.100.E
) and I was able to SSH to 10.10.100.Y
successfully again. After editing the sshd_config
to change the ListenAddress
to include the new IP, I was even able to SSH to 10.10.100.E
.
So, it looks like something was causing some sort of conflict between the two interfaces (eno1
and enp2s0
) when using the two different subnets. Perhaps the firewall was "freaking out" when trying to route the traffic between the two networks. I'm still stumped by this and would really like to understand why that's such a problem, but it looks like all of the connections to that server are working as expected for the moment.