Join Log in
Score:2
Micah Henning avatar Server

Cannot SSH to second network interface in Ubuntu 20.04 on EC2

br flag
Micah Henning

I have a VPC 10.0.0.0/16 with an Internet gateway and two subnets 10.0.100.0/24 and 10.0.200.0/24 in the same availability zone. A single security group allows tcp/22 inbound from 0.0.0.0/0 and everything outbound. I also have two network interfaces tied to the security group, one in each subnet. Each network interface has its own Elastic IP address. There is a routing table for each subnet pointing 0.0.0.0/0 to the Internet gateway.

Here's the issue I am facing: I have an EC2 instance paired with both network interfaces, but can only SSH from the Internet through the one paired to eth0.

This is the configuration:

$ uname -a
Linux ip-10-0-100-70 5.4.0-1045-aws #47-Ubuntu SMP Tue Apr 13 07:02:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc fq_codel state UP group default qlen 1000
    link/ether 02:88:59:6e:78:c0 brd ff:ff:ff:ff:ff:ff
    inet 10.0.100.70/24 brd 10.0.100.255 scope global dynamic eth0
       valid_lft 1806sec preferred_lft 1806sec
    inet6 fe80::88:59ff:fe6e:78c0/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc fq_codel state UP group default qlen 1000
    link/ether 02:7d:b2:d4:b5:ea brd ff:ff:ff:ff:ff:ff
    inet 10.0.200.118/24 brd 10.0.200.255 scope global dynamic eth1
       valid_lft 1807sec preferred_lft 1807sec
    inet6 fe80::7d:b2ff:fed4:b5ea/64 scope link 
       valid_lft forever preferred_lft forever

$ ip route
default via 10.0.100.1 dev eth0 proto dhcp src 10.0.100.70 metric 100 
default via 10.0.200.1 dev eth1 proto dhcp src 10.0.200.118 metric 200 
10.0.200.0/24 dev eth1 proto kernel scope link src 10.0.200.118 
10.0.200.1 dev eth1 proto dhcp scope link src 10.0.200.118 metric 200 
10.0.100.0/24 dev eth0 proto kernel scope link src 10.0.100.70 
10.0.100.1 dev eth0 proto dhcp scope link src 10.0.100.70 metric 100

$ ip rule
0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

$ sudo ufw status verbose
Status: inactive

$ ss -nlput
Netid   State     Recv-Q    Send-Q             Local Address:Port       Peer Address:Port   Process   
udp     UNCONN    0         0                  127.0.0.53%lo:53              0.0.0.0:*                
udp     UNCONN    0         0              10.0.200.118%eth1:68              0.0.0.0:*                
udp     UNCONN    0         0               10.0.100.70%eth0:68              0.0.0.0:*                
tcp     LISTEN    0         4096               127.0.0.53%lo:53              0.0.0.0:*                
tcp     LISTEN    0         128                      0.0.0.0:22              0.0.0.0:*                
tcp     LISTEN    0         128                         [::]:22                 [::]:*

So it appears that the network interfaces are configured properly and that the SSH daemon is listening on all interfaces. The SSH daemon is functioning properly because I am connected via SSH already to eth0. And both interfaces seem to pass traffic to the Internet just fine:

$ ping -I eth0 -c 5 1.1.1.1
PING 1.1.1.1 (1.1.1.1) from 10.0.100.70 eth0: 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=38 time=11.5 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=38 time=11.5 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=38 time=11.5 ms
64 bytes from 1.1.1.1: icmp_seq=4 ttl=38 time=11.6 ms
64 bytes from 1.1.1.1: icmp_seq=5 ttl=38 time=11.5 ms

--- 1.1.1.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4007ms
rtt min/avg/max/mdev = 11.470/11.515/11.567/0.031 ms

$ ping -I eth1 -c 5 1.1.1.1
PING 1.1.1.1 (1.1.1.1) from 10.0.200.118 eth1: 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=38 time=11.7 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=38 time=11.8 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=38 time=11.7 ms
64 bytes from 1.1.1.1: icmp_seq=4 ttl=38 time=11.8 ms
64 bytes from 1.1.1.1: icmp_seq=5 ttl=38 time=11.8 ms

--- 1.1.1.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4008ms
rtt min/avg/max/mdev = 11.705/11.755/11.829/0.042 ms

But cannot connect:

$ ssh -i ~/.ssh/cert.pem ubuntu@3.<redacted>  # EIP for eth0
Welcome to Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-1045-aws x86_64)
...

$ ssh -i ~/.ssh/cert.pem ubuntu@52.<redacted> # EIP for eth1
ssh: connect to host 52.<redacted> port 22: Operation timed out

I setup Netflow on the eth1 network adapter in AWS and can see the traffic:

2 840416055907 eni-07cc18b6f1b89378e <redacted> 10.0.200.118 54268 22 6 9 576 1623280724 1623280761 ACCEPT OK

So I feel like the OS is dropping the traffic. But there is no firewall configured and the SSH daemon is listening on all interfaces. I also tried tailing logs /var/log/{syslog,auth.log,kern.log} and dmesg, but nothing appeared in any of them while I attempted to connect.

I hope I'm missing something easy because I'm at a bit of a loss right now. Any help would be very appreciated!

1270
1 + 3
ssh
A.B avatar
cl flag
A.B
You tell "There is a routing **table** for each subnet pointing `0.0.0.0/0` to the Internet gateway." . In your network configuration dumps there aren't any additional routing tables, nor any additional routing rules pointing to these routing tables. So can you clarify what you meant with "There is a routing table for each subnet ..."? A routing table is not a route (entry). And correct multi-homing requires tables and rules
0
Reply
Micah Henning avatar
br flag
Micah Henning
Thanks for your reply @A.B. The routing tables are defined in AWS so that egress traffic from the EC2 instance can find the Internet gateway.
0
Reply
A.B avatar
cl flag
A.B
As sshd doesn't have a -I option like ping to lock routing to an interface, I'm afraid you'll have routing issues on your instance. Only the first default route works, the 2nd has no effect for sshd.
1
Reply
Score:0
Micah Henning avatar Server
br flag
Micah Henning

Traffic from eth1 needs be configured to route properly:

$ sudo ip rule add from 10.0.200.118 table default
$ sudo ip route add default via 10.0.200.1 dev eth1 table default
$ sudo ip route flush cache

Since we don't want to manually have to SSH into the instance to enter these commands, I created a systemd script to execute on startup.

/home/ubuntu/dual-home.sh

#!/bin/bash

ADDR=$(ip -f inet addr show eth1 | sed -En -e 's/.*inet ([0-9.]+).*/\1/p')
GATEWAY=$(echo $ADDR | sed -En -e 's/(([0-9]+\.){3}).*/\11/p') # assumes /24 mask
sudo ip rule add from $ADDR table default
sudo ip route add default via $GATEWAY dev eth1 table default
sudo ip route flush cache

/etc/systemd/system/dual-home.service

[Unit]
Description=Configure eth1 routing
After=network.target
After=cloud-final.service

[Service]
Type=simple
ExecStart=/bin/bash /home/ubuntu/dual-home.sh

[Install]
WantedBy=cloud-init.target

Then enable the service: $ sudo systemctl enable dual-home

0 + 0
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.