I am trying to configure two Network interfaces on an EC2 instance using Netplan. These interfaces have the same subnet. When the cloud-init configures it, the netplan configuration looks as follows:
network:
ethernets:
eth0:
dhcp4: true
dhcp4-overrides:
route-metric: 100
dhcp6: false
match:
macaddress: xx:xx:xx:xx:xx:xx
set-name: eth0
eth1:
dhcp4: true
dhcp4-overrides:
route-metric: 200
dhcp6: false
match:
macaddress: yy:yy:yy:yy:yy:yy
set-name: eth1
version: 2
The routing for these interfaces are setup in the same OS route table. The rules are as follows:
# ip route show
default via 172.31.0.1 dev eth0 proto dhcp src 172.31.7.250 metric 100
default via 172.31.0.1 dev eth1 proto dhcp src 172.31.5.137 metric 200
172.31.0.0/20 dev eth0 proto kernel scope link src 172.31.7.250 metric 100
172.31.0.0/20 dev eth1 proto kernel scope link src 172.31.5.137 metric 200
172.31.0.1 dev eth0 proto dhcp scope link src 172.31.7.250 metric 100
172.31.0.1 dev eth1 proto dhcp scope link src 172.31.5.137 metric 200
172.31.0.2 dev eth0 proto dhcp scope link src 172.31.7.250 metric 100
172.31.0.2 dev eth1 proto dhcp scope link src 172.31.5.137 metric 200
Now the problem comes when the traffic comes from secondary interface such that the services on the EC2 are not accessible from secondary interface if made outside from the subnet.
Taking the packet capture on the EC2, it is determined that even though the request comes in from eth1, it goes out via eth0.
# tcpdump -nni any host ${HOME_ADDRESS}
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
02:01:18.036735 eth1 In IP ${HOME_ADDRESS} > 172.31.5.137: ICMP echo request, id 36766, seq 5, length 40
02:01:18.036765 eth0 Out IP 172.31.5.137 > ${HOME_ADDRESS}: ICMP echo reply, id 36766, seq 5, length 40
Checking few links such as: https://wbhegedus.me/avoiding-asymmetric-routing/
It looks like even though the source is setup for the default route, the default route with metric 100 is prioritized as a result the packet comes out of eth0 rather than eth1.
For now, the solution that I have found is to setup static routes for the eth1, specifying a different route table with following netplan configuration:
network:
renderer: networkd
ethernets:
eth0:
dhcp4: true
dhcp4-overrides:
route-metric: 100
dhcp6: false
match:
macaddress: xx:xx:xx:xx:xx:xx
set-name: eth0
eth1:
dhcp4: false
addresses:
- 172.31.5.137/20
routes:
- to: 172.31.0.0/20
via: 172.31.5.137
table: 101
- to: 0.0.0.0/0
via: 172.31.0.1 # Default gateway
table: 101
routing-policy:
- from: 172.31.5.137
table: 101
match:
macaddress: yy:yy:yy:yy:yy:yy
set-name: eth1
version: 2
This works and I can verify that the ping goes out via same interface it comes from.
# tcpdump -nni any host ${HOME_ADDRESS}
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
02:01:18.036735 eth1 In IP ${HOME_ADDRESS} > 172.31.5.137: ICMP echo request, id 36766, seq 5, length 40
02:01:18.036765 eth1 Out IP 172.31.5.137 > ${HOME_ADDRESS}: ICMP echo reply, id 36766, seq 5, length 40
That being said, this configuration is not portable as everytime I have to add a new interface, I have to add static routes. So I was looking for a way to mix different route table approach with dhcp to make it more dynamic.
Is there are way to setup secondary network interfaces with netplan using DHCP with routes being created in a different route table ? I checked network dispatcher but did not find any examples on how to do this.
Any help would be appreciated.
OS: ubuntu 22.04