I have a Dual homed Ubuntu 22.04 LTS Server that has one NIC in DMZ and the other in LAN. I've been wrestling with the netplan configuration and my searching has found many mixed recommendations on how to accomplish my desired configuration due to the recent changes in gateway configurations and routes. I've read the netplan documentation: https://netplan.readthedocs.io/en/latest/netplan-tutorial/
Currently I have the following settings:
$ ip route:
default via 192.168.2.2 dev eth0 proto static metric 100 onlink
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
172.18.0.0/16 dev br-e9fa8283d45d proto kernel scope link src 172.18.0.1
192.168.2.0/24 dev eth0 proto kernel scope link src 192.168.2.39
192.168.14.0/24 dev eth1 proto kernel scope link src 192.168.14.2
Current netplan .yaml config file:
network:
version: 2
renderer: networkd
ethernets:
eth0:
addresses:
- 192.168.2.39/24
dhcp4: no
routes:
- to: 0.0.0.0/0
via: 192.168.2.2
metric: 100
on-link: true
nameservers:
addresses:
- 192.168.2.29
- 192.168.2.10
eth1:
addresses:
- 192.168.14.2/24
dhcp4: no
routing-policy:
- from: 192.168.14.0/24
table: 199
routes:
- to: 0.0.0.0/0
via: 192.168.14.1
metric: 100
table: 199
$ ip rule list
0: from all lookup local
32765: from 192.168.14.0/24 lookup DMZ proto static
32766: from all lookup main
32767: from all lookup default
Current rt_table:
xxx@xxx:/etc/iproute2$ cat /etc/iproute2/rt_tables
#
# reserved values
#
255 local
254 main
253 default
0 unspec
#
# local
#
#1 inr.ruhep
199 DMZ
xxx@xxx:/etc/iproute2$ ip -d route show table 199
unicast default via 192.168.14.1 dev eth1 proto static scope global metric 100
When I attempt to replace the route in this table 199 with the following:
sudo ip r replace default via 192.168.14.1 dev eth1 proto static scope link metric 100 onlink table 199
i receive the following error: "Error: Nexthop has invalid scope."
What I'm trying to accomplish:
1. Allow internet access and Local Networking to go out via LAN connection.
2. Allow inbound traffic from our NATed Public IP to the DMZ Interface: 192.168.14.2 to reach our web application and be returned on the originating NIC (DMZ).
Currently with this configuration I can access the internet, but my web application times out. I've done packet tracing to determine that the flag [S] traffic is making it into the Ubuntu server, but no replies are heading back out.
What I'm finding is if I replace the default route - default via 192.168.2.2 dev eth0 proto static metric 100 onlink with default via 192.168.14.1 dev eth1 proto static metric 100 onlink then my web application works correctly and I see the return traffic, but then my server has no internet access as I need that traffic (updates, etc) to go out the LAN interface.
Please provide some direction on what netplan configration I need to have to allow the dual-NIC to work and persist on reboot.
Thank you in advance for any help.
[EDIT:1]
Thank you - I have read completely through netplan.io and tried different configurations to no avail. I would love to have someone who has successfully done this if possible with netplan in it's modern version, as we've been stabbing at this for 2 weeks now with the questions posted on two other forums and no resolve because people are recommending deprecated options. I have removed the 'on-link: true' and it did not change any behavior.
[EDIT:2]
My internal access site 192.168.2.39:8444 works just fine in the above configuration. I cannot however access the other Public facing site from the DMZ 192.168.14.2:443. I can change the default route either using routing tables or in the main table: default via 192.168.2.2 dev eth0 proto static metric 100 onlink to default via 192.168.14.1 dev eth1 proto static metric 100 onlink adn then my public site works great, but I have no access to my internal application. This seems like such an easy problem to solve on Windows, but I am not proficient with routing configuration in Linux.
[EDIT:3]
I did packet capture with tcpdump and can see the public traffic having only one side of the conversation with Flag [S], but no return traffic via DMZ using this command: sudo tcpdump -i eth1 -n port 443 -S -vv. I can see the replies attempting to go out eth0 using this command sudo tcpdump -i eth0 -n port 443 -S -vv
sudo tcpdump -i eth1 -n port 443 -S -vv
18:46:03.273544 IP (tos 0x0, ttl 47, id 62306, offset 0, flags [none], proto TCP (6), length 60)
xxx.xx.xxx.xx.60921 > 192.168.14.2.443: Flags [S], cksum 0x0b11 (correct), seq 3026599530, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 178557874 ecr 0], length 0
sudo tcpdump -i eth0 -n port 443 -S -vv
18:46:51.177103 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
192.168.14.2.443 > xxx.xx.xxx.xx.18874: Flags [S.], cksum 0xbc3f (incorrect -> 0x7356), seq 78744143, ack 2652949040, win 65160, options [mss 1460,sackOK,TS val 1512654855 ecr 178596451,nop,wscale 7], length 0
[EDIT:4]
Could this have to do with the fact that we have 2 docker containers that need to respond on different physical NICs? the DMZ container needs to allow and respond to port 80/443 on eth1. the LAN container needs to respond to port 8443 on eth0 and access internet for updates. I wonder if the docker netwokring needs attention. HEre is the current route -e:
$ sudo route -e
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
default _gateway 0.0.0.0 UG 0 0 0 eth0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
172.18.0.0 0.0.0.0 255.255.0.0 U 0 0 0 br-e9fa8283d45d
192.168.2.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
192.168.14.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
It looks like all of the docker containers use br-e9fa8283d45d and it is default gateway 0.0.0.0, which in turn points to eth0. Would that cause incoming requests on eth1 to be routed to eth0 on replies?
Thank you-
