Join Log in
Score:0
avatar Server

Routing not working properly when two NICs connected

jp flag
user2181884

We're running a server with a NetXtreme BCM5720 2-port Gigabit Ethernet PCIe card. This has two ports, which each maps to different physical id and logical names;

sudo lshw -class network
  *-network:0               
       description: Ethernet interface
       product: NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
       vendor: Broadcom Inc. and subsidiaries
       physical id: 0
       bus info: pci@0000:04:00.0
       logical name: eno1
       version: 00
       serial: b8:cb:29:97:26:61
       size: 1Gbit/s
       capacity: 1Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pm vpd msi msix pciexpress bus_master cap_list rom ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=tg3 driverversion=3.137 duplex=full firmware=FFV21.81.3 bc 5720-v1.39 ip=192.168.1.211 latency=0 link=yes multicast=yes port=twisted pair speed=1Gbit/s
       resources: irq:17 memory:91930000-9193ffff memory:91940000-9194ffff memory:91950000-9195ffff memory:91d00000-91d3ffff
  *-network:1
       description: Ethernet interface
       product: NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
       vendor: Broadcom Inc. and subsidiaries
       physical id: 0.1
       bus info: pci@0000:04:00.1
       logical name: eno2
       version: 00
       serial: b8:cb:29:97:26:62
       size: 1Gbit/s
       capacity: 1Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pm vpd msi msix pciexpress bus_master cap_list rom ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=tg3 driverversion=3.137 duplex=full firmware=FFV21.81.3 bc 5720-v1.39 ip=192.168.3.201 latency=0 link=yes multicast=yes port=twisted pair speed=1Gbit/s
       resources: irq:18 memory:91900000-9190ffff memory:91910000-9191ffff memory:91920000-9192ffff memory:91d40000-91d7ffff

As shown by the output, they are given IPs on different subnets.

The 192.168.3 subnet faces outwards, whereas 192.168.1 is internal only. We have configured port forwarding in our router such that inbound traffic to port 80 and 443 goes to 192.168.3.201 The intent is to run web applications on that server, while leaving the ssh open on the internal network for maintenance.

This works ... partly

In order to test we run the default nginx docker image

docker run -d -p 192.168.3.201:80:80 -p 192.168.3.201:443:443 --restart=unless-stopped nginx:latest

Initially, we are NOT able to access the web interface from outside. However, if we pull the ethernet cable to eno0, it suddenly works. What really boggles my mind is that it actually continues to work when plugging eno0 back in.

This is highly reproducible. After system reboot it does not work, but unplugging / re-plugging eno0 and it suddenly works again.

What are we missing?

Adding output of ip route as per suggest by JFL; at rebbot;

default via 192.168.1.1 dev eno1 proto dhcp src 192.168.1.211 metric 100 
default via 192.168.3.1 dev eno2 proto dhcp src 192.168.3.201 metric 100 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 
192.168.1.0/24 dev eno1 proto kernel scope link src 192.168.1.211 
192.168.1.1 dev eno1 proto dhcp scope link src 192.168.1.211 metric 100 
192.168.3.0/24 dev eno2 proto kernel scope link src 192.168.3.201 
192.168.3.1 dev eno2 proto dhcp scope link src 192.168.3.201 metric 100

When unplugging eno0;

default via 192.168.3.1 dev eno2 proto dhcp src 192.168.3.201 metric 100 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 
192.168.3.0/24 dev eno2 proto kernel scope link src 192.168.3.201 
192.168.3.1 dev eno2 proto dhcp scope link src 192.168.3.201 metric 100

After replugging eno0

default via 192.168.3.1 dev eno2 proto dhcp src 192.168.3.201 metric 100 
default via 192.168.1.1 dev eno1 proto dhcp src 192.168.1.211 metric 100 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 
192.168.1.0/24 dev eno1 proto kernel scope link src 192.168.1.211 
192.168.1.1 dev eno1 proto dhcp scope link src 192.168.1.211 metric 100 
192.168.3.0/24 dev eno2 proto kernel scope link src 192.168.3.201 
192.168.3.1 dev eno2 proto dhcp scope link src 192.168.3.201 metric 100
49
0 + 0
JFL avatar
pk flag
JFL
Can you add the output of "ip route" before and after the unplugging / re-plugging?
2
Reply
Score:0
JFL avatar Server
pk flag
JFL

your issue is that you have configured a gateway on both interfaces, as shown by:

default via 192.168.1.1 dev eno1 proto dhcp src 192.168.1.211 metric 100 
default via 192.168.3.1 dev eno2 proto dhcp src 192.168.3.201 metric 100

Both have the same metric so the system has to choose one. How this choice is made depends on the operating system used.

Whatever the criteria your OS use, it select the route that goes out the eno1 interface at boot, so the traffic is not sent to the Internet.

When you unplug the NIC, there's only one gateway left (the correct one), so it works.

The fact it still works after you replug eno1 show that your OS probably use the "oldest" route (you can see the order of those two routes are reversed after re-pluging).

To fix this , you need to remove the default gateway on the internal NIC, and if needed replace it by routes to you internal network. If you don't have any other internal network than 192.168.1.X/X then you do not need those routes.

Since you are using DHCP you need to remove the "router" setting from the 192.168.3.X/X scope or to use a static IP address instead (if you need to leave the router option for other hosts).

0 + 0
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.