Join Log in
Score:2
Hisham Dawalibi avatar Ubuntu

2 Ubuntu servers, A is connected to B via ethernet. server B has 2 NIC, one of is connected to the internet. A cannot see B or the internet

mz flag
Hisham Dawalibi

I have one ubuntu server running 20.04 with 2 NIC – one connected to the internet gateway and the other connected to another ubuntu server running 22.04. I cannot get the computer running 22.04 to see the internet or the 20.04 server. The 20.4 server does see the internet.

The yaml file on the 20.04 server is

# This is the network config written by 'subiquity'
network:

  ethernets:
        enp3s0:
            dhcp4: false
            addresses: ['10.0.0.205/24']
            gateway4: 10.0.0.1
            nameservers:
                addresses: [10.0.0.1, 8.8.8.8]
        enp2s0:
            dhcp4: false
            addresses: ['10.0.0.207/24']

  version: 2

The lshw -C network command on the 20.04 server is

 sudo lshw -C network
  *-network
       description: Ethernet interface
       product: RTL8125 2.5GbE Controller
       vendor: Realtek Semiconductor Co., Ltd.
       physical id: 0
       bus info: pci@0000:02:00.0
       logical name: enp2s0
       version: 05
       serial: 1c:86:0b:22:73:5d
       capacity: 1Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress msix vpd bus_master cap_list rom ethernet physical tp mii 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=r8169 driverversion=5.15.0-75-generic firmware=rtl8125b-2_0.0.2 07/13/20 latency=0 link=no multicast=yes port=twisted pair
       resources: irq:17 ioport:e000(size=256) memory:df110000-df11ffff memory:df120000-df123fff memory:df100000-df10ffff
  *-network
       description: Ethernet interface
       product: RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
       vendor: Realtek Semiconductor Co., Ltd.
       physical id: 0
       bus info: pci@0000:03:00.0
       logical name: enp3s0
       version: 15
       serial: 30:9c:23:0c:90:d9
       size: 1Gbit/s
       capacity: 1Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress msix bus_master cap_list ethernet physical tp mii 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=r8169 driverversion=5.15.0-75-generic duplex=full firmware=rtl8168h-2_0.0.2 02/26/15 ip=10.0.0.205 latency=0 link=yes multicast=yes port=twisted pair speed=1Gbit/s
       resources: irq:18 ioport:d000(size=256) memory:df004000-df004fff memory:df000000-df003fff

The yaml file for the 22.04 server is

# This is the network config written by 'subiquity'
network:
  ethernets:
        enp3s0:
            addresses: ['10.0.0.206/24']
            gateway4: 10.0.0.1
            nameservers:
                addresses: [8.8.8.8]
            routes:
              - to: default
                via: 10.0.0.207
  version: 2

The lshw -C network command on the 22.04 server is

sudo lshw -C network
  *-network
       description: Wireless interface
       product: Intel Corporation
       vendor: Intel Corporation
       physical id: 14.3
       bus info: pci@0000:00:14.3
       logical name: wlo1
       version: 11
       serial: 98:59:7a:99:6c:2c
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress msix bus_master cap_list ethernet physical wireless
       configuration: broadcast=yes driver=iwlwifi driverversion=5.19.0-051900-generic firmware=72.a764baac.0 so-a0-gf-a0-72.uc latency=0 link=no multicast=yes wireless=IEEE 802.11
       resources: irq:18 memory:42314000-42317fff
  *-network
       description: Ethernet interface
       product: RTL8125 2.5GbE Controller
       vendor: Realtek Semiconductor Co., Ltd.
       physical id: 0
       bus info: pci@0000:03:00.0
       logical name: enp3s0
       version: 05
       serial: 74:56:3c:2c:f1:bc
       size: 1Gbit/s
       capacity: 1Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress msix vpd bus_master cap_list ethernet physical tp mii 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=r8169 driverversion=5.19.0-051900-generic duplex=full firmware=rtl8125b-2_0.0.2 07/13/20 ip=10.0.0.206 latency=0 link=yes multicast=yes port=twisted pair speed=1Gbit/s
       resources: irq:18 ioport:3000(size=256) memory:42100000-4210ffff memory:42110000-42113fff

Also I cannot ping one server to the other.

@Raffa

Thank you for your quick reply. I have made the changes according to your latest instruction, but I still cannot ping the gateway (10.0.0.1) from server A.

The following is the ipconfig from server A

ifconfig
enp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.0.206  netmask 255.255.255.0  broadcast 10.0.0.255
        inet6 fe80::7656:3cff:fe2c:f1bc  prefixlen 64  scopeid 0x20<link>
        inet6 2001:16a2:cb96:3c00::5  prefixlen 128  scopeid 0x0<global>
        inet6 2001:16a2:cb96:3c00:7656:3cff:fe2c:f1bc  prefixlen 64  scopeid 0x0<global>
        ether 74:56:3c:2c:f1:bc  txqueuelen 1000  (Ethernet)
        RX packets 306  bytes 26761 (26.7 KB)
        RX errors 0  dropped 60  overruns 0  frame 0
        TX packets 1621  bytes 150679 (150.6 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 3793  bytes 519354 (519.3 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3793  bytes 519354 (519.3 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

wlo1: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether 98:59:7a:99:6c:2c  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

The following is the ipconfig from server B

ifconfig
enp2s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.0.207  netmask 255.255.255.0  broadcast 10.0.0.255
        inet6 fe80::1e86:bff:fe22:735d  prefixlen 64  scopeid 0x20<link>
        ether 1c:86:0b:22:73:5d  txqueuelen 1000  (Ethernet)
        RX packets 1428  bytes 99754 (99.7 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 131  bytes 8666 (8.6 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.0.205  netmask 255.255.255.0  broadcast 10.0.0.255
        inet6 2001:16a2:cb96:3c00:329c:23ff:fe0c:90d9  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::329c:23ff:fe0c:90d9  prefixlen 64  scopeid 0x20<link>
        inet6 2001:16a2:cb96:3c00::3  prefixlen 128  scopeid 0x0<global>
        ether 30:9c:23:0c:90:d9  txqueuelen 1000  (Ethernet)
        RX packets 31323  bytes 33052255 (33.0 MB)
        RX errors 0  dropped 1228  overruns 0  frame 0
        TX packets 14667  bytes 4587072 (4.5 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 54250  bytes 38017605 (38.0 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 54250  bytes 38017605 (38.0 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

and the following are the iptables on sever B

 sudo iptables -L
Chain INPUT (policy DROP)
target     prot opt source               destination
f2b-phpmyadmin-syslog  tcp  --  anywhere             anywhere             multiport dports http,https
f2b-postfix-sasl  tcp  --  anywhere             anywhere             multiport dports smtp,submissions,submission,imap2,imaps,pop3,pop3s
f2b-phpmyadmin-syslog  tcp  --  anywhere             anywhere             multiport dports http,https
f2b-postfix-sasl  tcp  --  anywhere             anywhere             multiport dports smtp,submissions,submission,imap2,imaps,pop3,pop3s
f2b-sshd   tcp  --  anywhere             anywhere             multiport dports 2200
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:2200
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:smtp
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:pop3
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:imap2
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:https
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:submissions
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:submission
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:imaps
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:pop3s
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:webmin
ACCEPT     udp  --  anywhere             anywhere             udp spt:domain
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:3478
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere

Chain f2b-phpmyadmin-syslog (2 references)
target     prot opt source               destination
REJECT     all  --  103.175.198.129      anywhere             reject-with icmp-port-unreachable
RETURN     all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain f2b-postfix-sasl (2 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain f2b-sshd (1 references)
target     prot opt source               destination
REJECT     all  --  211.36.142.65        anywhere             reject-with icmp-port-unreachable
RETURN     all  --  anywhere             anywhere

Let me know if I can provide you with more details.

@Raffa,

Ok I have deleted all iptables rules and here is the iptables

/etc/iptables# cat rules.v4
# Generated by iptables-save v1.8.4 on Sat Aug 19 16:54:30 2023
*nat
:PREROUTING ACCEPT [8:428]
:INPUT ACCEPT [7:364]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A POSTROUTING -o enp3s0 -j MASQUERADE
COMMIT
# Completed on Sat Aug 19 16:54:30 2023
# Generated by iptables-save v1.8.4 on Sat Aug 19 16:54:30 2023
*filter
:INPUT ACCEPT [37731:25530374]
:FORWARD ACCEPT [1311:95992]
:OUTPUT ACCEPT [38754:27316299]
-A FORWARD -i enp2s0 -o enp3s0 -j ACCEPT
-A FORWARD -i enp3s0 -o enp2s0 -m state --state RELATED,ESTABLISHED -j ACCEPT
COMMIT
# Completed on Sat Aug 19 16:54:30 2023

I still cannot ping the gateway from the 10.0.0.206 server Any ideas ?

Thanks

Following your last comment find the following :

sudo ethtool enp2s0 from server B

sudo ethtool enp2s0
Settings for enp2s0:
        Supported ports: [ TP MII ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
                                2500baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
                                2500baseT/Full
        Advertised pause frame use: Symmetric Receive-only
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                             100baseT/Half 100baseT/Full
                                             1000baseT/Full
                                             2500baseT/Full
        Link partner advertised pause frame use: Symmetric Receive-only
        Link partner advertised auto-negotiation: Yes
        Link partner advertised FEC modes: Not reported
        Speed: 2500Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        MDI-X: Unknown
        Supports Wake-on: pumbg
        Wake-on: d
        Link detected: yes

And here is sudo ethtool enp3s0 from server A

Settings for enp3s0:
        Supported ports: [ TP    MII ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
                                2500baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
                                2500baseT/Full
        Advertised pause frame use: Symmetric Receive-only
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                             100baseT/Half 100baseT/Full
                                             1000baseT/Full
                                             2500baseT/Full
        Link partner advertised pause frame use: Symmetric Receive-only
        Link partner advertised auto-negotiation: Yes
        Link partner advertised FEC modes: Not reported
        Speed: 2500Mb/s
        Duplex: Full
        Auto-negotiation: on
        master-slave cfg: preferred slave
        master-slave status: slave
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: external
        MDI-X: Unknown
        Supports Wake-on: pumbg
        Wake-on: d
        Link detected: yes
331
2 + 15
nic
raj avatar
cn flag
raj
Two interfaces on the same network on 20.04 server? (10.0.0.205/24 and 10.0.0.207/24) That's asking for trouble... Usually, you should NOT configure two interfaces on the same network on the same system, because you don't know via which interface the packet will be sent out, and on which interface will the response be received (all interfaces reply to ARP for each other), and newer kernels by default block asymmetric routing. Rethink your network design.
0
Reply
Raffa avatar
jp flag
Raffa
@raj asymetric-routing(*where outgoing and incoming packets for the same connection (or flow) use different interfaces*) might happen with multiple networks, gateways, or exit "public" IPs i.e. the opposite situation of the OP ... See for example https://utcc.utoronto.ca/~cks/space/blog/tech/SymmetricAndAsymmetricIPRouting
0
Reply
raj avatar
cn flag
raj
@Raffa packet going out via 10.0.0.205 and reply returning via 10.0.0.207 is asymmetric routing as well.
0
Reply
Raffa avatar
jp flag
Raffa
@raj Agreed that's indeed asymmetric ,,, But, I guess you aren't addressing the situation in the question, right? ... As I don't see how would a packet leaving via 10.0.0.205 find it's way back in via 10.0.0.207 that is connected directly (*not through a switch*) to 10.0.0.206 ... Or why would the external internet router/NAT do that ... Or how would that happen even internally on the same sub-net level with the kernel's ARP tables in-place ... Or even why would communication between two hosts on the same sub-net be asymmetric at all ... That's a bit hard for me to imagine AFAIK.
0
Reply
raj avatar
cn flag
raj
If something on the net queries ARP for 10.0.0.205, either 10.0.0.205 or 10.0.0.207 may answer, because that's how Linux TCP/IP stack works. So packet destined for 10.0.0.205 may actually arrive at 10.0.0.207. Two interfaces on the same network are strongly discouraged.
0
Reply
Raffa avatar
jp flag
Raffa
@raj That would be just a hop in the route and not asymmetric routing per-se ... The ARP request will get echoed from the same interface that received the ping as I understand it ... And have actually previously traced it in fact out of boredom :-).
0
Reply
Raffa avatar
jp flag
Raffa
To be able to ping `10.0.0.1` add a route "to 10.0.0.0/24 via 0.0.0.0"
0
Reply
Hisham Dawalibi avatar
mz flag
Hisham Dawalibi
@Raffa, I am unable to ping 10.0.0.1 from 10.0.0.206. So what do you mean by your message above about adding a route to 10.0.0.0/24. Where should this be added ?
0
Reply
Raffa avatar
jp flag
Raffa
I thought you meant to ping the gateway from server B at `10.0.0.205` as `10.0.0.1` is it's gateway as you need the route "to 10.0.0.0/24 via 0.0.0.0" which should be automatically there ... For server A at `10.0.0.206` it's gateway is `10.0.0.207` and it should be pingable ... `10.0.0.1` means nothing to server A which you have deliberately isolated from the rest of it's subnet members behind server B, isn't that what you want? ... That is how it works ... Is everything else working though?
0
Reply
Hisham Dawalibi avatar
mz flag
Hisham Dawalibi
@Raffa. The problem has not been resolved. I still cannot ping from 10.0.0.206 to either 10.0.0.207 or 10.0.0.1. I spent so many hours on this and still cannot figure it out.
0
Reply
mpboden avatar
do flag
mpboden
In the least, you should be able to ping Server A at 10.0.0.207 from Server B at 10.0.0.206. If not, then perhaps the cable is faulty. Otherwise, how old is your hardware? If it’s old and you’re using a straight-through cable, then that could be a problem. Your hardware might not support [auto MDI-X](https://en.m.wikipedia.org/wiki/Medium-dependent_interface#Auto_MDI-X). In this case, a crossover cable is needed. Finally, have you attempted using a switch or hub between the servers?
0
Reply
Hisham Dawalibi avatar
mz flag
Hisham Dawalibi
Thanks for your reply. The NICs are 1Gb/s and fairly new. I have tried using a crossover cable and it did not work. However when I connect the 2 servers through a switch, then I am able to ping in both directions. I have done that with the same cables I used for the direct connect between the 2 servers, so I guess it cannot be a faulty cable !!!.
0
Reply
mpboden avatar
do flag
mpboden
Sounds like the addition of a switch is the solution. Meanwhile, can you post output of `sudo ethtool enp3s0` from server A and `sudo ethtool enp2s0` from Server B? You may need to install `ethtool` if it’s not already. This output will show you if you have a link established when the cables are plugged in. It’ll also tell you if Auto MDI-X is enabled.
0
Reply
Hisham Dawalibi avatar
mz flag
Hisham Dawalibi
I have added the posts that you have asked for, with the a link between thee 2 servers (not going through the switch)
0
Reply
mpboden avatar
do flag
mpboden
Since you’re able to ping with a switch inserted between the two hosts, and both nics indicate MDI-X as `unknown`, my presumption is that MDI-X is not auto-enabling. If so, this could explain why you’re having issues. You may want to force MDI-X by running `sudo ethtool -s enp3s0 mdix on` for Server A and `sudo ethtool -s enp2s0 mdix on` for Server B.
0
Reply
Score:0
Raffa avatar Ubuntu
jp flag
Raffa

On server A

The minimal net-plan configuration file needed is:

network:
  ethernets:
    enp3s0:
      dhcp4: false
      addresses: [10.0.0.206/24]
      nameservers:
        addresses: [10.0.0.1, 8.8.8.8]
      routes:
        - to: default
          via: 10.0.0.207
  version: 2

On server B

The minimal net-plan configuration file needed is:

network:
  ethernets:
    enp3s0:
      dhcp4: false
      addresses: [10.0.0.205/24]
      nameservers:
        addresses: [10.0.0.1, 8.8.8.8]
      routes:
        - to: default
          via: 10.0.0.1
    enp2s0:
      dhcp4: false
      addresses: [10.0.0.207/24]
  version: 2

The minimal system configuration

IP forwarding(routing)
sudo sysctl -w "net.ipv4.ip_forward=1"
Packet forwarding (both outbound and inbound) between the two interfaces

outbound:

sudo iptables -A FORWARD -i enp2s0 -o enp3s0 -j ACCEPT

inbound:

sudo iptables -A FORWARD -i  enp3s0 -o enp2s0 -m state --state RELATED,ESTABLISHED -j ACCEPT
NATing
sudo iptables -t nat -A POSTROUTING -o enp3s0 -j MASQUERADE
+ 0
Score:0
ognjen avatar Ubuntu
ky flag
ognjen

In this config on 22.04 you need to delete gateway4: 10.0.0.1, this is as i understand from you description interface connected to the 10.0.0.207 and having both options is not going to work. To make it easier i would have gone with connecting them all to the switch. If they are all on the the switch than delete from 22.04 all of these routes: - to: default via: 10.0.0.207

+ 0
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.