Score:0

Ubuntu

Why can't I access internet via vlan in private network?

stewie

4/27/24, 6:33 PM

As the title suggests, I'm having issues setting up the networking so that a server can access the internet. Bear with me while I lay out the infrastructure.

All 3 servers are hosted by Hetzner. The Gateway and Server 1 are cloud servers while Server 2 is a dedicated server connected via vSwitch. Here's some documentation about this setup: Connect Dedicated Servers (vSwitch)

Server 1 and Server 2 should not have a public IP address, and all traffic goes through the gateway.

All port 53 (DNS) requests are routed to the gateway.

I have a working VPN connection from my pc into the private network.

Network         10.1.0.0/16

Static Route    0.0.0.0/0 -> 10.1.0.2

Subnet          10.1.0.0/24
                        Gateway 10.1.0.2 (pfSense - OpenVpn, DnsResolver)
                        Server1 (Cloud) 10.1.0.3 
                        
Subnet          10.1.1.0/24
                        
                        Server2 (Dedicated) 10.1.1.3

Server 1

Ubuntu 22.04

Accessisble via vpn
Can ping/access 10.1.0.1, 10.1.0.2, 10.1.1.3
Can ping/access internet
Can resolve any domain (external and network internal via pfSense DNS Resolver)

ip route
default via 10.1.0.1 dev enp7s0 proto static onlink
10.1.0.0/16 via 10.1.0.1 dev enp7s0 proto dhcp src 10.1.0.3 metric 100
10.1.0.1 dev enp7s0 proto dhcp scope link src 10.1.0.3 metric 100
169.254.169.254 via 10.1.0.1 dev enp7s0 proto dhcp src 10.1.0.3 metric 100

Netplan

network:
  version: 2
  renderer: networkd
  ethernets:
    enp7s0:
      dhcp4: true
      routes:
        - to: default
          via: 10.1.0.1
      nameservers:
        addresses:
          - 10.1.0.2
          - 8.8.4.4
          - 8.8.8.8

tracepath -n 10.1.1.3
1?: [LOCALHOST]         pmtu 1450
1:  10.1.0.1            2.623ms
1:  10.1.0.1            1.490ms
2:  169.254.255.255     1.934ms asymm  1
3:  10.1.1.3            25.501ms reached
    Resume: pmtu 1450 hops 3 back 2

tracepath -n google.com
 1?: [LOCALHOST]    pmtu 1450
 1:  10.1.0.1
 1:  10.1.0.1
 2:  10.1.0.2
 3:  172.31.1.1

Server 2

Ubuntu 22.04

Accessisble via vpn
Can ping/access 10.1.0.2, 10.1.0.3
Can NOT ping 10.1.1.1
Can NOT ping/access internet
Can resolve any domain (external and network internal via pfSense DNS Resolver)

ip route
default via 10.1.1.1 dev enp2s0.4000 proto static
10.1.1.0/24 dev enp2s0.4000 proto kernel scope link src 10.1.1.3

netplan

network:
  version: 2
  renderer: networkd
  ethernets:
    enp2s0: {}
vlans:
  enp2s0.4000:
    id: 4000
    link: enp2s0
    mtu: 1400
    addresses:
      - 10.1.1.3/24
    routes:
      - to: default
        via: "10.1.1.1"
    nameservers:
      addresses:
        - 10.1.0.2
        - 8.8.4.4
        - 8.8.8.8

tracepath -n 10.1.0.3
1?: [LOCALHOST]     pmtu 1400
1:  10.1.1.1        0.393ms
1:  10.1.1.1        0.364ms
2:  10.1.0.3        25.265ms reached
    Resume: pmtu 1400 hops 2 back 3

tracepath -n google.com
1?: [LOCALHOST] pmtu 1400
1:  no reply

As you can hopefully see, the issue I have is with the dedicated server (Server 2) which does not seem to be able to connect to it's subnet gateway (10.1.1.1) and beyond, but it can connect to other servers in the private network just fine.

Maybe I'm looking at the wrong thing?

263

1 + 2

routing

gateway

netplan

vlan

Thomas Ward

4/28/24, 12:00 AM

Just to confirm, the connection line to your system **is a trunk port** accepting VLAN tagged traffic, yes? Otherwise the VLAN tagged packets won't work anyways going out the door (regardless of other answers)

stewie

4/28/24, 3:28 PM

@ThomasWard unfortunately i don't know what that means or how i would find that out since all the hardware is from hetzner

Score:0

Ubuntu

stewie

4/28/24, 3:40 PM

After having the chance to talk with someone that has lots of experience in that area he pointed me towards this section of the documentation.

His assumption was that the vSwitch is only allowing traffic inside the private network to go through and blocks everything else.

The clean solution would have been to connect the dedicated server (Server 2) to the same subnet (10.1.0.0/24) as my pfSense gateway but Hetzner doesn't allow this.

The workaround was using Wireguard:

Set up Wireguard in pfSense with the interface on 10.2.0.1/24 and the dedicated peer with the endpoint 10.1.1.3
Set up Wireguard on the dedicated server with the interface on 10.2.0.2/24 and the pfSense peer with the endpoint 10.1.0.2

Since the private traffic works between the two, the Wireguard endpoints can be the private ips.

Allow all traffic to any source from the Wireguard interface in pfSense
Setup the following netplan on the dedicated server to only route non private ips through the Wireguard tunnel

network:
  version: 2
  renderer: networkd
  ethernets:
    enp2s0: {}
vlans:
  enp2s0.4000:
    id: 4000
    link: enp2s0
    mtu: 1400
    addresses:
      - 10.1.1.3/24
    routes:
      - to: 10.1.0.0/16
        via: 10.1.1.1
        on-link: true
      - to: 10.1.1.0/24
        via: 10.1.1.1
        on-link: true
      - to: default
        via: 10.2.0.1
        on-link: true
    nameservers:
      addresses:
        - 10.1.0.2
        - 8.8.4.4
        - 8.8.8.8

Works like a charm!

+ 3

mpboden

4/29/24, 1:33 AM

This is a very interesting solution. I'm curious...if Server 2 isn't allowed to have public access due to the vSwitch, what is the intended purpose of it? Something like a web server to only allow incoming connections? Also, I still don't understand why you weren't able to ping the gateway at 10.1.1.1. Additionally, my familiarity with pfSense is that the resolver is on each gateway of the router. So in your case the DNS resolver would be on both 10.1.0.1 and 10.1.1.1. That's how it works on my network with pfSense. Each subnet references the DNS resolver at it's respective gateway address.

stewie

4/30/24, 6:54 PM

@mpboden Correct. Server 2 is a web server that should sit behind the firewall but still be able to call external services. I can only assume that hetzner is somehow blocking the ping to 10.1.1.1 in their vSwitch setup. But i don't know for sure. As for the pfSense setup, i'm very new to the whole network setup stuff so i might be trying unconventional methods^^ But as hetzner doesn't allow cloud and dedicated servers on the same subnet, i wouldn't be able to put a pfSense on the 10.1.1.0 subnet without having a whole dedicated server anyways.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Why can't I access internet via vlan in private network?

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.