I've set up a new Proxmox 8 server, and I'm struggling to get my networking set up correctly. A normal bridge interface with a single IP range is easy enough, but I can't seem to get my 2 IP ranges to route correctly when using 2 bridges.
Background:
- The Proxmox host has a single physical interface,
eno1, which plugs directly into my hosting provider's switch
- My hosting provider has given me 2 IP ranges.
- 129.212.157.200/29 (Gateway is 129.212.157.201)
- 129.212.175.24/29 (Gateway is 129.212.175.25)
- I have created two bridges,
vmbr0 and vmbr1. This is to provide layer 2 isolation, since I don't entirely trust the VMs on vmbr1, since a third-party vendor has root access to them.
- The primary IP address to access the Proxmox server should be 129.212.157.202.
My understanding is that this situation requires a routed configuration. However, this is where things start to fall apart.
My /etc/network/interfaces config looks like this:
auto lo
iface lo inet loopback
auto eno1
iface eno1 inet static
address 129.212.157.202/29
gateway 129.212.157.201
auto vmbr0
iface vmbr0 inet manual
bridge-ports none
bridge-stp off
bridge-fd 0
auto vmbr1
iface vmbr1 inet manual
bridge-ports none
bridge-stp off
bridge-fd 0
Ideally, I would like to mix-and-match IPs from both IP ranges on both vmbr0 and vmbr1. In other words, I don't want one IP range to be assigned exclusively to vmbr0 and the other IP range to be assigned exclusively to vmbr1.
I have manually added the following routes using ip ro:
default via 129.212.157.201 dev eno1 proto kernel onlink
129.212.157.200/29 dev eno1 proto kernel scope link src 129.212.157.202
129.212.175.24/29 via 129.212.175.25 dev eno1
129.212.175.25 dev eno1 scope link
Now, I have tried to create a VM and bound the virtual NIC to vmbr0, and I've also tried vmbr1. For both of these combinations, I have tried configuring the VM with IPs from both the 129.212.157.200/29 range as well as from the 129.212.175.24/29 range, using the gateways 129.212.157.201 and 129.212.175.25 respectively. No matter what combination of vmbr and IP address I use for the VM, I can't seem to get connectivity. I can't even ping the gateway IP from within the VM.
For comparison, and to test that the two IP ranges are in fact routable via the provided gateways, I have successfully set up a simple bridged implementation with a single vmbr0 per the Proxmox documentation, and that works fine, for both IP ranges and both gateways. Things only seem to fall apart when I switch to the routed approach.
I've also confirmed that Proxy ARP and IP Forwarding are both enabled on the Proxmox host.
To me, this looks like a routing issue on the Proxmox host, but for the life of me I can't see what I'm doing wrong.
As far as I can tell, I haven't configured anything special in terms of firewall rules. I do have the firewall enabled at the datacentre level, the host level, and the VM level. I have tried turning off firewalling at the host and VM level, but that also doesn't seem to make a difference.
One suggestion that has been made is to implement NAT, but I don't see why that would be necessary, since each of the VMs will only have a public IP, and no private IP ranges.
