Problem: after changing bridge-ports value of a network interface, all networking on that IP stops working. Base system is Proxmox VE 7.4, based on Debian Bullseye.
Background: I have a Proxmox VE home lab with 3 nodes. I've recently decided to move 10gbps networking.
Each node is configured as follows:
- two 1gbps ports (onbaord, Broadcom BCM5716) -
eno1 and eno2
- two 10gps ports (PCIE, Broadcom BCM57810) -
enp3s0f0 and enp30f1
- five network bridges
- kernel is PVE 5.15.108-1
Each bridge has one bridge-port, which is a sub-interface of one of physical ports on the node. Originally this was configured as:
┌─────────────────┐ ┌────────────┐ ┌───────────────────┐
│ cisco switch │ Trunks │ eno1 │ │ vmbr10 Admin │
│ ├────┬───────►│ ├────┬───────►│ eno1.10 │
│ │ │ │ │ │ │ IP.10.0.0.4x/24 │
│ │ │ │ │ │ │ GW.10.0.0.1 │
└─────────────────┘ │ └────────────┘ │ └───────────────────┘
│ │
│ │ ┌───────────────────┐
│ │ │ vmbr20 Secured │
│ ├───────►│ eno1.20 │
│ │ │ │
│ │ │ │
│ │ └───────────────────┘
│ │
│ │ ┌───────────────────┐
│ │ │ vmbr21 Insecure │
│ ├───────►│ eno1.21 │
│ │ │ │
│ │ │ │
│ │ └───────────────────┘
│ │
│ │ ┌───────────────────┐
│ │ │ vmbr100 ISP │
│ └───────►│ eno1.100 │
│ │ │
│ │ │
│ └───────────────────┘
│
│ ┌────────────┐ ┌───────────────────┐
│ │ eno2 │ │ vmbr101 Ceph │
└───────►│ ├────────────►│ eno2.101 │
│ │ │ IP.10.0.101.4x/24 │
│ │ │ │
└────────────┘ └───────────────────┘
Since then I was able to change all except vmbr10 to use the new 10gps cards:
┌─────────────────┐ ┌────────────┐ ┌───────────────────┐
│ cisco switch │ Trunks │ eno1 │ │ vmbr10 Admin │
┌────►│ ├────┬───────►│ ├────────────►│ eno1.10 │
│ │ │ │ │ │ │ IP.10.0.0.4x/24 │
│ │ │ │ │ │ │ GW.10.0.0.1 │
│ └─────────────────┘ │ └────────────┘ └───────────────────┘
│ │
│ │ ┌────────────┐
│ │ │ eno2 │
│ └───────►│ │
│ │ │
│ │ │
│ └────────────┘
│
│ ┌─────────────────┐ ┌────────────┐ ┌───────────────────┐
│ │ mikrotik switch │ Trunks │ enp3s0f0 │ │ vmbr20 Secured │
└────►│ ├────┬───────►│ ├─────┬──────►│ enp3s0f0.20 │
│ │ │ │ │ │ │ │
│ │ │ │ │ │ │ │
└─────────────────┘ │ └────────────┘ │ └───────────────────┘
│ │
│ │ ┌───────────────────┐
│ │ │ vmbr21 Insecure │
│ ├──────►│ enp3s0f0.21 │
│ │ │ │
│ │ │ │
│ │ └───────────────────┘
│ │
│ │ ┌───────────────────┐
│ │ │ vmbr100 ISP │
│ └──────►│ enp3s0f0.100 │
│ │ │
│ │ │
│ └───────────────────┘
│
│ ┌────────────┐ ┌───────────────────┐
│ │ enp3s0f1 │ │ vmbr101 Ceph │
└───────►│ ├────────────►│ enps30f1.101 │
│ │ │ IP.10.0.101.4x/24 │
│ │ │ │
└────────────┘ └───────────────────┘
Switching vmbr10 to use enp3s0f0 immediately brings connection down and the system will not communicate on that VLAN until I manually re-add eno1.10 to bridge-ports field. It can be configured as bridge-ports eno1.10 enp3s0f0.10 but will only use eno1.10.
STP is not enabled, and these bridges are not learning any MAC addresses.
This is not limited to Proxmox services but affects everything, including SSH and ping - this leads me to think this is a behaviour/configuration issue.
What I have examined and tried:
- changing
bridge-ports then rebooting
- using
tcpdump to examine traffic at physical interface and enp3s0f0 is receiving VLAN10 traffic for the node
- using
tcpdump to examine traffic on vmbr10 - only broadcast ARP requests are seen
- using
brctl to check port state and it is forwarding
- checked
iptables and ebtables - I do not use either and all chains/tables are in default accept state and do not have any rules
For reference here is the current content of /etc/network/interfaces - this is generated by Proxmox VE's Network config editor.
auto lo
iface lo inet loopback
iface eno1 inet manual
iface eno2 inet manual
iface enp3s0f0 inet manual
iface enp3s0f1 inet manual
auto vmbr10
iface vmbr10 inet static
address 10.0.0.41/24
gateway 10.0.0.1
bridge-ports eno1.10
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#VLAN 10 Admin
auto vmbr100
iface vmbr100 inet manual
bridge-ports enp3s0f0.100
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#VLAN 10 ISP
auto vmbr101
iface vmbr101 inet static
address 10.0.101.41/24
bridge-ports enp3s0f1.101
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#VLAN 101 Ceph Backend
auto vmbr20
iface vmbr20 inet manual
bridge-ports enp3s0f0.20
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#VLAN 20 Trusted
auto vmbr21
iface vmbr21 inet manual
bridge-ports enp3s0f0.21
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#VLAN 21 Untrusted
Does anyone have suggestions on how to further debug this?
