Hoping this is the right place - I originally posted on Network Engineering but it got closed and I was pointed to Server Fault.
I am currently attempting to setup a L2 bridge between two sites using VXLAN to provide the L2 connectivity and Wireguard as transport/L3. I've previously done a Layer 2 bridge like this using GRE over Wireguard and it's been rock-solid, but I'm trying to better understand VXLAN now, and am looking to replace the GRE tunnel with VXLAN.
I've been trying to make use of the info both here and here but for the life of me I can't get traffic to pass over the non-wireguard IPs between sites.
I have two Debian machines with bridge-utils installed. They're also running nftables with rules to drop all DHCP traffic as when I first setup the GRE tunnel I ended up with machines getting assigned IPs from the remote network. But everything else is set to allow and it's only exposed externally via the Wireguard port
Host A is setup with:
Wireguard wg0 - 172.30.100.1/24
Bridge br0 - 10.0.0.160/24
Host B is setup with:
Wireguard wg0 - 172.30.100.2/24
Bridge br0 - 10.1.0.160/24
The AllowedIPs on the Wireguard configs is only for the Wireguard subnet 172.30.100.0/24. This was working with the GRE config and I'd assume would work with VXLAN too, as the VXLAN traffic is encapsulated within the Wireguard tunnel.
The hosts can ping and ssh each other on their Wireguard IPs, so that bit is working fine.
The bridges both have port ens18, bridge-vlan-aware yes and bridge-vids 1-4096 in /etc/network/interfaces
I have a script based on the 'Recipe 2' from the first link I posted above, i.e. a single tunnel with multiple VNIs. The idea is that the script adds the VXLAN interface vx0 to br0, which waits until after wg0 is up (using a systemd service), and then loops to do the VLAN/VNI mapping.
#!/bin/bash
# Gets Wireguard interface IP address.
wgip=`ip a s wg0 | egrep -o 'inet [0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' | cut -d' ' -f2`
ip link add vx0 type vxlan dstport 4789 external local $wgip dev wg0 # Creates vxlan with wg0 IP as local
#Here is where I may be going wrong but I've tried various combinations ^^^
sleep 1
ip link set dev vx0 master br0 # Adds vxlan to bridge
bridge link set dev vx0 vlan_tunnel on # Enables vlan tunnel on vxlan
# Maps each VLAN to VNIs across the tunnel.
for vlan in 10 20 30; do
bridge vlan add vid $vlan dev vx0
bridge vlan add vid $vlan dev ens18
bridge vlan add dev vx0 vid $vlan tunnel_info id $vlan
# Think I can remove the below line if I switch to BGP-EVPN for learning later?
bridge fdb append 00:00:00:00:00:00 dev vx0 vni $vlan dst 10.1.0.160
done
ip link set dev vx0 up
I may be completely on the wrong track here, but if there's anything that looks off in the above, any guidance in the right direction would be greatly appreciated!
(It might even just be down to routing rather than config. Wireguard config is set to Table=off which I did in the GRE/WG config)
UPDATE
Looks like the external flag in the vx0 setup line is causing dramas.
If I change it to the below, and also change the static FDB entry to the remote wg0 IP rather than the remote ens18 IP, I can ping host to host:
ip link add vx0 type vxlan dstport 4789 vni 1000 local $wgip dev wg0
Then it looks like untagged traffic goes via VNI 1000, and VIDs 10, 20 and 30 are mapped to their respective VNIs.
What does the external flag do that would be preventing the original config from working? Is it the lack of explicit mapping of VID 1?
