Score:1

How to configure a hypervisor such that a KVM/QEMU virtual machine can reach hosts connected to the same trunk bridge?

au flag

1 Problem

My goal is that all LAN traffic is routed through a virtual machine (VM) acting as a router and firewall. Other VMs on the same hypervisor should also be routed through the firewall VM. See the following network sketch. Physical hosts connected to the trunk bridge can ping each other successfully, but VMs could not be reached yet.

2 Details

Network configuration on the hypervisor:

# =======
# Bridges
# =======
# ---------------------
# VLAN bridge br-vlan-3
# ---------------------
ip link add name br-vlan-3 type bridge
ip addr add 10.66.3.11/24 dev br-vlan-3
ip link set dev br-vlan-3 up

# ---------------------
# Trunk bridge Upstream
# ---------------------
ip link add name br-trunk-up type bridge vlan_filtering 1 vlan_default_pvid 0
ip link set br-trunk-up up

# ---------------
# Trunk bridge VM
# ---------------
ip link add name br-trunk-vm type bridge vlan_filtering 1 vlan_default_pvid 0
ip link set br-trunk-vm up

# =====
# Hosts
# =====

# Set physical devices up
ip link set enp5s0 up
ip link set enp9s0f0 up
ip link set enp9s0f1 up
ip link set enp9s0f2 up
ip link set enp9s0f3 up


# ----------------------
# desktop.3 br-vlan-3
# ----------------------
ip link add link enp9s0f1 name enp9s0f1.3 type vlan id 3
ip link set dev enp9s0f1.3 master br-vlan-3
ip link set enp9s0f1.3 up

# --------------------------
# desktop.100 br-trunk-vm
# --------------------------
ip link add link enp9s0f1 name enp9s0f1.100 type vlan id 100
ip link set enp9s0f1.100 master br-trunk-vm
ip link set enp9s0f1.100 up
bridge vlan add vid 100 dev enp9s0f1.100 master
bridge vlan add vid 100 dev br-trunk-vm self

bridge vlan add vid 100 dev fw.100 pvid 100

Network map

=======
DESKTOP
=======
localhost:user ~ $  ip a
2: enp34s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:d8:61:c4:b1:af brd ff:ff:ff:ff:ff:ff
4: br-vlan-100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 1a:b1:36:43:f0:e4 brd ff:ff:ff:ff:ff:ff
    inet 10.66.100.44/24 scope global br-vlan-100
       valid_lft forever preferred_lft forever
7: enp.100@enp34s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-vlan-100 state UP group default qlen 1000
    link/ether 00:d8:61:c4:b1:af brd ff:ff:ff:ff:ff:ff
    
localhost:~ # cat /proc/net/vlan/enp.100 
enp.100  VID: 100        REORDER_HDR: 1  dev->priv_flags: 1221
         total frames received            0
          total bytes received            0
      Broadcast/Multicast Rcvd            0

      total frames transmitted          287
       total bytes transmitted        53964
Device: enp34s0
INGRESS priority mappings: 0:0  1:0  2:0  3:0  4:0  5:0  6:0 7:0
 EGRESS priority mappings: 


==========
HYPERVISOR
==========
bridge name     bridge id               STP enabled     interfaces
br-vlan-100             8000.1ab13643f0e4       no              enp.100

[root@fedora system]# ip a  
4: enp9s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether a0:36:9f:5d:09:19 brd ff:ff:ff:ff:ff:ff
13: br-trunk-vm: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 56:7e:22:6a:23:2d brd ff:ff:ff:ff:ff:ff
16: enp9s0f1.100@enp9s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-trunk-vm state UP group default qlen 1000
    link/ether a0:36:9f:5d:09:19 brd ff:ff:ff:ff:ff:ff
21: fw.100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-trunk-vm state UNKNOWN group default qlen 1000
    link/ether fe:54:00:3e:97:72 brd ff:ff:ff:ff:ff:ff
    
    
[root@fedora system]# brctl show dev br-trunk-vm
bridge name     bridge id               STP enabled     interfaces
br-trunk-vm             8000.567e226a232d       no              enp9s0f1.100
                                                        fw.100
                                                        fw.103
                                                        fw.3


===============
VIRTUAL MACHINE 
===============                                                                    
[root@fedora system]# bridge vlan show
port              vlan-id  
br-trunk-vm       100
enp9s0f1.100      100
virbr0            1 PVID Egress Untagged
fw.100            100 PVID Egress Untagged


2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:3e:97:72 brd ff:ff:ff:ff:ff:ff
    inet 10.66.100.10/24 scope global enp1s0
       valid_lft forever preferred_lft forever

Libvirt XML network configuration of the firewall VM:

<interface type='bridge'>
  <mac address='52:54:00:3e:97:72'/>
  <source bridge='br-trunk-vm'/>
  <target dev='fw.100'/>
  <model type='virtio'/>
  <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</interface>

3 Attempts

I tried configuring both a tagged and untagged NIC on the firewall VM. For the tagged NIC I created a VLAN bridge on the firewall VM and created a virtual tap device from the virtual tap fw.100 with the VLAN 100: [email protected]. Then I set the trunk port of [email protected] to master: bridge vlan add vid 100 dev [email protected] master. For the untagged NIC approach I configured no VLAN on the firewall VM and set its trunk port to PVID 100 untagged bridge vlan add vid 100 dev fw.100 pvid 100 untagged. In both cases I could not ping between the desktop host and the firewall VM.

This issue happens only with the libvirt KVM/QEMU virtual machine. Two physical VLAN tagged hosts connected to the hypervisor's trunk bridge as master can ping each other.

4 Update

It looks like that libvirt does not support Linux bridges with VLANs. According to following article the VMs VLAN stops working if there is a second virtual device on the hypervisor with the same VLAN: https://alesnosek.com/blog/2015/09/07/bridging-vlan-trunk-to-the-guest/

Alternatively openvswitch can be used, yet using a bridge for each VM and physical host is a better approach. The hypervisor then allows forwarding the VMs' and hosts' bridges to the firewall VM's bridge and the remaining routing is then done inside the firewall's VM.

Nikita Kipriyanov avatar
za flag
Yes, libvirt does not support setting vlans for VMs with vlan-aware Linux bridge. Shame on them. The best you can do is to set up VLANs on the virtual bridge ports that belong to VMs by hand with `bridge` utility after the VM is started.
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.