Host has no internet access when VMs are running

How do I set up the network, so that both host and VMs can connect to the internet?

I've set up a server to host several virtual machines using KVM. It is intended to serve a library of downloadable books for blind people (further details below).

The current state is the result of trying to follow a number of tutorials on networking for virtual machines. The aim is to have internet access from both the host and VMs.

Both host and guests run Debian 10. The network card is configured as network bridge br0 with a static address ('interfaces' see below).

At present the VMs are started manually using virsh. When no VMs are running, the host has internet access (for example ping debian.org, get update, wget ...).

Once a VM is started, the VM has internet access using br0. Each VM has a static address. The host then loses internet access. Ping is possible to other machines in the local network as well as the router, but not beyond (either pinging a domain name or an IP address).

Both host and VMs can be reached using ssh from other local machines.

Once the VMs are set to autostart, it is no longer possible to update without shutting down the VMs manually, also the host doesn't connect to a time server. Furthermore ip shows dropped packages.

All of this most likely is the result of my very limited understanding of networks and bridges in particular. I am most grateful for any help!

Here some further information.

Purpose
One VM is supposed to serve users from outside the local network, using an NginX web server. It handles the download of books checked out by users which are stored on a local drive.

The second VM provides a PostgreSQL database server, to be accessed from local workstations only, where the library users and loans are administered.

The host should be accessible by ssh from the local network. Internet access is required for connecting to a time server and to be able to keep the software up to date.

PC
Motherboard: MSI MPG B550 GAMING PLUS
CPU: AMD Ryzen™ 7 3700X
RAM: Corsair DIMM 32 GB DDR4-3200 Kit
HD: Samsung 980 PRO 1 TB, SSD
Graphics card: MSI GeForce GT 710 1GD3H LP

OS
uname -r

4.19.0-17-amd64

lsb_release -a

No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 10 (buster)
Release:    10
Codename:   buster

Network
Until it's moved to the library, the server is at my home office connected to an AVM Fritz!Box 7490 router.

ls /sys/class/net/

br0  enp42s0  lo

cat /etc/network/interfaces

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
iface enp42s0 inet manual

# The br0 bridge settings
auto br0
iface br0 inet static
   bridge_ports enp42s0
      address 192.168.10.50
      network 192.168.10.0
      broadcast 192.168.10.255
      netmask 255.255.255.0
      gateway 192.168.10.1
      dns-nameservers 94.247.43.254 194.36.144.87 192.168.10.1
      bridge_stp off
      bridge_fd 0
      bridge_maxwait 0

(The VMs have adresses 192.168.10.51, 192.168.10.52)

ip -s link show dev br0

3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 2c:f0:5d:e4:36:d5 brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    206602     2218     0       1130    0       177     
    TX: bytes  packets  errors  dropped carrier collsns 
    99981      593      0       0       0       0       

cat /proc/net/dev

  Inter-|   Receive                                                | Transmit
   face |   bytes packets errs drop fifo frame compressed multicast|    bytes packets errs drop fifo colls carrier compressed
    br0:   210026    2268    0 1138    0     0          0       177    103273     615    0    0    0     0       0          0
  vnet0:  1384510   18903    0    0    0     0          0         0  58389276   40523    0    0    0     0       0          0
     lo:     1840      26    0    0    0     0          0         0      1840      26    0    0    0     0       0          0
enp42s0: 58580534   42260    0   38    0     0          0       289   1467123   19358    0    0    0     0       0          0

route -n

Kernel-IP-Routentabelle
Ziel            Router          Genmask         Flags Metric Ref    Use Iface
0.0.0.0         0.0.0.0         0.0.0.0         U     0      0        0 vnet0
0.0.0.0         192.168.10.1    0.0.0.0         UG    0      0        0 br0
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 vnet0
192.168.10.0    0.0.0.0         255.255.255.0   U     0      0        0 br0

ls /sys/class/net/

br0  enp42s0  lo  vnet0

Your bridge is dropping half the incoming packets! Amazing that you have any connectivity.

I only really see one obvious issue with your configuration and unfortunately it's a badly chosen Debian default:

      bridge_stp off

STP really should be on for any virtual bridge used by libvirt or for VMs. It is way too easy to construct a loop either by accident or intentionally. Which means it also needs to be on at your Fritz!Box, but it most likely already is. The same for whatever switch you plug it into at the library, but again it most likely already has STP on.

1. STP: the manufacturer of Fritz!Boxes confirmed that they don't support STP.

2. Dropped packets: I'll ignore that for the time being, since two linux laptops (Ubuntu Mate, Lubuntu) show appr. the same amount of dropped packets, while not having any obvious connectivity problems.

3. Lacking internet access of the host appears to have been related to connman setting the IP address of vnets by DHCP.

The first hint was, when I looked at the output of ping debian.org after a VM had been started:

PING debian.org (130.89.148.77) 56(84) bytes of data.
From blibu.local (169.254.210.100) icmp_seq=1 Destination Host Unreachable

ip a gave the following output (excerpt)

3: br0: ... inet 192.168.10.50/24
4: vnet0: ... inet 169.254.210.100/16

So ping was using vnet0 with an address different from the local network address space.

After trying a number of different approaches (2nd NIC, macvtap) with the same result, I finally remembered to add all vnets to /etc/connman/main.conf:

NetworkInterfaceBlacklist = vmnet,vboxnet,virbr,ifb,ve-,vb-,br,enp42s0,eno1,vnet0,vnet1,vnet2

Apparently each VM running adds one vnet.

This most likely is not the most elegant or efficient solution, and I'll greatly appreciate any help in improving my setup.

Still, both host and all VMs now have internet access and can get software updates. :-)