Score:0

Unable to access Internet from user-defined docker bridge network hosted on KVM

is flag

I am running Docker from a KVM:

Docker --- (hosted on) ----> KVM Guest ---- (hosted on) ----> KVM host

I created a docker bridge network with the following command:

docker create test

user@ubuntu-vm:~$ docker network inspect test
[
    {
        "Name": "test",
        "Id": "1dc0198e46d04e506a9b98b41bf9bdbd94673dbbb6a184874717cba89166946a",
        "Created": "2023-02-01T08:23:54.773507123Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.27.0.0/16",
                    "Gateway": "172.27.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {},
        "Labels": {}
    }
]

However, I cannot reach out to Internet when using this network (--net test):

user@ubuntu-vm:~$ docker run -it --net bridge busybox ping -c 1 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=114 time=4.304 ms
--- 8.8.8.8 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 4.304/4.304/4.304 ms

user@ubuntu-vm:~$ docker run -it --net test busybox ping -c 1 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
--- 8.8.8.8 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss

Interstingly this issue only happens when the Docker is hosted on KVM guest, which by itself runs on bridge mode. The issue does not repro when either:

  1. Docker is hosted directly on the physical machine (KVM host), or
  2. Docker is hosted on the KVM guest but using the default bridge.

I have confirmed that no firewall is enabled, the iptables also looks normal to me (the same rules work fine on a physical machine (KVM host):

ufil@gg-ubuntu-vm:~$ sudo iptables -t nat -L
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
DOCKER     all  --  anywhere            !localhost/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MASQUERADE  all  --  172.21.0.0/16        anywhere
MASQUERADE  all  --  172.27.0.0/16        anywhere

Chain DOCKER (2 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

I tried to collect the network trace (where src or dest equals "8.8.8.8") via iptables TRACE target on both the KVM guest machine (docker host) and the KVM host machine:

  • On KVM guest/docker host, the packet seems to stuck after FORWARD:
PACKET: 2 54ee0fbb IN=br-1dc0198e46d0 MACSRC=2:42:ac:1b:0:2 MACDST=2:42:59:ad:1b:35 MACPROTO=0800 SRC=172.27.0.2 DST=8.8.8.8 LEN=84 TOS=0x0 TTL=64 ID=43138DF
 TRACE: 2 54ee0fbb nat:PREROUTING:rule:0x35:CONTINUE  -4 -t nat -A PREROUTING -d 8.8.8.8/32 -j TRACE
 TRACE: 2 54ee0fbb nat:PREROUTING:return:
 TRACE: 2 54ee0fbb nat:PREROUTING:policy:ACCEPT
PACKET: 2 f7407f41 IN=br-1dc0198e46d0 OUT=enp1s0 MACSRC=2:42:ac:1b:0:2 MACDST=2:42:59:ad:1b:35 MACPROTO=0800 SRC=172.27.0.2 DST=8.8.8.8 LEN=84 TOS=0x0 TTL=63 ID=43138DF
 TRACE: 2 f7407f41 filter:FORWARD:rule:0x113:JUMP:DOCKER-USER  -4 -t filter -A FORWARD -j DOCKER-USER
 TRACE: 2 f7407f41 filter:DOCKER-USER:return:
 TRACE: 2 f7407f41 filter:FORWARD:rule:0x110:JUMP:DOCKER-ISOLATION-STAGE-1  -4 -t filter -A FORWARD -j DOCKER-ISOLATION-STAGE-1
 TRACE: 2 f7407f41 filter:DOCKER-ISOLATION-STAGE-1:rule:0x101:JUMP:DOCKER-ISOLATION-STAGE-2  -4 -t filter -A DOCKER-ISOLATION-STAGE-1 -i br-1dc0198e46d0 ! -o br-1dc0198e46d0 -j DOCKER-ISOLATION-STAGE-2
 TRACE: 2 f7407f41 filter:DOCKER-ISOLATION-STAGE-2:return:
 TRACE: 2 f7407f41 filter:DOCKER-ISOLATION-STAGE-1:return:
 TRACE: 2 f7407f41 filter:FORWARD:rule:0xf6:ACCEPT  -4 -t filter -A FORWARD -i br-1dc0198e46d0 ! -o br-1dc0198e46d0 -j ACCEPT
  • on KVM host, there is no trace generated matching the filter condition, which is different from what happened when I used the default bridge, in which case I can see traces indicating the packet being forwarded outbound and the returning packet coming inbound.

I am wondering what might have caused this problem and how I can further debug?

I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.