I am running Docker from a KVM:
Docker --- (hosted on) ----> KVM Guest ---- (hosted on) ----> KVM host
I created a docker bridge network with the following command:
docker create test
user@ubuntu-vm:~$ docker network inspect test
[
{
"Name": "test",
"Id": "1dc0198e46d04e506a9b98b41bf9bdbd94673dbbb6a184874717cba89166946a",
"Created": "2023-02-01T08:23:54.773507123Z",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": {},
"Config": [
{
"Subnet": "172.27.0.0/16",
"Gateway": "172.27.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {},
"Options": {},
"Labels": {}
}
]
However, I cannot reach out to Internet when using this network (--net test):
user@ubuntu-vm:~$ docker run -it --net bridge busybox ping -c 1 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=114 time=4.304 ms
--- 8.8.8.8 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 4.304/4.304/4.304 ms
user@ubuntu-vm:~$ docker run -it --net test busybox ping -c 1 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
--- 8.8.8.8 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
Interstingly this issue only happens when the Docker is hosted on KVM guest, which by itself runs on bridge mode. The issue does not repro when either:
- Docker is hosted directly on the physical machine (KVM host), or
- Docker is hosted on the KVM guest but using the default bridge.
I have confirmed that no firewall is enabled, the iptables also looks normal to me (the same rules work fine on a physical machine (KVM host):
ufil@gg-ubuntu-vm:~$ sudo iptables -t nat -L
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
DOCKER all -- anywhere anywhere ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
DOCKER all -- anywhere !localhost/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
MASQUERADE all -- 172.21.0.0/16 anywhere
MASQUERADE all -- 172.27.0.0/16 anywhere
Chain DOCKER (2 references)
target prot opt source destination
RETURN all -- anywhere anywhere
RETURN all -- anywhere anywhere
I tried to collect the network trace (where src or dest equals "8.8.8.8") via iptables TRACE target on both the KVM guest machine (docker host) and the KVM host machine:
- On KVM guest/docker host, the packet seems to stuck after FORWARD:
PACKET: 2 54ee0fbb IN=br-1dc0198e46d0 MACSRC=2:42:ac:1b:0:2 MACDST=2:42:59:ad:1b:35 MACPROTO=0800 SRC=172.27.0.2 DST=8.8.8.8 LEN=84 TOS=0x0 TTL=64 ID=43138DF
TRACE: 2 54ee0fbb nat:PREROUTING:rule:0x35:CONTINUE -4 -t nat -A PREROUTING -d 8.8.8.8/32 -j TRACE
TRACE: 2 54ee0fbb nat:PREROUTING:return:
TRACE: 2 54ee0fbb nat:PREROUTING:policy:ACCEPT
PACKET: 2 f7407f41 IN=br-1dc0198e46d0 OUT=enp1s0 MACSRC=2:42:ac:1b:0:2 MACDST=2:42:59:ad:1b:35 MACPROTO=0800 SRC=172.27.0.2 DST=8.8.8.8 LEN=84 TOS=0x0 TTL=63 ID=43138DF
TRACE: 2 f7407f41 filter:FORWARD:rule:0x113:JUMP:DOCKER-USER -4 -t filter -A FORWARD -j DOCKER-USER
TRACE: 2 f7407f41 filter:DOCKER-USER:return:
TRACE: 2 f7407f41 filter:FORWARD:rule:0x110:JUMP:DOCKER-ISOLATION-STAGE-1 -4 -t filter -A FORWARD -j DOCKER-ISOLATION-STAGE-1
TRACE: 2 f7407f41 filter:DOCKER-ISOLATION-STAGE-1:rule:0x101:JUMP:DOCKER-ISOLATION-STAGE-2 -4 -t filter -A DOCKER-ISOLATION-STAGE-1 -i br-1dc0198e46d0 ! -o br-1dc0198e46d0 -j DOCKER-ISOLATION-STAGE-2
TRACE: 2 f7407f41 filter:DOCKER-ISOLATION-STAGE-2:return:
TRACE: 2 f7407f41 filter:DOCKER-ISOLATION-STAGE-1:return:
TRACE: 2 f7407f41 filter:FORWARD:rule:0xf6:ACCEPT -4 -t filter -A FORWARD -i br-1dc0198e46d0 ! -o br-1dc0198e46d0 -j ACCEPT
- on KVM host, there is no trace generated matching the filter condition, which is different from what happened when I used the default bridge, in which case I can see traces indicating the packet being forwarded outbound and the returning packet coming inbound.
I am wondering what might have caused this problem and how I can further debug?