I have been playing with Google Kubernetes Engine cluster recently. I have a question regarding their CNI. I have read from GCP documents and other articles that there is a bridge which all veth interfaces connect to. Basically, for each container, a veth pair is created. One end of it is in container and the other end of it is connected to a bridge device. When containers on the same node communicates with each other, packets exchange is using layer 2 bridge device. This is also how GKE documentation describes.
https://cloud.google.com/kubernetes-engine/docs/concepts/network-overview#pods
https://medium.com/cloudzone/gke-networking-options-explained-demonstrated-5c0253415eba
I created a cluster on Google, I can see there is a bridge device docker0, but there is no interfaces associated with it.
gke-xxxxxxxxx /home/uuuuuuu # brctl show
bridge name bridge id STP enabled interfaces
docker0 8000.0242fd0b0cf4 no
gke-xxxxxxxxxx /home/uuuuuuu #
Then I created a cluster using Virtualbox, I can see interfaces are associated with the bridge device.
[root@k8s-2 ~]# brctl show
bridge name bridge id STP enabled interfaces
cni0 8000.36dae477639c no veth7f6c1f01
vethccd0d71d
vethe63e4285
What I am trying to reason about is why I cannot find the bridge device on Google VMs? Is there a special feature of Linux Kernel used in this scenario?
When I inspect each veth interface on Google VM, they all have the same ip address 10.188.2.1
gke-xxxxxxxxxxxxxxxxxxxxx /home/user.name # ifconfig
docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 169.254.123.1 netmask 255.255.255.0 broadcast 169.254.123.255
ether 02:42:fd:0b:0c:f4 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1460
inet 10.10.1.19 netmask 255.255.255.255 broadcast 0.0.0.0
inet6 fe80::4001:aff:fe0a:113 prefixlen 64 scopeid 0x20<link>
ether 42:01:0a:0a:01:13 txqueuelen 1000 (Ethernet)
RX packets 2192921 bytes 1682211226 (1.5 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1288701 bytes 468627202 (446.9 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 276348 bytes 153128345 (146.0 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 276348 bytes 153128345 (146.0 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
veth27cee774: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1460
inet 10.188.2.1 netmask 255.255.255.255 broadcast 10.188.2.1
inet6 fe80::10b7:98ff:fe2f:2e08 prefixlen 64 scopeid 0x20<link>
ether 12:b7:98:2f:2e:08 txqueuelen 0 (Ethernet)
RX packets 32 bytes 2306 (2.2 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 10 bytes 710 (710.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
veth6eba4cdf: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1460
inet 10.188.2.1 netmask 255.255.255.255 broadcast 10.188.2.1
inet6 fe80::c4e3:b0ff:fe5f:63da prefixlen 64 scopeid 0x20<link>
ether c6:e3:b0:5f:63:da txqueuelen 0 (Ethernet)
RX packets 537091 bytes 138245354 (131.8 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 477870 bytes 122515885 (116.8 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
veth8bcf1494: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1460
inet 10.188.2.1 netmask 255.255.255.255 broadcast 10.188.2.1
inet6 fe80::70cb:c4ff:fe8c:a747 prefixlen 64 scopeid 0x20<link>
ether 72:cb:c4:8c:a7:47 txqueuelen 0 (Ethernet)
RX packets 50 bytes 3455 (3.3 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 28 bytes 2842 (2.7 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
vethbb2135c7: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1460
inet 10.188.2.1 netmask 255.255.255.255 broadcast 10.188.2.1
inet6 fe80::1469:daff:fea0:8b5b prefixlen 64 scopeid 0x20<link>
ether 16:69:da:a0:8b:5b txqueuelen 0 (Ethernet)
RX packets 223995 bytes 82725559 (78.8 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 239258 bytes 60203574 (57.4 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
vetheee4e8e3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1460
inet 10.188.2.1 netmask 255.255.255.255 broadcast 10.188.2.1
inet6 fe80::ec6c:3bff:fef3:70c2 prefixlen 64 scopeid 0x20<link>
ether ee:6c:3b:f3:70:c2 txqueuelen 0 (Ethernet)
RX packets 311669 bytes 40562747 (38.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 304461 bytes 628195110 (599.0 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
What is behind these veth interfaces?
Thanks in advance