microk8s communication - vxlan.calico, bridge and veth

Maciek Leks

4/3/24, 10:50 AM

I've configured my microk8s instance (one node). Works well. I've started digging in some linux network internals and I was dumbfounded looking at this:

$ ip -c -br link

lo               UNKNOWN        00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP> 
enp0s3           UP             08:00:27:ad:36:a3 <BROADCAST,MULTICAST,UP,LOWER_UP> 
enp0s8           UP             08:00:27:86:35:40 <BROADCAST,MULTICAST,UP,LOWER_UP> 
enp0s9           UP             08:00:27:25:c8:6d <BROADCAST,MULTICAST,UP,LOWER_UP> 
br-e6dc1065d537  DOWN           02:42:92:65:b3:c3 <NO-CARRIER,BROADCAST,MULTICAST,UP> 
vxlan.calico     UNKNOWN        66:57:ed:b9:9c:1c <BROADCAST,MULTICAST,UP,LOWER_UP> 
cali9e5a2199a75@if3 UP             ee:ee:ee:ee:ee:ee <BROADCAST,MULTICAST,UP,LOWER_UP> 
caliec07808c9ad@if3 UP             ee:ee:ee:ee:ee:ee <BROADCAST,MULTICAST,UP,LOWER_UP> 
calid39e8c9a8ec@if3 UP             ee:ee:ee:ee:ee:ee <BROADCAST,MULTICAST,UP,LOWER_UP> 
cali6a901a5bd4a@if3 UP             ee:ee:ee:ee:ee:ee <BROADCAST,MULTICAST,UP,LOWER_UP>

$ brctl show br-e6dc1065d537

bridge name bridge id       STP enabled interfaces
br-e6dc1065d537     8000.02429265b3c3   no

As you can see linux bridge is DOWN, and it does not have anything connected to it, e.g. calico veths.

So I've started sniffing traffic on the vxlan.calico vxlan when I curl some workloads between each other. Nothing there (on that device). No traffic found. As far as I know and I've tested to communicate veths you need a bridge. To run vxlan over it you need vxlan be connected to the bridge,...But as you see nothing is here.

The only communication I see is on the veths, but how can it be, that two unpaired veths talks to each other?

221

1 + 0

vxlan

microk8s

Score:0

Server

Maciek Leks

4/5/24, 6:59 AM

OK, the truth is quite simple. For pod-to-pod communications with a single node case there is no need to use bridging, so the bridge is DOWN (I suppose), while the whole communications is based on routing routes. ip -br -c route shows the true and the way calico manages routes dynamically.

$ ip -br -c route

default via 10.0.2.2 dev enp0s3 proto dhcp src 10.0.2.15 metric 100 
10.0.2.0/24 dev enp0s3 proto kernel scope link src 10.0.2.15 metric 100 
10.0.2.2 dev enp0s3 proto dhcp scope link src 10.0.2.15 metric 100 
10.0.2.3 dev enp0s3 proto dhcp scope link src 10.0.2.15 metric 100 
blackhole 10.1.238.128/26 proto 80 
10.1.238.138 dev cali6a901a5bd4a scope link 
10.1.238.142 dev caliec07808c9ad scope link 
10.1.238.146 dev cali9e5a2199a75 scope link 
10.1.238.149 dev cali00db5abf42d scope link 
192.168.57.0/24 dev enp0s9 proto kernel scope link src 192.168.57.4 metric 100 
192.168.59.0/24 dev enp0s8 proto kernel scope link src 192.168.59.118 metric 100

One thing puzzled me. If we look at the interfaces all calico veths in the root namespace have got the same MACs: ee:ee:ee:ee:ee:ee, while

$ ips neigh

shows paired veth MACs in container namespaces

10.1.238.149                            cali00db5abf42d  2e:b1:c3:59:e4:09
10.1.238.138                            cali6a901a5bd4a  a6:b8:a9:c3:e7:77
10.1.238.146                            cali9e5a2199a75  6e:c4:7d:a1:3b:d2
10.1.238.142                            caliec07808c9ad  32:dc:93:ed:eb:51

So it means that devices in the root ns don't need "correct" MACs since we've got routings and IP resolution table.

+ 0

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: microk8s communication - vxlan.calico, bridge and veth

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.