After some investigations, I found that cri-dockerd service was missing some args:
CGroup: /system.slice/cri-docker.service
└─1098 /usr/local/bin/cri-dockerd --container-runtime-endpoint fd:// --network-plugin=
I added them manually to /etc/systemd/system/cri-docker.service:
...
ExecStart=/usr/local/bin/cri-dockerd --network-plugin=cni --cni-bin-dir=/opt/cni/bin --cni-cache-dir=/var/lib/cni/cache --cni-conf-dir=/etc/cni/net.d --pod-infra-container-image=k8s.gcr.io/pause:3.7
...
Reload service:
sudo systemctl daemon-reload
sudo systemctl restart cri-docker.service
at this point cri-dockerd is configured correctly, but the problem persists, later I noticed that /opt/cni/bin is empty (no container networking plugins) :
masterzulu@master-zulu:~$ sudo /usr/local/bin/cri-dockerd
INFO[0000] Connecting to docker on the Endpoint unix:///var/run/docker.sock
INFO[0000] Start docker client with request timeout 0s
INFO[0000] Hairpin mode is set to none
ERRO[0000] Error validating CNI config list ({
"name": "cbr0",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
): [failed to find plugin "portmap" in path [/opt/cni/bin]]
INFO[0000] Docker cri networking managed by network plugin kubernetes.io/no-op
...
INFO[0000] Setting cgroupDriver cgroupfs
INFO[0000] Docker cri received runtime config &RuntimeConfig{NetworkConfig:&NetworkConfig{PodCidr:,},}
INFO[0000] Starting the GRPC backend for the Docker CRI interface.
INFO[0000] Start cri-dockerd grpc backend
I think I deleted /opt/cni/bin by mistake, so I added its contents again (get the Latest release):
cd /tmp && mkdir cni-plugins && wget https://github.com/containernetworking/plugins/releases/download/v1.1.1/cni-plugins-linux-amd64-v1.1.1.tgz && cd cni-plugins && tar zxfv ../cni-plugins-linux-amd64-v1.1.1.tgz
sudo cp /tmp/cni-plugins/* /opt/cni/bin/
ls /opt/cni/bin
bandwidth bridge dhcp firewall flannel host-device host-local ipvlan loopback macvlan portmap ptp sbr static tuning vlan vrf
after restarting cri-docker service, everything start working as expected:
masterzulu@master-zulu:~$ kubectl get pods -Ao wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
django-space django-588cb669d4-4zz7f 1/1 Running 0 11s 10.244.0.4 master-zulu
django-space postgres-deployment-b58d5ff94-scmrx 1/1 Running 0 12s 10.244.0.5 master-zulu
kube-system coredns-6d4b75cb6d-rnjlm 1/1 Running 0 73m 10.244.0.2 master-zulu
kube-system coredns-6d4b75cb6d-s6zl7 1/1 Running 0 73m 10.244.0.3 master-zulu
cni0 is up:
masterzulu@master-zulu:~$ ifconfig -a
cni0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 10.244.0.1 netmask 255.255.255.0 broadcast 10.244.0.255
inet6 fe80::8c8:84ff:fe78:d999 prefixlen 64 scopeid 0x20<link>
ether 0a:c8:84:78:d9:99 txqueuelen 1000 (Ethernet)
RX packets 27714 bytes 5010722 (5.0 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 26936 bytes 2898949 (2.8 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
cri-docker status:
masterzulu@master-zulu:~$ sudo systemctl status cri-docker
● cri-docker.service - CRI Interface for Docker Application Container Engine
Loaded: loaded (/etc/systemd/system/cri-docker.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2022-05-27 22:39:06 BST; 1h 57min ago
TriggeredBy: ● cri-docker.socket
Docs: https://docs.mirantis.com
Main PID: 187399 (cri-dockerd)
Tasks: 11
Memory: 17.1M
CGroup: /system.slice/cri-docker.service
└─187399 /usr/local/bin/cri-dockerd --network-plugin=cni --cni-bin-dir=/opt/cni/bin --cni-cache-dir=/var/lib/cni/cache --cni-conf-dir=/etc/cni/net.d --po>
May 28 00:36:20 master-zulu cri-dockerd[187399]: time="2022-05-28T00:36:20+01:00" level=info msg="Using CNI configuration file /etc/cni/net.d/10-flannel.conflist"
my conclusion
the absence of --network-plugin=cni in cri-dockerd startup args or any other problem in CNI configs may cause this problem where the cri-docker considers that the CNI is missing and uses the interface docker0 directly so the pods get thier IP from this range 172.17.0.x.
Hope this helps anyone having the same problem.
