I'm trying to set up a kubernetes cluster. My kubeadm init
times out with The kubelet is unhealthy due to a misconfiguration of the node in some way
. Looking at Kubelet I find that is continuously reporting Container runtime network not ready
, Error getting node
, and Nameserver limits exceeded
.
Looking around for similar problems and I find that existing solutions for similar problems dont work or don't make sense:
- one candidate is that I'm missing a pod network addon but that is solved by using
kubectl apply -f
to install the pod network which I cant do because kubectl cant run without a existing cluster (which I cannot create, hence the problem).
- the other candidate is that I havent correctly installed docker for kublet to find it but that doesn't make sense because cri-docker is running and seems to be fine (see below)
- Too many entries in my
/etc/resolve.conf
- I checked and I only have one entry.
● cri-docker.service - CRI Interface for Docker Application Container Engine
Loaded: loaded (/etc/systemd/system/cri-docker.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2022-11-19 17:04:57 UTC; 3h 31min ago
TriggeredBy: ● cri-docker.socket
Docs: https://docs.mirantis.com
Main PID: 1522 (cri-dockerd)
Tasks: 30
Memory: 51.6M
CPU: 41.468s
CGroup: /system.slice/cri-docker.service
└─1522 /usr/local/bin/cri-dockerd --container-runtime-endpoint fd://
sudo journalctl -xeu kubelet
gives me:
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Sat 2022-11-19 19:59:33 UTC; 39min ago
Docs: https://kubernetes.io/docs/home/
Main PID: 33473 (kubelet)
Tasks: 36 (limit: 76768)
Memory: 37.5M
CPU: 39.709s
CGroup: /system.slice/kubelet.service
└─33473 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=remote --container-ru>
Nov 19 20:38:33 master-node kubelet[33473]: E1119 20:38:33.296924 33473 kubelet.go:2448] "Error getting node" err="node \"node\" not found"
...
kubelet.go:2448] "Error getting node" err="node \"node\" not found"
Nov 19 20:38:47 master-node kubelet[33473]: E1119 20:38:47.474190 33473 event.go:276] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"node.172914c534e4e80a">
Nov 19 20:38:47 master-node kubelet[33473]: E1119 20:38:47.501529 33473 kubelet.go:2448] "Error getting node" err="node \"node\" not found"
...
kubelet.go:2448] "Error getting node" err="node \"node\" not found"
Nov 19 20:38:50 master-node kubelet[33473]: E1119 20:38:50.044530 33473 kubelet.go:2373] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin>
Nov 19 20:38:50 master-node kubelet[33473]: W1119 20:38:50.075372 33473 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.RuntimeClass: Get "https://1.2.3.4:6443/apis/node.k8s.io/v1/>
Nov 19 20:38:50 master-node kubelet[33473]: I1119 20:38:50.075518 33473 trace.go:205] Trace[23698761]: "Reflector ListAndWatch" name:vendor/k8s.io/client-go/informers/factory.go:134 (19-Nov-2022 20:38:20.073) (total tim>
Nov 19 20:38:50 master-node kubelet[33473]: Trace[23698761]: ---"Objects listed" error:Get "https://1.2.3.4:6443/apis/node.k8s.io/v1/runtimeclasses?limit=500&resourceVersion=0": dial tcp 1.2.3.4:6443: i/o timeout 30001ms >
Nov 19 20:38:50 master-node kubelet[33473]: Trace[23698761]: [30.00168996s] [30.00168996s] END
Nov 19 20:38:50 master-node kubelet[33473]: E1119 20:38:50.075544 33473 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.RuntimeClass: failed to list *v1.RuntimeClass: Get "https:/>
Nov 19 20:38:50 master-node kubelet[33473]: E1119 20:38:50.120558 33473 kubelet.go:2448] "Error getting node" err="node \"node\" not found"
...
my kube-config.yaml that I pass to kubeadm init --config kube-config.yaml
:
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
token: abcdef.0123456789abcdef
ttl: 24h0m0s
usages:
- signing
- authentication
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 1.2.3.4
bindPort: 6443
nodeRegistration:
criSocket: unix:///var/run/cri-dockerd.sock
imagePullPolicy: IfNotPresent
name: node
taints: null
---
apiServer:
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: registry.k8s.io
kind: ClusterConfiguration
kubernetesVersion: 1.25.0
networking:
dnsDomain: cluster.local
serviceSubnet: 10.96.0.0/12
scheduler: {}
---
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: systemd
What is going wrong here?