Score:0

Kubelet node not found

us flag

I'm trying to set up a kubernetes cluster. My kubeadm init times out with The kubelet is unhealthy due to a misconfiguration of the node in some way. Looking at Kubelet I find that is continuously reporting Container runtime network not ready, Error getting node, and Nameserver limits exceeded.

Looking around for similar problems and I find that existing solutions for similar problems dont work or don't make sense:

  1. one candidate is that I'm missing a pod network addon but that is solved by using kubectl apply -f to install the pod network which I cant do because kubectl cant run without a existing cluster (which I cannot create, hence the problem).
  2. the other candidate is that I havent correctly installed docker for kublet to find it but that doesn't make sense because cri-docker is running and seems to be fine (see below)
  3. Too many entries in my /etc/resolve.conf - I checked and I only have one entry.
● cri-docker.service - CRI Interface for Docker Application Container Engine
     Loaded: loaded (/etc/systemd/system/cri-docker.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2022-11-19 17:04:57 UTC; 3h 31min ago
TriggeredBy: ● cri-docker.socket
       Docs: https://docs.mirantis.com
   Main PID: 1522 (cri-dockerd)
      Tasks: 30
     Memory: 51.6M
        CPU: 41.468s
     CGroup: /system.slice/cri-docker.service
             └─1522 /usr/local/bin/cri-dockerd --container-runtime-endpoint fd://

sudo journalctl -xeu kubelet gives me:

● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: active (running) since Sat 2022-11-19 19:59:33 UTC; 39min ago
       Docs: https://kubernetes.io/docs/home/
   Main PID: 33473 (kubelet)
      Tasks: 36 (limit: 76768)
     Memory: 37.5M
        CPU: 39.709s
     CGroup: /system.slice/kubelet.service
             └─33473 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=remote --container-ru>

Nov 19 20:38:33 master-node kubelet[33473]: E1119 20:38:33.296924   33473 kubelet.go:2448] "Error getting node" err="node \"node\" not found"
...
kubelet.go:2448] "Error getting node" err="node \"node\" not found"
Nov 19 20:38:47 master-node kubelet[33473]: E1119 20:38:47.474190   33473 event.go:276] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"node.172914c534e4e80a">
Nov 19 20:38:47 master-node kubelet[33473]: E1119 20:38:47.501529   33473 kubelet.go:2448] "Error getting node" err="node \"node\" not found"
...
kubelet.go:2448] "Error getting node" err="node \"node\" not found"
Nov 19 20:38:50 master-node kubelet[33473]: E1119 20:38:50.044530   33473 kubelet.go:2373] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin>
Nov 19 20:38:50 master-node kubelet[33473]: W1119 20:38:50.075372   33473 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.RuntimeClass: Get "https://1.2.3.4:6443/apis/node.k8s.io/v1/>
Nov 19 20:38:50 master-node kubelet[33473]: I1119 20:38:50.075518   33473 trace.go:205] Trace[23698761]: "Reflector ListAndWatch" name:vendor/k8s.io/client-go/informers/factory.go:134 (19-Nov-2022 20:38:20.073) (total tim>
Nov 19 20:38:50 master-node kubelet[33473]: Trace[23698761]: ---"Objects listed" error:Get "https://1.2.3.4:6443/apis/node.k8s.io/v1/runtimeclasses?limit=500&resourceVersion=0": dial tcp 1.2.3.4:6443: i/o timeout 30001ms >
Nov 19 20:38:50 master-node kubelet[33473]: Trace[23698761]: [30.00168996s] [30.00168996s] END
Nov 19 20:38:50 master-node kubelet[33473]: E1119 20:38:50.075544   33473 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.RuntimeClass: failed to list *v1.RuntimeClass: Get "https:/>
Nov 19 20:38:50 master-node kubelet[33473]: E1119 20:38:50.120558   33473 kubelet.go:2448] "Error getting node" err="node \"node\" not found"
...

my kube-config.yaml that I pass to kubeadm init --config kube-config.yaml:

apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 1.2.3.4
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/cri-dockerd.sock
  imagePullPolicy: IfNotPresent
  name: node
  taints: null
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.k8s.io
kind: ClusterConfiguration
kubernetesVersion: 1.25.0
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
scheduler: {}
---
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: systemd

What is going wrong here?

I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.