Score:15

Server

failed to run Kubelet: validate service connection: CRI v1 runtime API is not implemented for endpoint

best_of_man

12/14/23, 11:46 PM

I have installed kubelet 1.26.0 on Ubuntu 22.04 using apt install kubelet command, but when I try journalctl -xeu kubelet I get the following result:

░░ 
░░ The unit kubelet.service has entered the 'failed' state with result 'exit-code'.
Dec 14 15:41:16 a systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 86.
░░ Subject: Automatic restarting of a unit has been scheduled
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ Automatic restarting of the unit kubelet.service has been scheduled, as the result for
░░ the configured Restart= setting for the unit.
Dec 14 15:41:16 a systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
░░ Subject: A stop job for unit kubelet.service has finished
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A stop job for unit kubelet.service has finished.
░░ 
░░ The job identifier is 26301 and the job result is done.
Dec 14 15:41:16 a systemd[1]: Started kubelet: The Kubernetes Node Agent.
░░ Subject: A start job for unit kubelet.service has finished successfully
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit kubelet.service has finished successfully.
░░ 
░░ The job identifier is 26301.
Dec 14 15:41:16 a kubelet[18015]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.27. Image garbage collector will get sandbox image information from CRI.
Dec 14 15:41:16 a kubelet[18015]: I1214 15:41:16.367525   18015 server.go:198] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the rem>
Dec 14 15:41:16 a kubelet[18015]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.27. Image garbage collector will get sandbox image information from CRI.
Dec 14 15:41:16 a kubelet[18015]: I1214 15:41:16.371255   18015 server.go:412] "Kubelet version" kubeletVersion="v1.26.0"
Dec 14 15:41:16 a kubelet[18015]: I1214 15:41:16.371272   18015 server.go:414] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Dec 14 15:41:16 a kubelet[18015]: I1214 15:41:16.371499   18015 server.go:836] "Client rotation is on, will bootstrap in background"
Dec 14 15:41:16 a kubelet[18015]: I1214 15:41:16.372757   18015 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
Dec 14 15:41:16 a kubelet[18015]: I1214 15:41:16.373608   18015 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
Dec 14 15:41:16 a kubelet[18015]: I1214 15:41:16.399357   18015 server.go:659] "--cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /"
Dec 14 15:41:16 a kubelet[18015]: I1214 15:41:16.399717   18015 container_manager_linux.go:267] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Dec 14 15:41:16 a kubelet[18015]: I1214 15:41:16.399832   18015 container_manager_linux.go:272] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName>
Dec 14 15:41:16 a kubelet[18015]: I1214 15:41:16.399866   18015 topology_manager.go:134] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
Dec 14 15:41:16 a kubelet[18015]: I1214 15:41:16.399883   18015 container_manager_linux.go:308] "Creating device plugin manager"
Dec 14 15:41:16 a kubelet[18015]: I1214 15:41:16.399940   18015 state_mem.go:36] "Initialized new in-memory state store"
Dec 14 15:41:16 a kubelet[18015]: E1214 15:41:16.402173   18015 run.go:74] "command failed" err="failed to run Kubelet: validate service connection: CRI v1 runtime API is not implemented for endpoint \">
Dec 14 15:41:16 a systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ An ExecStart= process belonging to unit kubelet.service has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 1.
Dec 14 15:41:16 a systemd[1]: kubelet.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ The unit kubelet.service has entered the 'failed' state with result 'exit-code'.
lines 2547-2600/2600 (END)

I don't know what is the problem. How can I get rid of it?

31107

9 + 0

configuration

systemd

systemctl

kubernetes

kubeadm

Score:14

Server

J.R.

12/24/23, 1:42 AM

I have struggled with this a bit and the answer of @Bogd got me onto the right track. Too long to comment, so here are the steps taken to configure a stable Kubernetes 1.26 master on a new Ubuntu 22.04 node using Flannel.

sudo apt-get update
sudo apt install apt-transport-https curl

Install containerd (reference: https://docs.docker.com/engine/install/ubuntu/)

sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install containerd.io

Create containerd configuration

sudo mkdir -p /etc/containerd
sudo containerd config default | sudo tee /etc/containerd/config.toml

Edit /etc/containerd/config.toml

sudo nano /etc/containerd/config.toml set SystemdCgroup = true
sudo systemctl restart containerd

Install Kubernetes

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add
sudo apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"
sudo apt install kubeadm kubelet kubectl kubernetes-cni

Disable swap

sudo swapoff -a

Check and remove any swap entry if exists

sudo nano /etc/fstab

Avoid error "/proc/sys/net/bridge/bridge-nf-call-iptables does not exist" on kubeinit (reference https://github.com/kubernetes/kubeadm/issues/1062). This is not necessary if docker is also installed in step 6.

sudo modprobe br_netfilter
sudo nano /proc/sys/net/ipv4/ip_forward Edit entry in ip_forward file and change to 1. (Or use sysctl -w net.ipv4.ip_forward=1 - thanks to @dpjanes, see comments)

kubeinit for use with Flannel

sudo kubeadm init --pod-network-cidr=10.244.0.0/16

Copy to config as kubadm command says

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Apply Flannel (reference https://github.com/flannel-io/flannel)

kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/v0.20.2/Documentation/kube-flannel.yml

All should be running now:

kubectl get pods --all-namespaces

NAMESPACE      NAME                                  READY   STATUS    RESTARTS   AGE
kube-flannel   kube-flannel-ds-mcjmm                 1/1     Running   0          76s
kube-system    coredns-787d4945fb-fb59g              1/1     Running   0          8m8s
kube-system    coredns-787d4945fb-t25tj              1/1     Running   0          8m8s
kube-system    etcd-kube-master                      1/1     Running   0          8m19s
kube-system    kube-apiserver-kube-master            1/1     Running   0          8m19s
kube-system    kube-controller-manager-kube-master   1/1     Running   0          8m19s
kube-system    kube-proxy-2hz29                      1/1     Running   0          8m8s
kube-system    kube-scheduler-kube-master            1/1     Running   0          8m19s

+ 5

DevMJ

1/19/24, 8:30 AM

This worked for me on s390x arc and kub 1.26.0 version. thanks

0

Reply

dpjanes

2/7/24, 9:55 PM

If step 17 fails use `sysctl -w net.ipv4.ip_forward=1`

0

Reply

Daniel Poirot

2/10/24, 1:04 AM

PRAISE AND GLORY TO J.R. I have tried a dozen poor copies of the same instructions. This is the FIRST set which got me to eight RUNNING pods! Holy carp... I have to stick with 1.24.10 for another app, so, the fully qualified command on line 13 for me, is: sudo apt install kubeadm=1.24.10-00 kubelet=1.24.10-00 kubectl=1.24.10-00 kubernetes-cni

0

Reply

Daniel Poirot

2/10/24, 1:07 AM

It takes all day and it takes a village... The kubeinit command on line 18 results in a Kubernetes 1.24.10 FOR ME, as I intended.

0

Reply

Danielson

2/17/24, 9:52 AM

Conclusion seems to be: don't use `containerd` but use `containerd.io` instead. The former might work, the latter will work

0

Reply

Score:12

Server

Bogd

12/18/23, 7:39 PM

Just encountered the same problem today - seems to be related to this change in K8s 1.26.

The solution would be to install containerd 1.6, but... there doesn't seem to be an APT package for that yet :( . I was only able to find 1.6+ packages for Ubuntu 22.10 and above.

Possible workarounds:

The one listed in the link above - running an older version of the kubelet (1.25)

apt remove --purge kubelet
apt install -y kubeadm kubelet=1.25.5-00

Manually upgrading containerd to 1.6 or above, by downloading and replacing the binaries

wget https://github.com/containerd/containerd/releases/download/v1.6.12/containerd-1.6.12-linux-amd64.tar.gz
tar xvf containerd-1.6.12-linux-amd64.tar.gz
systemctl stop containerd
cd bin
cp * /usr/bin/
systemctl start containerd

I am currently testing both options, and they do seem to work. Hopefully, in the (very near) future, we will get a containerd 1.6+ apt package to make things easier for us :)

Editing to add a third option (which seems to be the easiest so far):

In the Docker repos, there are packages for containerd 1.6 and above. So you can also add the Docker repos, and install containerd.io from there:

sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo   "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install containerd.io

+ 3

user282246

12/21/23, 12:37 PM

The solutions works for me. Done the first variant with 'apt'

0

Reply

ben_wing

1/10/24, 2:19 AM

to keep all k8s tools in sync i executed `apt remove --purge kubelet kubeadm kubectl` `apt autoremove -y #because there were no longer needed packages` `apt install -y kubeadm kubeadm=1.25.5-00 kubelet=1.25.5-00 kubectl=1.25.5-00`

0

Reply

Dean Schulze

2/6/24, 6:27 PM

Tried ben_wing's solution and it failed. I still get `CRI v1 runtime API is not implemented for endpoint`

0

Reply

Score:2

Server

Shubham Jain

2/10/24, 9:38 AM

Kubernetes is deprecating Docker as a container runtime after v1.20 and so Kubernetes latest versions have not written supported CRI API's for docker, we can do this using the below step by using CRI develops by Mirantis for docker:

Install cri-dockerd from below URL:

https://github.com/Mirantis/cri-dockerd

git clone https://github.com/Mirantis/cri-dockerd.git

# Run these commands as root
###Install GO###
wget https://storage.googleapis.com/golang/getgo/installer_linux
chmod +x ./installer_linux
./installer_linux
source ~/.bash_profile

cd cri-dockerd
mkdir bin
go build -o bin/cri-dockerd
mkdir -p /usr/local/bin
install -o root -g root -m 0755 bin/cri-dockerd /usr/local/bin/cri-dockerd
cp -a packaging/systemd/* /etc/systemd/system
sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service
systemctl daemon-reload
systemctl enable cri-docker.service
systemctl enable --now cri-docker.socket

Note: Install each command separately, In bulk execution I got errors sometimes as few commands may need -y or yes to type

Now start the kube admin cluster using below :

kubeadm init --pod-network-cidr=192.168.0.0/16 --cri-socket=unix:///var/run/cri-dockerd.sock

+ 0

Score:2

Server

alonegame

3/21/24, 7:46 AM

Check the /etc/containerd/config.toml configuration file to see if the cri interface is disabled, such as: disabled_plugins = ["cri"]. After viewing, it is found that the configuration disabled_plugins = ["cri"] is changed to disabled_plugins = [""]. Then restart and restart the containerd service: systemctl restart containerd.

+ 0

Score:2

Server

dev.faz

1/20/24, 11:23 AM

Just in case you found this issue, because you got the same error while trying to do stuff with kubeadm < 1.26.

For unknown reasons the kubeadm deb-pkg depends on cri-tools >=1.25.

So since 1.26 version of cri-tools is in the repo, your containerd 1.5 will no longer work.

Possible fixes:

upgrade containerd to 1.6
downgrad/pin cri-tools to 1.25

+ 0

Score:1

Server

Bhargav Patel

3/24/24, 12:09 PM

Try with the following command and make sure your containerd service is running fine

kubeadm init --pod-network-cidr=192.168.0.0/16 --cri-socket=unix:///var/run/cri-dockerd.sock

+ 0

Score:0

Server

maryam da

2/20/24, 8:36 PM

This Solution worked for me:

Check if your containerd version is >= 1.6.x
- If not, reinstall it from docker repository.
remove file : /etc/containerd/config.toml
Restart containerd sservice
restart docker and kubelet services
init kuber cluster

Hope it helps.

+ 0

Score:0

Server

Aleksandr Shkraba

3/3/24, 3:22 PM

Please try update your /etc/contained/conf.toml to the next:

version = 2
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

And restart the contained service.

+ 0

Score:0

Server

H4kunaMatat4

5/8/24, 9:01 AM

I created this account only to reply to this. I only got kubectl 1.27 working after joining @Bogd and @beng_wing comments together.

First:

apt remove --purge kubelet kubeadm kubectl 
apt install -y kubeadm=1.25.5-00 kubelet=1.25.5-00 kubectl=1.25.5-00

Then:

wget https://github.com/containerd/containerd/releases/download/v1.6.12/containerd-1.6.12-linux-amd64.tar.gz
tar xvf containerd-1.6.12-linux-amd64.tar.gz
systemctl stop containerd
cd bin
cp * /usr/bin/
systemctl start containerd

Then, when running kubeadm init

I received the error:

error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR KubeletVersion]: Kubelet version "1.25.5" is lower than kubeadm can support. Please upgrade kubelet

Which was solved with an apt-get update;apt-get upgrade -y and voila, my homelab cluster is up and running :)

+ 0

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: failed to run Kubelet: validate service connection: CRI v1 runtime API is not implemented for endpoint

failed to run Kubelet: validate service connection: CRI v1 runtime API is not implemented for endpoint

Post an answer