/var/lib/kubelet/pki/kubelet.crt is expired, How to renew it?

The kubernetes cluster in on 1.21.2. The kubelet is also on 1.21.2

kubelet --version
Kubernetes v1.21.2

kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean", BuildDate:"2021-06-16T12:57:56Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}

When i am trying to "kl get no" I am getting below error

kl get no
error: You must be logged in to the server (Unauthorized)
**Note: kl is alias of kubectl**
I checked my /var/lib/kubelet/pki/kubelet.crt and it was expired.

In kubelet.crt, the issuer is below

Subject: CN=aparapu@1591592441
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption

Need help in renew this kubelet.crt.

I've seen this issues in many other forums, and I finally found a solutions that works from me.

First, my issues was purely due to the certificate /var/lib/kubelet/pki/kubelet.crt, which I can see expired with either:

echo -n | openssl s_client -connect localhost:10250 2>&1 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' | openssl x509 -text -noout | grep -A 2 Validity

or

sudo openssl x509 -in /var/lib/kubelet/pki/kubelet.crt -text -noout  | grep -A 2 Validity

First, you need to enable --rotate-certificates=true and --rotate-server-certificates=true in your kubelet. In my case, I installed the cluster using kubeadm, so I can edit the /etc/systemd/system/kubelet.service.d/10-kubeadm.conf, and add the following to KUBELET_EXTRA_ARGS:

Environment="KUBELET_EXTRA_ARGS=--rotate-certificates=true --rotate-server-certificates=true --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"

In general, it is just adding these flags to the kubelet exeuction /usr/bin/kubelet --rotate-certificates=true --rotate-server-certificates=true.

And, reload and restart kubelet with:

sudo systemctl daemon-reload
sudo service kubelet restart

After the restart, I see something like 14114 log.go:172] http: TLS handshake error from 20.0.0.13:57738: no serving certificate available for the kubelet, which indicates that the certificate needs to be added and approve.

Secondly, we need to approve the csr from kubernetes (this is something I've never look at before...):

kubectl get csr

There will see the certificate waiting to be approved, so just approved:

kubectl certificate approve csr-dlcf6

And your cluster should now have the server kubelet certificate renew. To verify again:

echo -n | openssl s_client -connect localhost:10250 2>&1 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' | openssl x509 -text -noout | grep -A 2 Validity

Some notes:

• We have enabled the rotation for both client and server. Client rotation is also part of the automatic cert renew script (https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/)
• After we have enable the rotation the /var/lib/kubelet/pki/kubelet.crt is no longer used, instead the symbolic link /var/lib/kubelet/pki/kubelet-server-current.pem is used and points to the latest rotated certificate.

References:

• https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/
• https://devopstales.github.io/kubernetes/k8s-cert/
• https://kubernetes.io/docs/tasks/tls/certificate-rotation/
• https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#kubelet-serving-certs