Join Log in
Score:0
Joe S avatar Server

kubelet error, status stuck on 'Active: activating (auto-restart)'

it flag
Joe S

I'm facing this kubelet error on my k8s cluster running CentOS 7. This error surfaced after I reboot the cluster nodes recently. I have not faced similar issue previously I reboot the machines.

I tried running swapoff -a to disable swap but it does not fix the problem.

Here's the systemctl status kubelet -l log:

[root@test-master ~]# systemctl status kubelet -l
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: activating (auto-restart) (Result: exit-code) since Mon 2022-05-30 13:59:51 +08; 822ms ago
     Docs: https://kubernetes.io/docs/
  Process: 9325 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=1/FAILURE)
 Main PID: 9325 (code=exited, status=1/FAILURE)

May 30 13:59:51 test-master kubelet[9325]: Insecure values: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_3DES_EDE_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_RC4_128_SHA. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
May 30 13:59:51 test-master kubelet[9325]: --tls-min-version string                                   Minimum TLS version supported. Possible values: VersionTLS10, VersionTLS11, VersionTLS12, VersionTLS13 (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
May 30 13:59:51 test-master kubelet[9325]: --tls-private-key-file string                              File containing x509 private key matching --tls-cert-file. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
May 30 13:59:51 test-master kubelet[9325]: --topology-manager-policy string                           Topology Manager policy to use. Possible values: 'none', 'best-effort', 'restricted', 'single-numa-node'. (default "none") (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
May 30 13:59:51 test-master kubelet[9325]: --topology-manager-scope string                            Scope to which topology hints applied. Topology Manager collects hints from Hint Providers and applies them to defined scope to ensure the pod admission. Possible values: 'container', 'pod'. (default "container") (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
May 30 13:59:51 test-master kubelet[9325]: -v, --v Level                                                  number for the log level verbosity
May 30 13:59:51 test-master kubelet[9325]: --version version[=true]                                   Print version information and quit
May 30 13:59:51 test-master kubelet[9325]: --vmodule pattern=N,...                                    comma-separated list of pattern=N settings for file-filtered logging (only works for text log format)
May 30 13:59:51 test-master kubelet[9325]: --volume-plugin-dir string                                 The full path of the directory in which to search for additional third party volume plugins (default "/usr/libexec/kubernetes/kubelet-plugins/volume/exec/") (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
May 30 13:59:51 test-master kubelet[9325]: --volume-stats-agg-period duration                         Specifies interval for kubelet to calculate and cache the volume disk usage for all pods and volumes.  To disable volume calculations, set to a negative number. (default 1m0s) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)

This is the content of /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf mentioned in previous log:

[root@test-master ~]# cat  /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

I've also tried adding Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --fail-swap-on=false" to the file mentioned here along with systemctl daemon-reload and systemctl restart kubelet but it doesn't help.

Unfortunately, I'm not exactly sure about the server's version but it should be the same as my client version [v1.23.3].

[root@test-master ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:25:17Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server 10.17.98.171:6443 was refused - did you specify the right host or port?

Is there anyway I can salvage my cluster without having to reset the entire cluster? I'm hoping I can access my deployments previously running in the cluster.

Update:

I tried to find the error messages in the log using journalctl -fu kubelet and this is the closest thing I can find.

May 31 08:58:12 test-master systemd[1]: kubelet.service holdoff time over, scheduling restart.
May 31 08:58:12 test-master systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
May 31 08:58:12 test-master systemd[1]: Started kubelet: The Kubernetes Node Agent.
May 31 08:58:12 test-master kubelet[5280]: Error: failed to parse kubelet flag: unknown flag: --network-plugin

Also, here's my kubelet's version Kubernetes v1.24.1.

Seems like it's related to this issue on GitHub, https://github.com/kubernetes/website/issues/33640.

70
0 + 0
Joe S avatar
it flag
Joe S
@SYN Thanks for the suggestion, that helped me narrow down the issue.
1
Reply
SYN avatar
hk flag
SYN
kubernetes API is unreachable, according to your "kubectl version". Check your kubernetes API service (or pod). Then check "journalctl -fu kubelet", which would be more telling than the systemctl status kubelet you shared so far. There would be lots of logs, look for some "E" prefix (errors).
0
Reply
Score:0
Ric L. avatar Server
us flag
Ric L.

Im getting similar issues here. The other issue with Port 6443 was refused.

I also tried running swapoff -a, stop and retstart kubelet. Checked Network, kubernetes config file, etc.

#systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: activating (auto-restart) (Result: exit-code) since Wed 2023-05-24 13:07:50 UTC; 7s ago
       Docs: https://kubernetes.io/docs/home/
    Process: 111682 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=1/FAILURE)
   Main PID: 111682 (code=exited, status=1/FAILURE)

#kubectl version --short
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.27.1
Kustomize Version: v5.0.1
The connection to the server k8s-master.dcs2.local:6443 was refused - did you specify the right host or port?
+ 2
in flag
Gerald Schneider
Is this supposed to be a solution? If yes, it's hard to see it, you should improve the answer. If you have a problem yourself you should use the `Ask question` button to ask a new question.
0
Reply
HBruijn avatar
in flag
HBruijn
If you have a new question, please ask it by clicking the [Ask Question](https://serverfault.com/questions/ask) button. Include a link to this question if it helps provide context. - [From Review](/review/late-answers/552084)
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.