I'm having an issue where kube-apiserver.service will always fail on my local Fedora 36.
Getting the namespaces from a context I was experiencing certificate issues that prevented me from succeeding. I was using kubens and getting the error:
> error: You must be logged in to the server (Unauthorized)
> error getting namespace list
First thing I checked my ~/.kube/config and everything seemed alright. So after some reading, and convinced that it was a certificate error (we were experiencing certificates errors with a particular kube cluster) I installed kubeadm via yum (sudo yum install kubernetes-kubeadm.x86_64 ). I used it to automatically renew all of the certificates that needed it, with the command kubeadm certs renew all .
The command came out with a clean output, no error signaled. Checking kubens still gives the same error. So I tried restarting the kube services, and all restarted fine except for kube-apiserver. It always gets the same error, too many restart commands repeated too quickly.
This is the output of sudo systemctl status kube-apiserver -l :
> × kube-apiserver.service - Kubernetes API Server
> Loaded: loaded (/usr/lib/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled)
> Active: failed (Result: exit-code) since Thu 2022-11-17 09:07:44 CET; 12min ago
> Docs: https://kubernetes.io/docs/concepts/overview/components/#kube-apiserver
> https://kubernetes.io/docs/reference/generated/kube-apiserver/
> Process: 1752 ExecStart=/usr/bin/kube-apiserver $KUBE_LOGTOSTDERR $KUBE_LOG_LEVEL $KUBE_ETCD_SERVERS $KUBE_API_ADDRESS $KUBE_API_PORT
> $KUBELET_PORT > Main PID: 1752 (code=exited, status=1/FAILURE)
> CPU: 48ms
>
> Nov 17 09:07:44 fedora systemd[1]: kube-apiserver.service: Scheduled
> restart job, restart counter is at 5. Nov 17 09:07:44 fedora
> systemd[1]: Stopped kube-apiserver.service - Kubernetes API Server.
> Nov 17 09:07:44 fedora systemd[1]: kube-apiserver.service: Start
> request repeated too quickly. Nov 17 09:07:44 fedora systemd[1]:
> kube-apiserver.service: Failed with result 'exit-code'. Nov 17
> 09:07:44 fedora systemd[1]: Failed to start kube-apiserver.service -
> Kubernetes API Server.
So I looked into journalctl and I'm finding this log section:
> Nov 16 16:33:30 fedora audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0
> msg='unit=kube-apiserver comm="systemd" exe="/usr/lib/systemd/systemd"
> hostname=? addr=? terminal=? res=failed'
> Nov 16 16:33:30 fedora systemd[1]: kube-apiserver.service: Scheduled restart job, restart counter is at 5.
> ░░ Automatic restarting of the unit kube-apiserver.service has been scheduled, as the result for
> Nov 16 16:33:30 fedora systemd[1]: Stopped kube-apiserver.service - Kubernetes API Server.
> ░░ Subject: A stop job for unit kube-apiserver.service has finished
> ░░ A stop job for unit kube-apiserver.service has finished.
> Nov 16 16:33:30 fedora audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0
> msg='unit=kube-apiserver comm="systemd" exe="/usr/lib/systemd/systemd"
> hostname=? addr=? terminal=? res=success'
> Nov 16 16:33:30 fedora audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0
> msg='unit=kube-apiserver comm="systemd" exe="/usr/lib/systemd/systemd"
> hostname=? addr=? terminal=? res=success'
> Nov 16 16:33:30 fedora systemd[1]: kube-apiserver.service: Start request repeated too quickly.
> Nov 16 16:33:30 fedora systemd[1]: kube-apiserver.service: Failed with result 'exit-code'.
> ░░ The unit kube-apiserver.service has entered the 'failed' state with result 'exit-code'.
> Nov 16 16:33:30 fedora systemd[1]: Failed to start kube-apiserver.service - Kubernetes API Server.
> ░░ Subject: A start job for unit kube-apiserver.service has failed
> ░░ A start job for unit kube-apiserver.service has finished with a failure.
> Nov 16 16:33:37 fedora kubelet[8800]: --rotate-certificates <Warning: Beta feature> Auto rotate the kubelet client certificates by
> requesting new certificates from the kube-apiserver when the
> certificate expiration approaches. (DEPRECATED: This parameter should
> be set via the config file specified by the Kubelet's --config flag.
> See
> https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/
> for more information.)
> Nov 16 16:33:37 fedora kubelet[8800]: --rotate-server-certificates Auto-request and rotate the kubelet serving certificates by requesting
> new certificates from the kube-apiserver when the certificate
> expiration approaches. Requires the RotateKubeletServerCertificate
> feature gate to be enabled, and approval of the submitted
> CertificateSigningRequest objects. (DEPRECATED: This parameter should
> be set via the config file specified by the Kubelet's --config flag.
> See
> https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/
> for more information.)
> Nov 16 16:33:47 fedora kubelet[8818]: --rotate-certificates <Warning: Beta feature> Auto rotate the kubelet client certificates by
> requesting new certificates from the kube-apiserver when the
> certificate expiration approaches. (DEPRECATED: This parameter should
> be set via the config file specified by the Kubelet's --config flag.
> See
> https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/
> for more information.)
> Nov 16 16:33:47 fedora kubelet[8818]: --rotate-server-certificates Auto-request and rotate the kubelet serving certificates by requesting
> new certificates from the kube-apiserver when the certificate
> expiration approaches. Requires the RotateKubeletServerCertificate
> feature gate to be enabled, and approval of the submitted
> CertificateSigningRequest objects. (DEPRECATED: This parameter should
> be set via the config file specified by the Kubelet's --config flag.
> See
> https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/
> for more information.)
> Nov 16 16:33:57 fedora kubelet[8834]: --rotate-certificates <Warning: Beta feature> Auto rotate the kubelet client certificates by
> requesting new certificates from the kube-apiserver when the
> certificate expiration approaches. (DEPRECATED: This parameter should
> be set via the config file specified by the Kubelet's --config flag.
> See
> https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/
> for more information.)
> Nov 16 16:33:57 fedora kubelet[8834]: --rotate-server-certificates Auto-request and rotate the kubelet serving certificates by requesting
> new certificates from the kube-apiserver when the certificate
> expiration approaches. Requires the RotateKubeletServerCertificate
> feature gate to be enabled, and approval of the submitted
> CertificateSigningRequest objects. (DEPRECATED: This parameter should
> be set via the config file specified by the Kubelet's --config flag.
> See
> https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/
> for more information.)
The output of kubectl version is:
> Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.7",
> GitCommit:"e6f35974b08862a23e7f4aad8e5d7f7f2de26c15",
> GitTreeState:"archive", BuildDate:"2022-10-14T00:00:00Z",
> GoVersion:"go1.18.7", Compiler:"gc", Platform:"linux/amd64"}
> Kustomize Version: v4.5.4
> error: You must be logged in to the server (the server has asked for the client to provide credentials)
(yes, it has an error message in it).
Really don't know where to go from here. What would you try to get the kube-apiserver.service back on track?
I tried uninstalling each and every kubernetes package I could find on my system:
sudo rpm -e kubernetes-client-1.24.7-1.fc36.x86_64 kubernetes-1.24.7-1.fc36.x86_64 kubernetes-master-1.24.7-1.fc36.x86_64
kubernetes-node-1.24.7-1.fc36.x86_64
after having removed all kubectl plugins through krew. Then I backed up my .kube/config and changed name to the whole ~/.kube folder.
I reinstalled kubernetes, at this point kubectl version was returning the port 8080 error, and I thought this must be because I don't have a .kube/config yet. I reinstalled krew and my favorite kubectl plugins (ctx, ns, cm) and rebuilt the config for all the kubernetes clusters I need to access (with aws eks update-kubeconfig and kubecm add -f <file> commands). Now kubectl version has a more normal output:
> Client Version: version.Info{Major:"1", Minor:"24",
> GitVersion:"v1.24.7",
> GitCommit:"e6f35974b08862a23e7f4aad8e5d7f7f2de26c15",
> GitTreeState:"archive", BuildDate:"2022-10-14T00:00:00Z",
> GoVersion:"go1.18.7", Compiler:"gc", Platform:"linux/amd64"} Kustomize
> Version: v4.5.4 Server Version: version.Info{Major:"1", Minor:"21+",
> GitVersion:"v1.21.14-eks-fb459a0",
> GitCommit:"b07006b2e59857b13fe5057a956e86225f0e82b7",
> GitTreeState:"clean", BuildDate:"2022-10-24T20:32:54Z",
> GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"} WARNING:
> version difference between client (1.24) and server (1.21) exceeds the
> supported minor version skew of +/-1
running just sudo kube-apiserver gives the output:
> W1117 10:13:55.819927 16008 services.go:37] No CIDR for service
> cluster IPs specified. Default value which was 10.0.0.0/24 is
> deprecated and will be removed in future releases. Please specify it
> using --service-cluster-ip-range on kube-apiserver. I1117
> 10:13:56.031051 16008 serving.go:342] Generated self-signed cert
> (/var/run/kubernetes/apiserver.crt, /var/run/kubernetes/apiserver.key)
> I1117 10:13:56.031063 16008 server.go:558] external host was not
> specified, using 192.168.XX.XX W1117 10:13:56.031069 16008
> authentication.go:526] AnonymousAuth is not allowed with the
> AlwaysAllow authorizer. Resetting AnonymousAuth to false. You should
> use a different authorizer E1117 10:13:56.031184 16008 run.go:74]
> "command failed" err="[--etcd-servers must be specified,
> service-account-issuer is a required flag,
> --service-account-signing-key-file and --service-account-issuer are required flags]"
sudo systemctl status kube-apiserver still shows failed state, and sudo systemctl restart kube-apiserver still results in failure
