Join Log in
Score:1
Mikha avatar Server

kube-apiserver started to die on start-up after a few months of good work - how can I check its logs?

cn flag
Mikha

The stage: a control plane machine, kubernetes 1.24.3 on a baremetal Ubuntu 22.04, installed with kubeadm, there is also one worker node. The whole set-up worked like a charm for 4 months until some unknown silent kaboom yesterday (I actually don't exclude a sudden hardware issue)

The problem: port 6443 is listed by netstat for the first few minutes after the control plane machine start-up, and then disappears. While it's on, the apiserver is irresponsive anyway - any connection attempt to it is reset by the peer. I.e. there should be some serious problems on the kube-apiserver side, but I can't figure out what it's unhappy with.

I checked some obvious things - ip address didn't change, enough disk space, k8s' certificates are not expired. So I need to check kube-apiserver logs somehow.

As for the logs, the official page says:

On systemd-based systems, you may need to use journalctl instead of examining log files.

But... what component should I run journalctl for?? If I run it for kubelet (journalctl -u kubelet), I don't see much of the logs related to apiserver apart from "can't connect to :6443"

And I don't see any service named kube-apiserver or alike when I run e.g. just systemctl... Also, there are no logs in /var/log/ (not surprising since it's a systemd-based system, but I checked nevertheless)

I wonder is there a way to check the apiserver's logs, or is there some gotcha that I'm missing? Would appreciate any help on this subject!

587
1 + 4
Mikha avatar
cn flag
Mikha
@larsks uhhuh! I use containerd, so got it... that's a good lead, thanks. I will try it on Monday, will write the result down here. Thanks!
0
Reply
Mikha avatar
cn flag
Mikha
@larsks You were absolutely right. Looking into pod logs on the control plane helped. TL/DR: etcd data corruption So logs for a pod with kube-apiserver I saw this: "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused" 2379 turns out to be an ETCD port, and its pod log is showing this: "error":"walpb: crc mismatch". There is enough disk space. Might be a bug somewhere, but probably a HW issue, as noticed e.g. here: https://groups.google.com/g/coreos-dev/c/fRm2o58_N1U?pli=1 As I understand, I can't recover the etcd, so reinstall it is.
1
Reply
Mikha avatar
cn flag
Mikha
@larsks if you put what you already wrote as an answer, I'll be happy to mark it as an accepted one. Thanks!
0
Reply
pt flag
larsks
Glad it helped! I've moved my comment to an answer.
0
Reply
Score:1
avatar Server
pt flag
larsks

On my clusters, kube-apiserver runs in a pod (and logs are available via kubectl logs).

If the apiserver was down, I would log into a node directly and use the container runtime to examine the container logs (e.g., crictl logs or docker logs or whatever is appropriate for your system).

For example:

# Find the kube-apiserver pod
root@infra-control-plane:/# crictl pods | grep kube-apiserver
ca01411d447a6       3 days ago          Ready               kube-apiserver-infra-control-plane                  kube-system          0                   (default)

# Find container in that pod
root@infra-control-plane:/# crictl ps --pod ca01411d447a6
CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID              POD
44da6bf198244       400e6a4878256       3 days ago          Running             kube-apiserver      0                   ca01411d447a6       kube-apiserver-infra-control-plane

# Look at logs for that container
root@infra-control-plane:/# crictl logs 44da6bf198244 |& tail -2
Trace[1944612718]: ---"Writing http response done" 1254ms (21:59:45.444)
Trace[1944612718]: [1.255113794s] [1.255113794s] END
+ 1
Mikha avatar
cn flag
Mikha
Awesome, thanks @larsks ! Like I wrote, that's exactly what helped. Cheers!
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.