Score:1

Server

How to debug containerLogMaxSize not taking effect on the kubelet?

merlin2011

3/29/24, 2:45 AM

I'm running the following EKS version:

kubelet --version
Kubernetes v1.22.12-eks-ba74326

I've set the following parameter as documented here.

  "containerLogMaxSize": "100 Mi"

I then restarted the kubelet:

systemctl daemon-reload
systemctl restart kubelet

But my logs are still rotating at 10 MB.

Any ideas for how to debug or fix this?

431

2 + 6

kubernetes

amazon-eks

mdaniel

3/29/24, 4:47 AM

I don't believe it tolerates spaces between the number and the unit; did you check the `journalctl -u kubelet` to see if it whined on startup?

merlin2011

3/29/24, 6:54 PM

@mdaniel I do not see any complaining but I do see `FLAG: --container-log-max-size="10Mi"`, so the option is certainly being ignored or overridden somehow. I also tried removing the space between the number and the unit, but that did not do anything different.

mdaniel

3/30/24, 2:49 AM

Hmm, ok, then the next thing I'd check is (as you said) see if EKS's scripts are doing something underhanded; start with `systemctl cat kubelet.service` and chase all the files and environment mentioned. You may have some luck with `grep -r container-log-max /etc` but that may not be the whole story since systemd is a complicated thing

merlin2011

3/30/24, 11:24 PM

My current suspicion is that it's this issue, so I'm going to try changing the runtime and see if that helps: https://github.com/kubernetes/kubernetes/issues/106855#issuecomment-990759441

Kiran Kotturi

4/3/24, 12:55 PM

@merlin2011 as you have mentioned above, can you please confirm if changing the runtime has resolved your issue.

merlin2011

4/4/24, 7:12 PM

@KiranKotturi It did not resolve the issue. :( I'm still investigating.

Score:0

Server

Kiran Kotturi

3/29/24, 5:58 AM

As per the official documentation

ContainerLogMaxSize is a quantity defining the maximum size of the container log file before it is rotated. For example: "5Mi" or "256Ki". If DynamicKubeletConfig is on, when dynamically updating this field, consider that it may trigger log rotation. Default: "10Mi"

If the value mentioned in the containerLogMaxSize is not accepted then by default the log rotation is returning 10 MB of log file.Please update the value as mentioned in the above example as 100Mi (without spaces) as your requirement is 100 MB.

containerLogMaxSize: 100Mi

+ 1

merlin2011

3/29/24, 5:43 PM

How can I tell if it's not accepted?

Score:0

Server

jbert

8/18/24, 3:03 PM

I ran into this same issue and after some digging fixed it with the following.

Systemd was using the dropin file /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf for kubelet startup and this file pointed to a config file /var/lib/kubelet/config.yaml, this is where I made my changes.

$ sudo vi /var/lib/kubelet/config.yaml 
  added the following to the bottom of the file:  
    containerLogMaxFiles: 6  
    containerLogMaxSize: "50Mi"  

Then restarted the kubelet:
$ sudo systemctl daemon-reload && sudo systemctl restart kubelet

Finally I used the following to check the settings, I ran this on the
kubernetes control plane server. Make sure to change NODE_SERVER_NAME to
your node server name:
$ kubectl get --raw "/api/v1/nodes/NODE_SERVER_NAME/proxy/configz"

Easier to read if you have jq installed
$ kubectl get --raw "/api/v1/nodes/NODE_SERVER_NAME/proxy/configz" | jq .

+ 0

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: How to debug containerLogMaxSize not taking effect on the kubelet?

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.