Score:0

How to automatically kill a Kubernetes pod that uses too much CPU?

I administrate kubernetes clusters running software that is not always perfect. Some have memory leaks, some have cpu leaks, some have both.

I would like to automatically kill the leaking pods. The ones who leak memory are easy, I just put a memory limit and they die eventually when they reach the limit. However CPU seems more tricky. I do not want to put a CPU limit, that will just make them slower and not kill them.

These days I visit regularly the grafana showing the prometheus cpu charts, and once in a while I can spot a pod stuck in an infinite loop, or a pod increasing its CPU usage steadily. I then kill them manually.

I believe no such feature is standard in kubernetes, but I wonder if a plugin exists, or a simple hack.

asktyagi avatar
in flag
check this [link](https://stackoverflow.com/questions/66996065/kubernetes-how-to-restart-pod-on-memory-limit-threshold#:~:text=If%20a%20Container%20allocates%20more,other%20type%20of%20runtime%20failure.)
e741af0d41bc74bf854041f1fbdbf avatar
@asktyagi the link is about memory limits.
asktyagi avatar
in flag
I believe answer is in discussion itself, you can use probes to find container or application health and kubelet will kill it for you based on your policies.
Matt W avatar
ly flag
Do you have any say as to what resides in the pod images either during the build or with a base image? My guess is you're going to need to implement a liveness probe via a command to check CPU usage and let it die if it goes above a threshold.
Abhijith Chitrapu avatar
tr flag
Please check [this](https://stackoverflow.com/questions/52669380/can-you-auto-destroy-a-kubernetes-pod-deployment).
e741af0d41bc74bf854041f1fbdbf avatar
@asktyagi Oh I see. My issue is that the applications claim they are healthy. They just use a lot of CPU.
e741af0d41bc74bf854041f1fbdbf avatar
@MattW I could always edit the images to change the probes, but I would rather implement a Kubernetes plugin than doing that instead. If it doesn't exist already.
e741af0d41bc74bf854041f1fbdbf avatar
@AbhijithChitrapu I'm not sure it's related to my question.
Abhijith Chitrapu avatar
tr flag
As per my knowledge we cannot kill a Pod automatically apart from resource management.
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.