Score:0

Where should I start to troubleshoot high load average?

gt flag

I have a kubernetes cluster running with 4 worker nodes (Ubuntu 20.04 with 4 CPU) and I am seeing high load average on all of them. Here is the result of top on one of the nodes: top

I am unsure whether or not this is expected since this node is running 111 containers. Could there be something else causing the high load average or is it simply down to having too many containers and I just need more resources?

us flag
can you please post the number of cores on the node? Also the output of ``` vmstat 1 10 ``` would be very helpful
Score:0
nc flag

Load average is a count of programs running or waiting to run.

Top is nice, but only gives you half the picture -- things running or waiting to run in the cpu. The other half is things waiting for disk I/O to complete.

For disk I/O, atop is nice, hit d to show disk use by process. (You'll typically need to run it as root to get that.)

Score:0
in flag

A typical rule of thumb is that you do not want your load average to be more than the number of cores on your host. That is the ideal situation. Going above that is not always an issue. From my experience load average alone is not always an indicator of a problem. If you have high load, high sys cpu usage, or high iowait, then you are likely in a bad place.

With kubernetes, the one thing I hear a lot is that you need to set resource limits. If you have no limits, then each pod has full access a a node. Right sizing applications is not simple IMO. Especially when everyone seems to trying to be throwing non cloud native apps in k8s.

As a suggestion, you could reduce your workloads, add more workers (scale out), or add more resources to nodes (scale up), or a combination.

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.