Why does System CPU usage degrade so much

Bobby

10/13/23, 4:28 PM

Background: We are running a set of Ubuntu servers that gradually degrade until they reach a point of total unsuitability. What runs on those servers are solely Java Springboot services running on Java8, multiple of them on each server. Those servers run on top of EXSI and have an HaProxy balancer in-front of them to split the load in a roundrobin fashion.

The problem: Over time the System CPU usage goes up until we max all cores and get load around x10 times then what the server should handle.

Observed behaviour: The CPU usage is generally linked to one or two of the service PIDs. Stopping the service leads to a PID from another service becoming the huge cpu user. Stopping all services on the machine gets us to close to none CPU usage. Starting the services back up results in CPU usage hitting the roof again. Time Wait connections are low during the period, usually around 30 - 40. Open files are low and far from set limits. Restarting the VM results in the issue being temporary resolved.

Ubuntu version: Ubuntu 20.04.4 LTS Kernel: 5.4.0-128-generic EXSI: 7.0.3

I hope this is enough information for people to make suggestions at what to look at and what might be the problem.

Thanks you.

1 + 4

linux

ubuntu

Zac67

10/13/23, 5:50 PM

I'm afraid you'd need to debug the service code....

djdomi

10/13/23, 5:53 PM

looks for me also like an code issue.

joeqwerty

10/13/23, 6:13 PM

Over what period of time does this issue escalate? Hours or Days?

Marcel

10/28/23, 7:53 AM

How many cores? How much RAM? Do you see High IO/Wait in your CPUs? I think the code is leaking memory or there is an unreasonable amount of IO interrupting CPU-bound work. Look at my answers about Apache and Nginx for capacity and network stack kernel tuning.

Score:1

Server

生きがい

10/17/23, 1:29 PM

I recommend that you get the logs from the affected server then provide the logs to the support of ubuntu and let them investigate, they most likely to provide feed back on what to do next on your issue.

+ 0

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Why does System CPU usage degrade so much

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.