Score:0

Server

Apache using up all the memory

Will

6/5/24, 12:51 PM

My site (which as running fine until yesterday - CPU was at 2%, memory at 50%) keeps maxing out the resources - CPU and Memory at 100% and "load" and Disc IO very high.

the site is a php (silverstripe) site running on Ubuntu linux with apache and mysql.

Looking at the memory:

ps aux  | awk '{print $6/1024 " MB\t\t" $11}'  | sort -n

I get 150 entries for apache2 all with 40-90Mb. plus 475Mb for mysql and 331Mb for Java.

So it seems apache is starting too many processes and eating all the resources. Is that right?

If I restart apache, the site comes back up and works fine for an hour or so and then builds up lots of apache entries in ps and crashes.

how can i configure apache to not use up all the resources and avoid the website crashing? How can I find out what is causing the site resources to max out?

Update:

#/etc/apache2/mods-enabled# cat mpm_prefork.conf
<IfModule mpm_prefork_module>
    StartServers             5
    MinSpareServers       5
    MaxSpareServers      10
    MaxRequestWorkers     100
    MaxConnectionsPerChild   0
</IfModule>

Phpinfo

https://gist.github.com/asecondwill/bd9640cae780fa87a3af05be335aa3fb

1 + 8

linux

apache2

HBruijn

6/5/24, 1:15 PM

When you check your logs, the reason for the increased load may simply be more real site visitors; but also look for abuse. Fairly typical abuse avenues are unpatched control panels, e-mail forms and open proxy configurations but abuse comes in many other forms as well. For example: https://serverfault.com/q/1130594/37681

HBruijn

6/5/24, 1:16 PM

Note that simply adding the RSS memory reported by `ps` is over estimating the memory consumption because in each individual apache2 process the RSS includes and counts all shared libraries that are typically loaded only once in memory. - By default most LAMP distributions default to using the [prefork mpm](https://serverfault.com/questions/383526) and mod_php which does not offer the most performance when traffic increases. Switching to other MPM and PHP FPM offers improvement.

Will

6/5/24, 1:28 PM

thanks @HBruijn - it wont be increased genunie traffic, the jump is too big. am I right looking at ps that the problem is too many apache threads / processes? so thats just too much traffic from some bots or something? and thats maxing out memory, triggering swap and maxing out cpu.

Will

6/5/24, 1:42 PM

ps over estimation: it adds up to 7g of memory, server has 4g. and not much else shows in the list, so number accuracy aside, its apache using the memory, and then swap using the cpu, right?

Wilson Hauck

6/5/24, 3:26 PM

Additional DB information request at 30 min uptime, please. OS, Version? # cores, any SSD or NVME devices on MySQL Host server? Post TEXT data on justpaste.it and share the links. From your SSH login root, Text results of: A) SELECT COUNT(*), sum(data_length), sum(index_length), sum(data_free) FROM information_schema.tables; B) SHOW GLOBAL STATUS; after minimum 24 hours UPTIME C) SHOW GLOBAL VARIABLES; D) SHOW FULL PROCESSLIST; E) STATUS; not SHOW STATUS, just STATUS; G) SHOW ENGINE INNODB STATUS; for server workload tuning analysis to provide suggestions.

John Mahowald

6/5/24, 5:13 PM

Please edit your question to add your httpd configuration, especially MPM and PHP configuration. mod_php and prefork is very different from php-fpm and event. Please *do not* respond with these details in comments, the amount of information becomes unreadable without formatting.

Will

6/5/24, 6:11 PM

@JohnMahowald - ok thanks, mpm_prefork.conf added

Wilson Hauck

6/10/24, 11:29 PM

Please consider posting Additional DB information requested for workload analysis after 24 hours of uptime.

Score:1

Server

symcbean

6/5/24, 1:24 PM

This goes way beyond the scope of an answer here. But some pointers might be helpful. First, don't try to measure memory usage using ps - reality is so much more complicated. Better to use 'free'. Limit the number of connection with max_clients.

On a web server, avoiding swap is really critical. If you have it switched on, turn it off now. Also limit the memory overcommit to capacity + 15% (and be prepared to revisit this).

Use mysql_tuner.pl to check your mysql config. Ensure that the memory is capped at a sensible level.

Since your main issue appears to be capacity, do make sure you are running php-fpm, not mod_php, and consider switching out Apache for nginx.

+ 9

Will

6/5/24, 1:26 PM

Thanks for this. Switching to php-fpm and nginx are only going to make 10% max difference though right? We have gone from 50% server capacity to maxing out every 20mins. Feels like something is broken / hacked / misconfigured that needs fixing before just switching setups for optimisation

symcbean

6/5/24, 1:32 PM

Depends - if you are serving predominatly static content it could make 300% difference. If you are tunnelling Java traffic via Apache the difference will be much bigger too.

Will

6/5/24, 1:41 PM

its a php site (silverstripe). I suppose i could just vertically scale the server and see if that fixes it, and then optimise the site until we can turn the server back down. i think though something has gone wrong, rather than just needs a tweak

Andrew Henle

6/5/24, 2:28 PM

*Also limit the memory overcommit to capacity + 15% (and be prepared to revisit this).* IMO there's not much point in allowing **any** memory overcommit unless you want to risk the OOM killer killing the web server process that's the entire reason the system exists in the first place, especially if it's using as much RAM as the OP states. Just disable memory overcommit entirely and make sure you have enough RAM to support what your services need. The nice thing about that is any later non-critical process that wants to hog RAM won't get it and it won't wake up the OOM killer.

symcbean

6/5/24, 2:31 PM

IME 15% is conservative, especially on a PHP host. You can't disable overcommit - you can only limit it to 0%.

Will

6/5/24, 3:38 PM

turning off swap and setting overcommit - ill try these. where do I set them?

symcbean

6/5/24, 3:45 PM

`swapoff -a ; echo vm.overcommit_memory=2 >>/etc/sysctl.conf ; echo overcommit_ratio=15 >> /etc/systctl.conf ; sysctl -p` (memory changes will persist, to make swap change persist you need to edit /etc/fstab)

Wilson Hauck

6/10/24, 12:50 AM

Turning SWAP OFF is never a winner. It is much better to run slow for a few seconds than be faced with Out Of Memory error and termination of customers attempt to use your system. SWAP size should be 20% of RAM or 6GB minimum according to Red Hat - they have been in this long enough to know what is successful.

symcbean

6/10/24, 6:35 PM

Running in swap on a webserver is a recipe for failure and *entirely* unnecessary.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Apache using up all the memory

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.