Score:0

Apache Getting Killed By OOM

US flag
user1010500

I Have apache2 running on Ubuntu 22. My Apache2 often getting killed by OOM. I have tried to get log and its showing something like this

> root@localhost:~# dmesg -e | grep -i kill [Mar15 03:00] shopping_cart
> invoked oom-killer:
> gfp_mask=0x1100dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), order=0,
> oom_score_adj=0 [  +0.000004]  oom_kill_process.cold+0xb/0x10 [ 
> +0.000001] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/apache2.service,task=shopping_cart,pid=1676,uid=33
> [  +0.000012] Out of memory: Killed process 1676 (shopping_cart)
> total-vm:2398524kB, anon-rss:1834584kB, file-rss:0kB, shmem-rss:0kB,
> UID:33 pgtables:4044kB oom_score_adj:0 [  +4.614029] changelog.k2
> invoked oom-killer:
> gfp_mask=0x1100dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), order=0,
> oom_score_adj=0 [  +0.000003]  oom_kill_process.cold+0xb/0x10 [ 
> +0.000002] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/apache2.service,task=changelog.k2,pid=6244,uid=33 [  +0.000016] Out of memory: Killed process 6244 (changelog.k2)
> total-vm:2398524kB, anon-rss:1856988kB, file-rss:0kB, shmem-rss:0kB,
> UID:33 pgtables:4048kB oom_score_adj:0

I am not getting idea with above logs.

when I check status of apache2 when its get killed I am getting something like this

root@localhost:~# sudo systemctl status apache2
× apache2.service - The Apache HTTP Server
     Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
     Active: failed (Result: oom-kill) since Wed 2023-03-15 03:00:07 UTC; 9min ago
       Docs: https://httpd.apache.org/docs/2.4/
    Process: 620 ExecStart=/usr/sbin/apachectl start (code=exited, status=0/SUCCESS)
    Process: 1746 ExecStop=/usr/sbin/apachectl graceful-stop (code=exited, status=0/SUCCESS)
   Main PID: 736 (code=exited, status=0/SUCCESS)
        CPU: 5min 4.064s

If I restart apache2, its start working fine but after some minutes, its again getting killed

I have tried apache2buddy and its showing me something like this

---------------------------------------------
### GENERAL FINDINGS & RECOMMENDATIONS ###
---------------------------------------------
Apache2buddy.pl report for server: localhost (45.33.10.196):

Settings considered for this report:
[ !! ] *** LOW UPTIME ***.
[ @@ ] The following recommendations may be misleading - apache has been restarted within the last 24 hours.

        Your server's physical RAM:                                   3923 MB
        Remaining Memory after other services considered:             3923 MB
        Apache's MaxRequestWorkers directive:                         115      <--------- Current Setting
        Apache MPM Model:                                             prefork
        Largest Apache process (by memory):                           40 MB
[ !! ]  Your MaxRequestWorkers setting is too high.
        Your recommended MaxRequestWorkers setting (based on available memory) is between 87 and 97. <-- Acceptable Range (90-100% of Remaining RAM)
        Max potential memory usage:                                   4646 MB
        Percentage of TOTAL RAM allocated to Apache:                  118.43  %
        Percentage of REMAINING RAM allocated to Apache:              118.43  %
---------------------------------------------

I have only changed mode_prefork like this

<IfModule mpm_prefork_module>
        StartServers              2
        MinSpareServers           5
        MaxSpareServers          10
        MaxRequestWorkers        115
        MaxConnectionsPerChild   0
</IfModule>

I think its getting because of memory but I am not able to figure out which process of apache is consuming much memory and how I can solve it. I have tried htop and free but its not giving me any clue.

Let me know if someone here can help me for solve the puzzle.

Thanks!

Nikita Kipriyanov avatar
za flag
We need to see your Apache config, obviously. Don't dump the whole config, only the parts you were added/changed. Also, try to use `mod_status` to know what requests it was processing when killed, and then trace requests to whatever program they invoke.
jp flag
Reduce `MaxRequestWorkers` by half, wait for a day to make sure that you are not getting OOMs. Start increasing the number daily until you start getting OOMs again. Reduce to previous value.
paladin avatar
id flag
apache2buddy already told you what the problem is. But you should also read it. "Apache's MaxRequestWorkers directive: 115 <--------- Current Setting", "Your recommended MaxRequestWorkers setting (based on available memory) is between 87 and 97. <-- Acceptable Range (90-100% of Remaining RAM)", "Percentage of TOTAL RAM allocated to Apache: 118.43 %"... Either reduce MaxRequestWorkers, increase RAM or add some SWAP (not recommend).
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.