Score:1

Server

How can I see the memory content of nginx process?

Ronaldo

4/27/23, 12:05 PM

Nginx 1.20.1 server working as a reverse proxy consuming a lot o memory and making the service stop requiring a system reboot. ModSecurity library is also configured on Nginx.

You can see in the next graph that the number of established connections keep constant around 200 while the second graph shows memory continually increasing until it reaches 100%.

When I filter the output of the top command COMMAND=nginx I see:

top - 16:31:18 up 1 day,  1:54,  2 users,  load average: 0.26, 0.57, 0.66
Tasks: 179 total,   2 running, 177 sleeping,   0 stopped,   0 zombie
%Cpu(s):  6.0 us,  0.2 sy,  0.0 ni, 93.4 id,  0.3 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  12010.4 total,    184.6 free,   3818.6 used,   8007.3 buff/cache
MiB Swap:    975.0 total,    968.7 free,      6.3 used.   7858.8 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
26553 nginx     20   0 2624044   2.4g   9004 S   7.0  20.3   1:34.85 nginx
26554 nginx     20   0 2625580   2.4g   8932 S   0.7  20.4   0:18.12 nginx
26555 nginx     20   0 2627508   2.4g   8872 S   0.7  20.4   0:04.52 nginx
10284 root      20   0 2624568   2.4g   7772 S   0.0  20.3   0:29.08 nginx
26556 nginx     20   0 2624572   2.4g   8812 S   0.0  20.3   0:00.93 nginx

Here's the output of pmap -d 26553 command:

26553:   nginx: worker process
Address           Kbytes Mode  Offset           Device    Mapping
0000557a808e7000     164 r---- 0000000000000000 008:00001 nginx
0000557a80910000     824 r-x-- 0000000000029000 008:00001 nginx
0000557a809de000     204 r---- 00000000000f7000 008:00001 nginx
0000557a80a12000       8 r---- 000000000012a000 008:00001 nginx
0000557a80a14000     132 rw--- 000000000012c000 008:00001 nginx
0000557a80a35000     128 rw--- 0000000000000000 000:00000   [ anon ]
0000557a81b7c000  264740 rw--- 0000000000000000 000:00000   [ anon ]
0000557a91e05000 1965680 rw--- 0000000000000000 000:00000   [ anon ]
0000557b09da1000    1536 rw--- 0000000000000000 000:00000   [ anon ]
00007fefb919a000  292672 rwx-- 0000000000000000 000:00000   [ anon ]

I feel it is not normal for Nginx to consume that much memory when working as a reverse proxy. The server has 4 vCPUs and 12GB of memory (it was 6GB until last week when it started starving the server and making it unresponsive). But I'm not sure how to proceed from here.

292

0 + 0

troubleshooting

nginx

memory-usage

AlexD

4/27/23, 1:23 PM

show your config with `nginx -T`.

Ronaldo

4/27/23, 2:52 PM

@AlexD in order not to pollute the question I've added it [here](https://ideone.com/sQ25vU).

AlexD

4/27/23, 2:58 PM

The main suspect is probably mod_security. If you disable it, do you still experience the same memory consumption growth?

AlexD

4/27/23, 3:00 PM

See this [mod_security issue](https://github.com/SpiderLabs/ModSecurity-nginx/issues/236)

Ronaldo

4/27/23, 3:40 PM

@AlexD I haven't disabled mod_security yet as it's the main reason we have that server running. I'm gonna plan to do that test. And thanks for sharing the memory leak link, I'm reading it now and trying to understand if a new version of mod_security has been released since I installed it.

djdomi

4/27/23, 5:51 PM

dont use any kind of nopaste solutions, add it directly to the question, this improves the quality of your question

Ronaldo

4/27/23, 8:03 PM

@AlexD it seems my problem was related to [Michal256](https://github.com/SpiderLabs/ModSecurity-nginx/issues/236#issuecomment-759569912)'s post from the link you provided. I was loading modsecurity for every `server`. I've changed it to the `http` level and it appears to have fixed it. I'm gonna observe for the rest of the week and if it confirms to be the solution, I'll tell you so that you can write an answer to be accepted. Thanks!

Ronaldo

5/3/23, 12:23 PM

Happy New Year, @AlexD. And thanks once again, your advice about modsecurity proved to be right. The server hasn't had any memory problems for the past week. The only change I had to make was to remove several `modsecurity on;` and `modsecurity_rules_file` from server context and add it only once at the http context. You can write the answer to be accepted!

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: How can I see the memory content of nginx process?

TH: ฉันจะดูเนื้อหาหน่วยความจำของกระบวนการ nginx ได้อย่างไร

RO: Cum pot vedea conținutul memoriei procesului nginx?

RU: Как я могу увидеть содержимое памяти процесса nginx?

VI: Làm cách nào tôi có thể xem nội dung bộ nhớ của tiến trình nginx?

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.