Join Log in
Score:6
hoolaboy avatar Server

Committed memory in Linux is less than expected

au flag
hoolaboy

One of my linux server's memory utilization is weird. As far as I know larger Committed memory (Committed_AS in /proc/meminfo or kbcommit in sar -r) than application used memory is normal. But my server's commited memory is significantly small.

# cat /proc/meminfo
MemTotal:       32877480 kB
MemFree:         3511812 kB
Buffers:           19364 kB
Cached:         12080112 kB
SwapCached:            0 kB
Active:         22658640 kB
Inactive:        5906936 kB
Active(anon):   16460116 kB
Inactive(anon):     6652 kB
Active(file):    6198524 kB
Inactive(file):  5900284 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:               544 kB
Writeback:             4 kB
AnonPages:      16482208 kB
Mapped:            17228 kB
Shmem:               672 kB
Slab:             529344 kB
SReclaimable:     460220 kB
SUnreclaim:        69124 kB
KernelStack:       12304 kB
PageTables:        51412 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    16438740 kB
Committed_AS:    4484680 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       66424 kB
VmallocChunk:   34359651996 kB
HardwareCorrupted:     0 kB
AnonHugePages:  15376384 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        4096 kB
DirectMap2M:     2093056 kB
DirectMap1G:    31457280 kB

As you can see Committed_AS is about 4.4GB. But one of process is using more than 14GB. This process doesn't have any mmaped file or any shared memory.

# pidstat -r 1 1
Linux 2.6.32-696.16.1.el6.x86_64 (appintprdsearch01)    08/23/21    _x86_64_    (8 CPU)

16:56:34          PID  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
16:56:35          414  19476.24      0.00 17260248 14356476  43.67  isc_mc
# pmap -x 414 | egrep '^Address|^total'
Address           Kbytes     RSS   Dirty Mode   Mapping
total kB        17268588 14363848 14360208
# pmap -x 414 | grep anon | awk '{s+=$3} END {print s}'
14326736

I wonder why this process's used memory does not affect committed memory. Is there something I lost?

# cat /etc/system-release
Red Hat Enterprise Linux Server release 6.8 (Santiago)
# uname -r
2.6.32-696.16.1.el6.x86_64

Thanks in advance!!

EDIT: I tried to show full pmap -x output taken today. but it's 758 lines long and exceed the post limit. So here's the summary. And I found out its anon pages are getting bigger.

# pmap -x 414 | grep -vw anon
414:   /usr/local/search/sf-1v530/bin/isc_mc -license /usr/local/search/sf-1v530/license/license.xml -conf ../config/config_mc.xml -pid /usr/local/search/sf-1v530/pid/isc_mc.pid -log /usr/local/search/sf-1v530/log/isc_mc
Address           Kbytes     RSS   Dirty Mode   Mapping
0000000000400000    4616    1720       0 r-x--  isc_mc
0000000000a81000    1304     716      20 rw---  isc_mc
0000003c54200000     128     108       0 r-x--  ld-2.12.so
0000003c54420000       4       4       4 r----  ld-2.12.so
0000003c54421000       4       4       4 rw---  ld-2.12.so
0000003c54600000    1576     584       0 r-x--  libc-2.12.so
0000003c5478a000    2048       0       0 -----  libc-2.12.so
0000003c5498a000      16      16       8 r----  libc-2.12.so
0000003c5498e000       8       8       8 rw---  libc-2.12.so
0000003c54a00000      92      72       0 r-x--  libpthread-2.12.so
0000003c54a17000    2048       0       0 -----  libpthread-2.12.so
0000003c54c17000       4       4       4 r----  libpthread-2.12.so
0000003c54c18000       4       4       4 rw---  libpthread-2.12.so
0000003c55600000     524      80       0 r-x--  libm-2.12.so
0000003c55683000    2044       0       0 -----  libm-2.12.so
0000003c55882000       4       4       4 r----  libm-2.12.so
0000003c55883000       4       4       4 rw---  libm-2.12.so
0000003c55a00000      84      16       0 r-x--  libz.so.1.2.3
0000003c55a15000    2044       0       0 -----  libz.so.1.2.3
0000003c55c14000       4       4       4 r----  libz.so.1.2.3
0000003c55c15000       4       4       4 rw---  libz.so.1.2.3
0000003c56600000      88      16       0 r-x--  libgcc_s-4.4.7-20120601.so.1
0000003c56616000    2044       0       0 -----  libgcc_s-4.4.7-20120601.so.1
0000003c56815000       4       4       4 rw---  libgcc_s-4.4.7-20120601.so.1
0000003c57200000     928     488       0 r-x--  libstdc++.so.6.0.13
0000003c572e8000    2048       0       0 -----  libstdc++.so.6.0.13
0000003c574e8000      28      28      28 r----  libstdc++.so.6.0.13
0000003c574ef000       8       8       8 rw---  libstdc++.so.6.0.13
00007ffdcbf56000      84      48      48 rw---    [ stack ]
----------------  ------  ------  ------
total kB        18462068 15806748 15802956

This is sorted size of anon map and its count.

# pmap -x 414 | grep -w anon | awk '{s[$2]++} END {for (x in s) {print x,s[x]}}' | sort
-rnk 1
254728 1
225520 1
197220 1
196608 1
131480 2
131072 2
124008 1
65740 3
65536 142
65532 5
65528 3
65524 2
65520 1
65512 2
65508 2
65504 3
65500 3
65488 2
65484 5
65480 2
65476 1
65464 2
65456 1
65452 1
65448 2
65440 4
65436 3
65432 3
65428 2
65424 1
65420 28
65412 1
65404 1
65400 1
65372 1
65192 1
65188 1
64804 1
64636 1
63940 1
63912 1
63792 1
63584 1
63408 1
63400 2
63244 1
63008 2
63004 1
61996 1
61848 1
61792 1
61624 1
60976 1
59940 1
58276 1
57736 1
55992 1
52348 1
51212 1
50084 1
40840 1
24696 1
15452 1
14324 1
13188 1
10396 1
10240 1
9544 1
7800 1
7260 1
7064 1
5596 1
4560 1
3912 1
3744 1
3688 1
3540 1
3216 1
2532 1
2528 2
2292 1
2136 2
2128 1
1952 1
1744 1
1624 1
1596 1
1024 169
900 1
732 1
348 1
344 1
164 1
136 1
132 1
124 1
116 28
112 1
108 2
104 3
100 3
96 4
88 2
84 2
80 1
72 2
60 1
56 2
52 5
48 2
36 3
32 3
28 2
24 2
16 3
12 2
8 4
4 179

EDIT: I've requested customer support to Redhat. I'm afraid I can not receive any anwser as RHEL6 was EOSed. Anyway I'll post here the result.

553
1 + 7
Matthew Ife avatar
jo flag
Matthew Ife
Would be interesting to see the full `pmap -x` output.
0
Reply
John Mahowald avatar
cn flag
John Mahowald
What is that program that apparently does a lot of anon allocations? Is it using malloc() mmap() or some other allocation function to do it?
0
Reply
hoolaboy avatar
au flag
hoolaboy
@John Mahowald As this is commercial program we can not inspect the source code. Only we can do is strace it. This is a kind of search engine. And it is natural that this program use a huge memory.
0
Reply
Matthew Ife avatar
jo flag
Matthew Ife
Another thing to check into is `/proc/414/smaps` it should break down how many of the page allocations are transparent hugepages. My current 'hunch' is that the THP allocation counts as a 4096 byte page in `Committed_AS` when it should be counted as a 2048 kbyte page instead.
0
Reply
Matthew Ife avatar
jo flag
Matthew Ife
I wrote a program to test the THP not counted properly theory and its untrue. So whatever it is that causes this isn't related to that. Might need the smaps output at this point.
0
Reply
hoolaboy avatar
au flag
hoolaboy
@Matthew Ife: Thanks for your advice. What point should I inspect in smaps output?
0
Reply
Matthew Ife avatar
jo flag
Matthew Ife
@hoolaboy It typically breaks down pages by backing store and current arrangement of the mappings pages (if its in swap, etc). I'm hoping it will give a more detailed explanation of what the mappings are and how/where they came from - that might explain the anomalous behaviour of `/proc/meminfo`.
0
Reply
Score:3
John Mahowald avatar Server
cn flag
John Mahowald

Half of this host's memory is in anon private pages. Most of that is the process you identified. Committed_AS relatively low at 14% of MemTotal implies not much is in use yet for actual data.

Operating systems are lazy. Linux will not touch all 14 GB this program asked for. Instead, as this x86 CPU supports 2 MB and 1 GB pages, Linux assigns several of those to this process and pretends they are entirely zeroed. Notice DirectMap showing many hardware pages of 1 GB. HugePages can be configured explicitly by the administrator, but that meminfo shows zero of those configured.

Just this allocation takes very little physical memory, so Committed_AS starts low. If and when the application fills these with real data, it will increase.

Regarding capacity planning, right now this host currently has plenty of memory. Anon and shared memory is limited to half, 36% cached for fast file access, and a few GB free and unused to satisfy immediate memory allocation demands. And of course Committed_AS far under MemTotal

Even though the application's memory allocations fit the size of the host, find out why it was sized that way. If this is an in memory database sized for future growth, that might make sense. Or maybe 32 GB is your medium sized VM / small sized physical server, over sizing may be easier operationally.

0 + 4
Matthew Ife avatar
jo flag
Matthew Ife
Not convinced this is true; submitter shows that their programs 'rss' and 'dirty' fields are 14GB. These pages seemingly have been touched somewhere as they were dirtied. Albeit could be there are pages being shared and/or files in that mapping not accounted for as its being wrapped up in an awk sum.
0
Reply
marcelm avatar
ng flag
marcelm
_"Operating systems are lazy. Linux will not touch all 14 GB this program asked for. [...] Just this allocation takes very little physical memory, so Committed_AS starts low. If and when the application fills these with real data, it will increase."_ - Hmm, the [Linux kernel documentation for Committed_AS](https://www.kernel.org/doc/html/v5.13/filesystems/proc.html#meminfo) seems to contradict this behaviour. Perhaps the documentation is out of date or incorrect, or I'm reading it incorrectly. Do you happen to have sources supporting your explanation?
0
Reply
Matthew Ife avatar
jo flag
Matthew Ife
@marcelm it's possible the answer is hinting that transparent hugepages aren't in included in committed_as calculation, but I'm neither at a box to check VM source code or test for that. I personally feel there's more going on but a fuller pmap might help there.
0
Reply
John Mahowald avatar
cn flag
John Mahowald
I am probably wrong about the specifics, and this is far from a complete discussion of huge pages. Yet, we observe system wide anon pages far in excess of `Committed_AS`. Exploring this in detail may require knowing which application, whether it uses malloc or mmap, and how it arrived at allocating that much. Which doesn't really matter for capacity planning, even if we assume all anonymous pages are to be used it fits easily in a 32 GB sized host.
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.