Join Log in
Score:0
domsch avatar Ubuntu

Out of memory, "shared" and "buff/cache" consume entire memory

pk flag
domsch

The oom_reaper always kills our application although the memory is not used by the process itself.

free -m, just before the OOM killer terminated the process, indicates that "shared" and "buff/cache" need the entire memory:

root@local:/tmp# free -m
              total        used        free      shared  buff/cache   available
Mem:           3861         627          95        3037        3138          26
Swap:             0           0           0

Clearing the cache by using echo 3 > /proc/sys/vm/drop_caches doesn't do anything.

linuxatemyram.com does not apply here.

I have also already ruled out that the memory is used by a RAM disk/tempfs by using df -t tmpfs --total -h.

meminfo, just before the OOM killer terminated the process, returned the following:

root@local:/tmp# awk '$3=="kB"{$2=$2/1024;$3="MB"} 1' /proc/meminfo | column -t
MemTotal:           3861.26      MB
MemFree:            108.297      MB
MemAvailable:       49.6445      MB
Buffers:            2.82422      MB
Cached:             3069.16      MB
SwapCached:         0            MB
Active:             42.8164      MB
Inactive:           578.258      MB
Active(anon):       2.78125      MB
Inactive(anon):     542.719      MB
Active(file):       40.0352      MB
Inactive(file):     35.5391      MB
Unevictable:        2964.51      MB
Mlocked:            18.6484      MB
SwapTotal:          0            MB
SwapFree:           0            MB
Dirty:              0.09375      MB
Writeback:          0            MB
AnonPages:          513.977      MB
Mapped:             85.9766      MB
Shmem:              2987.97      MB
KReclaimable:       37.6602      MB
Slab:               104.531      MB
SReclaimable:       37.6602      MB
SUnreclaim:         66.8711      MB
KernelStack:        6.625        MB
PageTables:         11.4961      MB
NFS_Unstable:       0            MB
Bounce:             0            MB
WritebackTmp:       0            MB
CommitLimit:        1930.63      MB
Committed_AS:       5638.47      MB
VmallocTotal:       3.35544e+07  MB
VmallocUsed:        27.3516      MB
VmallocChunk:       0            MB
Percpu:             3.23438      MB
HardwareCorrupted:  0            MB
AnonHugePages:      0            MB
ShmemHugePages:     0            MB
ShmemPmdMapped:     0            MB
FileHugePages:      0            MB
FilePmdMapped:      0            MB
HugePages_Total:    0
HugePages_Free:     0
HugePages_Rsvd:     0
HugePages_Surp:     0
Hugepagesize:       2            MB
Hugetlb:            0            MB
DirectMap4k:        179.078      MB
DirectMap2M:        3850         MB

ipcs -m --human returns:

root@local:~# ipcs -m --human

------ Shared Memory Segments --------
key        shmid      owner      perms      size       nattch     status
0x00000000 2          user       600          512K     2          dest
0x00000000 262202     user       600          512K     2          dest

When our application is terminated, the "buff/cache" and "shared" are also released again. The memory profiler also shows no problems. How can I find out which data is in "shared" and "buff/cache"?

System: Ubuntu Server 20.04, 5.11.0-41-generic (HWE)

43
0 + 0
waltinator avatar
it flag
waltinator
"memory allocation" varies rapidly - your stats are from after OOM-Killer saved your system. Read `man ps top strace` to see how to get more information. It's too early in the problem diagnosis to assert the innocence of any component. If you Measure, you will Know.
0
Reply
domsch avatar
pk flag
domsch
I did the stats (free, memstat) just before the OOM killer terminated the process, I updated the the post, thanks for your advice, will look into it.
0
Reply
waltinator avatar
it flag
waltinator
There's some resource being continuously consumed by your "app" that results in memory exhaustion.
0
Reply
waltinator avatar
it flag
waltinator
"`shared`" memory is where "shared libraries" (`*.so`) are loaded, and mapped into the address space of processes that use the libraries. This technique avoids each process having its own copy of the library, and makes programs smaller, makes it easier to update the library, and has been in use since the olden days. "`buff/cache`" memory is lightly held, and used to speed up access to the buffered or cached data. By "lightly held", I mean that the processes using `buff/cache` memory can be told to release it by the memory allocater.
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.