Score:0

Out of Memory Mismatch compared to what system rerports

sb flag
Oct 25 07:41:32 KVM-BOX kernel: memory: usage 255216608kB, limit 255216640kB, failcnt 28058

[root@KVM-BOX ~]# free -hm
              total        used        free      shared  buff/cache   available
Mem:           282G        203G         78G        173M        955M         78G
Swap:            0B          0B          0B           0B          0B          0B 

[root@KVM-BOX ~]# cat /proc/meminfo
MemTotal:       296615980 kB
MemFree:        82997080 kB
MemAvailable:   82829388 kB

Every metric I can see suggests that there should be 296615980kb of memory available, yet OOM seems to think only 255216612kB of memory is available?

# lsmem -a 
RANGE SIZE STATE REMOVABLE BLOCK
0x0000000000000000-0x000000007fffffff 2G online no 0 
0x0000000100000000-0x000000017fffffff 2G online no 2 
0x0000000180000000-0x00000001ffffffff 2G online no 3 
0x0000000200000000-0x000000027fffffff 2G online no 4 
0x0000000280000000-0x00000002ffffffff 2G online no 5 
0x0000004800000000-0x000000487fffffff 2G online no 144 

Memory block size: 2G 
Total online memory: 288G 
Total offline memory: 0B
br flag
what does 'lsmem -a' show?
sb flag
# lsmem -a RANGE SIZE STATE REMOVABLE BLOCK 0x0000000000000000-0x000000007fffffff 2G online no 0 0x0000000100000000-0x000000017fffffff 2G online no 2 0x0000000180000000-0x00000001ffffffff 2G online no 3 0x0000000200000000-0x000000027fffffff 2G online no 4 0x0000000280000000-0x00000002ffffffff 2G online no 5 // 0x0000004800000000-0x000000487fffffff 2G online no 144 Memory block size: 2G Total online memory: 288G Total offline memory: 0B
digijay avatar
mx flag
I just added the output of `lsmem -a` to the question. Please don't post those outputs as comments, thx!
Score:0
cn flag

Read the rest of the log lines you omitted, and check if the kill message matches "Out of memory" or "Memory cgroup out of memory". In the latter case, memory limits are a resource control feature on cgroups.

OOM killer can trigger when the cgroup limit is reached. When a group's quota is exhausted, the least bad option is the same as with the full system: kill something. Free up memory in that group without endangering the system as a whole.

Look at whatever memory cgroups you have on this host.

Particularly systemd: systemd-cgtop --order=memory

Even if there were no cgroups in play, 243 used of 282 GB is under some memory pressure. 86% utilized is getting high. Do some capacity planning. If KVM-BOX is a VM host with large memory allocated to guests, do not start more above this utilization.

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.