I have a Xen VM running Debian Linux with a weirdly large amount of RAM used. What might be the culprit? I believe I've ruled out the usual suspects of disk caching, kernel slab usage, and memory ballooning.
free -h shows:
total used free shared buff/cache available
Mem: 3.8Gi 3.6Gi 87Mi 0.0Ki 55Mi 33Mi
Swap: 1.0Gi 275Mi 748Mi
So almost all of the VM's 3.8 GiB of RAM is used, and only a tiny fraction of that usage is from the kernel's disk cache.
smem -kt doesn't attribute the memory to any process:
PID User Command Swap USS PSS RSS
⋮
---------------------------------------------
52 1 214.4M 128.4M 133.2M 146.6M
smem -w -kt says the memory belongs to the kernel:
Area Used Cache Noncache
firmware/hardware 0 0 0
kernel image 0 0 0
kernel dynamic memory 3.6G -20205568.0 3.6G
userspace memory 133.6M 77.6M 56.0M
free memory 87.8M 87.8M 0
---------------------------------------------
3.8G 146.1M 3.6G
(Also, that negative number seems weird.)
/proc/meminfo says, among other things, that the kernel slab allocator is only responsible for about 95 MiB of the RAM usage (and also doesn't really give an indication about what is using the RAM):
MemTotal: 3973392 kB
MemFree: 89144 kB
MemAvailable: 41860 kB
Buffers: 5084 kB
Cached: 37972 kB
SwapCached: 30000 kB
Active: 52356 kB
Inactive: 77832 kB
Active(anon): 30756 kB
Inactive(anon): 56396 kB
Active(file): 21600 kB
Inactive(file): 21436 kB
Unevictable: 80 kB
Mlocked: 80 kB
SwapTotal: 1048572 kB
SwapFree: 775576 kB
Dirty: 104 kB
Writeback: 0 kB
AnonPages: 78660 kB
Mapped: 91004 kB
Shmem: 20 kB
KReclaimable: 28348 kB
Slab: 96740 kB
SReclaimable: 28348 kB
SUnreclaim: 68392 kB
KernelStack: 4288 kB
PageTables: 6768 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 3035268 kB
Committed_AS: 1697316 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 10028 kB
VmallocChunk: 0 kB
Percpu: 2352 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 428032 kB
DirectMap2M: 3799040 kB
DirectMap1G: 0 kB
If I understand the Xen memory stuff correctly, the system hasn't lost much RAM to memory ballooning. /sys/devices/system/xen_memory/xen_memory0/info/current_kb contains 4079228, which matches the system's reported RAM well enough. In the same directory, high_kb and low_kb contain 0 and 16384, respectively. I believe that means the Xen balloon driver has only claimed 16 MiB of RAM.
What am I missing? What else could be using so much of the system's memory?
