This answer is written in attempt to troubleshoot possible causes for "unaccounted" memory usage, that may go beyond what the OP is experiencing.
Background
In Linux, memory usage can largely be attributed to two elements:
- Usage by processes/threads
- Usage by a filesystem
In the case of this question, the OP can't account for all used memory by processes/threads. Hence, it's very likely that the remaining memory is used by the kernel for filesystem operations.
Probable causes
I'm currently aware of 3 main reasons for excessive filesystem memory usage:
- ZFS ARC cache (or similar filesystem cache, that registers as "normal" memory usage)
- Ramdisks (
tmpfs
and similar filesystems)
- Native filesystem compression, combined with excessive disk I/O (so that compression/recompression happens with constant memory consumption as a result)
To troubleshoot these 3 scenarios, I would recommend the following:
1. ZFS ARC cache
Run this command to get detailed information on ZFS ARC cache:
arcstat -a
The stat reported by size
will show up as normal used memory.
From your post, it's obvious that 0.5 GB is used here.
2. Ramdisks
Run this command to get information about any current tmpfs
filesystem ramdisks:
df -hl -t"tmpfs"
Any amount reported by used
will show up as normal used memory.
Ubuntu (and many other Linux variants) have a default ramdisk under /dev/shm
. Applications may use this space, and thus you can easily check with:
ls -ahl /dev/shm
In addition, tmpfs
usage is also shown under shared
in the free
output, so from your post its obvious that 1.8 GB is used here.
3. FS compression
Check for disk IO with this command: (iostat
is a part of the sysstat
package)
iostat
Combine this with knowledge about enabled filesystem compression for the relevant disks. For ZFS, run this command to get compression properties for all Zpools and datasets:
zfs get compression
If you have high disk IO, combined with enabled native compression, this can result in excessive memory reported as normal used memory.