High memory usage that cannot be traced to a service or application

In Ubuntu 22.04 I am experiencing a strange issue with memory. I can't seem to find out why my memory usage is as high as it is. When I boot my laptop memory usage is as expected, but over time somethings seems to eat up my memory, but the system does not seem to know what it is used for.

After the laptop has been running for a couple of days and been in sleep over night free -m reports the following:

               total        used        free      shared  buff/cache   available
Mem:           14812        7329        2683        1810        4799        5348
Swap:           2047         416        1631

All applications has been closed, and according to this tool the accumulated memory used by all applications is 1.7 GiB. So what is using the last ~5.5GiB (7.2-1.7) ?

Am I missing something?

This answer is written in attempt to troubleshoot possible causes for "unaccounted" memory usage, that may go beyond what the OP is experiencing.

Background

In Linux, memory usage can largely be attributed to two elements:

• Usage by processes/threads
• Usage by a filesystem

In the case of this question, the OP can't account for all used memory by processes/threads. Hence, it's very likely that the remaining memory is used by the kernel for filesystem operations.

Probable causes

I'm currently aware of 3 main reasons for excessive filesystem memory usage:

1. ZFS ARC cache (or similar filesystem cache, that registers as "normal" memory usage)
2. Ramdisks (tmpfs and similar filesystems)
3. Native filesystem compression, combined with excessive disk I/O (so that compression/recompression happens with constant memory consumption as a result)

To troubleshoot these 3 scenarios, I would recommend the following:

1. ZFS ARC cache

Run this command to get detailed information on ZFS ARC cache:

arcstat -a 

The stat reported by size will show up as normal used memory.

From your post, it's obvious that 0.5 GB is used here.

2. Ramdisks

Run this command to get information about any current tmpfs filesystem ramdisks:

df -hl -t"tmpfs"

Any amount reported by used will show up as normal used memory.

Ubuntu (and many other Linux variants) have a default ramdisk under /dev/shm. Applications may use this space, and thus you can easily check with:

ls -ahl /dev/shm

In addition, tmpfs usage is also shown under shared in the free output, so from your post its obvious that 1.8 GB is used here.

3. FS compression

Check for disk IO with this command: (iostat is a part of the sysstat package)

iostat

Combine this with knowledge about enabled filesystem compression for the relevant disks. For ZFS, run this command to get compression properties for all Zpools and datasets:

zfs get compression

If you have high disk IO, combined with enabled native compression, this can result in excessive memory reported as normal used memory.

Are you using compression on your filesystem or a copy-on-write filesystem like BTRFS or ZFS?

It's possible that the excessive memory usage is due to the kernel trying to (re)compress files in real time or copying /versioning each write when big files are changing often.

Memory pressure becomes especially noticeable if you have large files like images for virtual machines where files change often and are sized in gigabytes instead of megabytes.

You can look into disabling compression on a filesystem/subvolume/volume level or individual file level and or disabling COW (copy on write)

BTRFS:
Its supported removing compression for particular files with the btrfs command:

 property set <file> compression none

to remove COW (copy on write) for a particular folder:

chattr -R +C /directory/of/your/vm_images/

ZFS:
I don't believe you can remove compression on individual files. Disabling compression on a volume will only affect future writes so you will need to copy each file individually for a files to have compression disabled.

zfs set compression=off name_of_zfs_vol