So as many are probably aware, the memory statistics reported by ps (and the metric RSS itself) aren't entirely accurate because of shared memory pages.
There is another tool called smem, which correctly reports the proportionate set size for each process - that is each process' memory consumption where the shared memory is divided in a sane way between processes.
From man smem:
smem reports physical memory usage, taking shared memory pages into account.
Unshared memory is reported as the USS (Unique Set Size). Shared memory is
divided evenly among the processes sharing that memory. The unshared memory
(USS) plus a process's proportion of shared memory is reported as the PSS
(Proportional Set Size). The USS and PSS only include physical memory usage.
They do not include memory that has been swapped out to disk.
However, there are a few handy tricks for installing and configuring smem to easily report actual used memory by all processes.
Installing and configuring smem
First, if you want to install smem, you need to have the Universe repository enabled. Make sure this line is uncommented in /etc/apt/sources.list (for Ubuntu 22.04):
deb http://archive.ubuntu.com/ubuntu jammy universe
When you're normally trying to install smem, it wants to install a ton of dependencies (a little over 100 packages), because it has a recommended dependency on python3-matplotlib for generating graphical charts.
Since I don't need this (and anybody with a server probably don't need it either), it's possible to only install the base package with this command:
sudo apt install smem python3-matplotlib-
Notice the dash after python3-matplotlib - this indicates that this packages should not be installed. Another option is to use:
sudo apt install --no-install-recommends smem
This has the same result, which will only install the smem package.
The final trick is to fix the columns, so you can correctly use awk, or export the metrics in a way where you can trust the columns. To do this, we need to move the command column to the rightmost place, since the command field can include additional spaces we can't control.
To fix the column layout and size, smem should be run with these options:
smem -ac "user pid swap maps vss uss pss rss command"
In addition, to include all processes, it has to be run as sudo. I've defined the following alias for smem:
alias smem='sudo smem -ac "user pid swap maps vss uss pss rss command"'
So in the following, when I type the command smem, it does in fact run the command with the above options.
Comparing smem and ps metrics
Now we can correctly compare the metrics of ps aux and smem. I was expecting a "real" used memory of 5.2 GB, but ps aux reported 6.35 GB used. Below are the results from smem.
Count of RSS from smem:
$ smem | awk '{sum +=$8}END{print sum}'
6464644
Count of PSS from smem:
$ smem | awk '{sum +=$7}END{print sum}'
5281005
Now it's clear that the RSS reported is roughly similar (a little higher from smem). But it's also clear that the proportionate set size (PSS) more correctly matches the expected memory usage among processes - 5.2 GB.
So to correctly measure total process memory usage, the PSS metric from smem seems to give the most meaningful result.
