Hypothesis: deleted but still opened files
Given the information given so far, and hinted at the presence of a high volume of docker-related files, I would suspect this is caused by deleted-but-still-opened files, that is files that are created by a program, then the program deletes the filesystem path while keeping the file descriptor still opened.
This may be the result of Docker activity on the filesystem.
Principle of solution
- Reveal the deleted files that are still referenced by processes (see below for a way).
- Ideally reveal process names and/or ID so that you have hints about what is happening.
- Close those processes, observe that space is freed.
- If all else fails, just reboot the machine. If space is freed, this is consistent with the hypothesis.
How to reveal the information
The commands below will test the hypothesis by finding and displaying what space is used by what file.
First glimpse
For a basic glimpse, you can issue this:
lsof -n | egrep -w "deleted|^COMMAND"
But this will also list a lot in-memory-only pseudo-files that don't take up any actual storage space.
Example:
COMMAND PID TID TASKCMD USER FD TYPE DEVICE SIZE/OFF NODE NAME
Xorg 1183 root 78u REG 0,1 4 2058 /memfd:xshmfence (deleted)
Xorg 1183 root 85u REG 0,1 4 7182 /memfd:xshmfence (deleted)
Xorg 1183 root 92u REG 0,1 4 7137 /memfd:xshmfence (deleted)
Xorg 1183 root 94u REG 0,1 4 7870 /memfd:xshmfence (deleted)
Filtered simple list
This filters and mostly shows real files:
lsof -F "sn" -lnPX -M | sed -n 's|^n/|/|p' | grep deleted | egrep -v '^/(dev/shm|memfd:|proc)' | LC_ALL=C sort -n | uniq
Example:
/tmp/#someinodenumber (deleted)
Complete information, with size, process and task name
This is more interesting: it will list all files along with the space they occupy in bytes and more.
First, slow part, gather data
# You may want to run this part as root to make sure all is reported
lsof -F "ctsupMin" -lnPX -M >|/tmp/lfosoutput
Then process and format for a nice display, complete and sorted by increasing size
# Can be run as regular user, no need for root
{ echo "SIZE^UID^PID^PROCESS NAME^TASK NAME^INODE^PATH"
</tmp/lfosoutput \
python3 -c $'import sys ; f={}
def g(c): return f.get(c,"(unknown)")
for line in sys.stdin:
c=line[0] ; r=line[1:].rstrip() ; f[c]=r
if c=="n" and f["t"]=="REG" \
and "(deleted)" in f["n"] \
and not f["n"].startswith("/memfd:") \
and not f["n"].startswith("/dev/shm") :
print(f'\''{g("s")}^{g("u")}^{g("p")}^\"{g("c")}\"^\"{g("M")}\"^{g("i")}^{g("n")}'\'')
f={}' \
| LC_ALL=C sort -n | uniq
echo "SIZE^UID^PID^PROCESS NAME^TASK NAME^INODE^PATH"
} | column -t -s '^'
Sample output: a file of 36 megabytes used by Firefox
SIZE UID PID PROCESS NAME TASK NAME INODE PATH
36012032 1234 12345 "Isolated Web Co" "StyleThread#2" 1234567 /tmp/mozilla-temp-12345 (deleted)
SIZE UID PID PROCESS NAME TASK NAME INODE PATH
(Actually there are many lines like these, this is only a sample line.)
Testing if the script indeed reveals such files by creating one
In another terminal, copy-paste this:
# Run python interactive interpreter
python3
# Now in Python
n="/tmp/whatever_file_name_you_want"
f=open(n,mode='a')
import os
os.unlink(n)
f.write("some sentence")
f.flush()
# Don't exit now or the file will really disappear
In the first terminal you can run both steps above (the slow lsof then the formatting part).
And as long as the python process above is alive, this line is reported:
SIZE UID PID PROCESS NAME TASK NAME INODE PATH
13 1000 1387343 "python3" "gdbus" 1308894 /tmp/whatever_file_name_you_want (deleted)
SIZE UID PID PROCESS NAME TASK NAME INODE PATH
You can then exit the python interpreter above (press Control-D or type exit(0)). If you run both parts (the slow lsof then the formatting part) you will observe that the test file no longer appears.
The script above can be modified to write huge amounts of data (like hundreds of gigabytes) and using your usual tools you'll see that the space is indeed freed only after the creating process has closed the file descriptor. Ending the process is enough to ensure the file descriptor is closed.
Back to your case
Running this, you will most probably see process names, task names and files. Either a few big files like images that Docker fetched from network, or a huge number of small files, from Docker again.
Or something else.
Please tell if this helps you.
