Score:1

Keep running out of space, where have my harddrive space gone?

mm flag

I'm kind of a Linux noob and have been for almost 15 years, but I am trying to keep it active and learn more.

I have a Intel NUC with 120-ish NVMe SSD storage which I'm using for a home server. It has Ubuntu Server 18.04 installed and the only thing on it should be about 10 Docker containers and their respective volumes. I have Synology NAS shares mounted for larger data (main use is Plex server).

The thing is, the last couple of days Docker applications have been malfunctioning because of no space left on the system. Data size of the Docker volumes on the machine should be about 10 GB and I think there should be considerable free space after doing some Docker pruning, but no.

I googled some commands.

$ df -h
Filesystem                      Size  Used Avail Use% Mounted on
udev                             16G     0   16G   0% /dev
tmpfs                           3.2G  2.5M  3.2G   1% /run
/dev/nvme0n1p2                  117G  102G  8.8G  93% /
tmpfs                            16G     0   16G   0% /dev/shm
tmpfs                           5.0M     0  5.0M   0% /run/lock
tmpfs                            16G     0   16G   0% /sys/fs/cgroup
/dev/loop0                      384K  384K     0 100% /snap/bbk-cli/2
/dev/loop1                       56M   56M     0 100% /snap/core18/2074
/dev/loop2                       33M   33M     0 100% /snap/snapd/12704
/dev/loop3                       33M   33M     0 100% /snap/snapd/12398
/dev/nvme0n1p1                  511M  6.7M  505M   2% /boot/efi
//${NAS_IP}/data              11T  7.0T  3.8T  65% /media/fantasio/data
//${NAS_IP}/backup/fantasio   11T  7.0T  3.8T  65% /media/fantasio/backup
//${NAS_IP}/downloads         11T  7.0T  3.8T  65% /media/fantasio/downloads
//${NAS_IP}/media             11T  7.0T  3.8T  65% /media/fantasio/media
overlay                         117G  102G  8.8G  93% /var/lib/docker/overlay2/15d984587dbd5f06928774e6c6660555df4056cd3aef5f442fdc5f0997528bca/merged
shm                              64M     0   64M   0% /var/lib/docker/containers/b3e49e1538d406829d9bc44e47dc1532c52667bc6dedc819df2bc5da329a6d91/mounts/shm
tmpfs                           3.2G     0  3.2G   0% /run/user/1000
overlay                         117G  102G  8.8G  93% /var/lib/docker/overlay2/0aa3216bc24646335c9ec1f3049c4adf5e41e3cd476cdbc72253f45d71431168/merged
shm                              64M     0   64M   0% /var/lib/docker/containers/0d9cbcaa1fa1f1bd4c45fd9c2889ccb5c68e6c096911242258426a67c2576a25/mounts/shm
overlay                         117G  102G  8.8G  93% /var/lib/docker/overlay2/00616e391ee4c278507b1f97d5354dc9dfc291d1a46d2e3222cea190b75d3cf1/merged
shm                              64M     0   64M   0% /var/lib/docker/containers/34ea6b5f4bc83e1a26ae4d76211c71fad62402717ea564455c2c434c3ec462e4/mounts/shm
overlay                         117G  102G  8.8G  93% /var/lib/docker/overlay2/1bc1aaa9ae65a6f9bf49c4ece1186d8e92d8da2ec94563fc9daeb9a7db217977/merged
shm                              64M  8.0K   64M   1% /var/lib/docker/containers/57fc71220bb1e8838920307471ad1534137fb873576556d70115f25f6a10b0f0/mounts/shm
overlay                         117G  102G  8.8G  93% /var/lib/docker/overlay2/8ecb98e1e23b20cd502f80214a560b394f7b87f1b3b285e8eba0acf87439e521/merged
shm                              64M     0   64M   0% /var/lib/docker/containers/9c451431d36fac9834ba22060fd01fc97011304a8f5785b0ddd2d6b8c2de3b6d/mounts/shm
overlay                         117G  102G  8.8G  93% /var/lib/docker/overlay2/248fc09f5fa0056cf9eebfb06ce989302736639781e89aab6db850decbfb18a6/merged
shm                              64M     0   64M   0% /var/lib/docker/containers/1b236ae6e5b1c7c7d4d98039b1ee1f3d554ee3bce10032d718b0e44f678666d6/mounts/shm
overlay                         117G  102G  8.8G  93% /var/lib/docker/overlay2/f88c99964c992d9bf794a47234c28edb7b5756f2c1787a253ed95e0e9a3db7c5/merged
shm                              64M     0   64M   0% /var/lib/docker/containers/af40d9700810cc2fceb030b2d9d72a84db8f0a5749ce8af7182c49dfdda14c24/mounts/shm
overlay                         117G  102G  8.8G  93% /var/lib/docker/overlay2/dac68542864cb37b0b04aabbd724cbf3c9f810038f7265c7fd2f7caf2e169bb1/merged
shm                              64M     0   64M   0% /var/lib/docker/containers/9efa24d4795fe98fc54e0a0e2a8bbeb543ed0cbe79bc6271a0666ca665b5491d/mounts/shm
overlay                         117G  102G  8.8G  93% /var/lib/docker/overlay2/8f4dd8d257685d3ee9e981ebce5d04f5ea53674a217762a5a0f37147a264467c/merged
shm                              64M     0   64M   0% /var/lib/docker/containers/446b170c5d44ffbc4149cb42c8a8780fddffd679b30738cb6b3b699ce1b17a5e/mounts/shm
overlay                         117G  102G  8.8G  93% /var/lib/docker/overlay2/a7590f1d96c1d44229c03079ff7aab190ec76696d568ce89d9e09a28fb2f1ed8/merged
shm                              64M     0   64M   0% /var/lib/docker/containers/32639971ba9685f49981b407b4e1c0bd08115665d295bdb5e1a59cd67c92f3b5/mounts/shm
overlay                         117G  102G  8.8G  93% /var/lib/docker/overlay2/fb7987ad06b95e2839e67d569f20a69d59654c5fdeb3805b8cd46162d9bbacfe/merged
shm                              64M     0   64M   0% /var/lib/docker/containers/8823a92076272d8f0e9ca400eab182ea908d95752766c274af50c3a8bd1bd225/mounts/shm
overlay                         117G  102G  8.8G  93% /var/lib/docker/overlay2/2cb3da2fded9d4b49822c697c7b09a35c4078403bd645b02463ab3af12d4e761/merged
shm                              64M  4.0K   64M   1% /var/lib/docker/containers/a91598c5939a6f72218ae55c8957ccf354a7ecc29e68647e64cbd5b6fcb64098/mounts/shm
overlay                         117G  102G  8.8G  93% /var/lib/docker/overlay2/1c4bafbb4e3bbbda41b60f6106e7fff2eeab81d6541fd2eb08c021d2df667782/merged
shm                              64M     0   64M   0% /var/lib/docker/containers/03a87f7df36e4fc95885f8810bc1b5acaba3dc40cb3d8a68c523f95303aef048/mounts/shm
overlay                         117G  102G  8.8G  93% /var/lib/docker/overlay2/18888dbc515222774bbfe8429f7b113c85c329e58599427a1854ded8a9f83d73/merged
shm                              64M  4.0K   64M   1% /var/lib/docker/containers/ca716e6d9eef1c05c94599585e1e40c545cabee9e4b61d86ce830c1de35b5b3c/mounts/shm
overlay                         117G  102G  8.8G  93% /var/lib/docker/overlay2/79797757da994272f282cab5a95052fd1fdbfaf0c536fdc71a2f12273958be84/merged
shm                              64M     0   64M   0% /var/lib/docker/containers/78d1c027bba29b333af6e315644d8ee4f181c4268c364a31d3d34fcab9b3fcac/mounts/shm
$ sudo du -cha --max-depth=1 / | grep -E "M|G"
15M     /bin
1.2G    /home
224M    /boot
5.9M    /etc
1.1G    /lib
5.8M    /lib32
32G     /var
2.0G    /usr
409M    /snap
15M     /sbin
du: cannot access '/proc/11858/task/11858/fd/4': No such file or directory
du: cannot access '/proc/11858/task/11858/fdinfo/4': No such file or directory
du: cannot access '/proc/11858/fd/3': No such file or directory
du: cannot access '/proc/11858/fdinfo/3': No such file or directory
2.1G    /swapfile
2.5M    /run

From df I should have about 117 GB storage on /dev/nvme0n1p2 mounted at /, but adding up size from du doesn't really add up to 117 GB. Can you please help me with what's going on? Docker seems to take up a lot of that 32 GB /var space, but I should have at least 70 GB to play with. Thank you in advance!

Artur Meinild avatar
vn flag
Could you try and show output of this command? `sudo du -h --max-depth=1 / 2> /dev/null | sort -hr`
Score:2
vn flag

I have an idea about what might be the problem here - it could be a "classic" issue with mounted volumes.

My theory is that at some point, one of your mounted NAS disks in /media/fantasio/xxx actually didn't mount (maybe your NAS wasn't turned on, took too long to start etc.), and you wrote some data to the folder (it would seem appr. 70 GB of data).

This time however, instead of writing to the NAS disk, you actually wrote to the system disk, in the folder where you would usually mount the NAS (/media/fantasio/xxx).

Now the next time the NAS drives are mounted, the data already written "disappears", and is not visible on the filesystem at all - but it's still there.

To check if this is the case (which I strongly suspect), you should unmount all your NAS disks. Now check each folder where you mount your NAS for existing data, i.e.:

  • /media/fantasio/data
  • /media/fantasio/backup
  • /media/fantasio/downloads
  • /media/fantasio/media

I'm certain that when unmounted, one of these folder has some hidden content - remove it!

This is sort of a duplicate of this Q&A on superuser.com.

mm flag
You were spot on! A mount had failed and data quickly filled up the SSD. Thank you!
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.