Score:0

Disk Space Eaten & Missing

tr flag

My root drive has a 143.9 GB of space. I have a vbox vdi that 60GB and the operating system, which should be another 30 GB... leaving me 54GB of free space.

Problem:
This morning I randomly got a "disk space full" error message for my root drive.

I immediately deleted a few gigs of old logs... but than that space eaten (not freed) and I still get the error...

I am unable to locate to 54GB of disk space.

This is: root@machine:/# lsblk

NAME                                          MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
loop0                                           7:0    0     4K  1 loop  /snap/bare/5
loop1                                           7:1    0     9M  1 loop  /snap/canonical-livepatch/146
loop2                                           7:2    0 114.9M  1 loop  /snap/core/14056
loop3                                           7:3    0   8.9M  1 loop  /snap/pdfmixtool/864
loop4                                           7:4    0  17.9M  1 loop  /snap/pdftk/9
loop5                                           7:5    0  91.7M  1 loop  /snap/gtk-common-themes/1535
loop6                                           7:6    0  55.6M  1 loop  /snap/core18/2632
loop7                                           7:7    0  55.6M  1 loop  /snap/core18/2620
loop8                                           7:8    0 260.7M  1 loop  /snap/kde-frameworks-5-core18/32
loop9                                           7:9    0   115M  1 loop  /snap/core/13886
sda                                             8:0    0 931.5G  0 disk  
├─sda1                                          8:1    0  95.9G  0 part  
└─sda2                                          8:2    0 835.6G  0 part  /home
sdb                                             8:16   0   1.8T  0 disk  
└─sdb1                                          8:17   0   1.8T  0 part  
  └─luks-8b6272af-069d-4f00-9a33-57472f0aed47 253:0    0   1.8T  0 crypt /run/timeshift/340930/backup
nvme0n1                                       259:0    0 238.5G  0 disk  
├─nvme0n1p1                                   259:1    0   100M  0 part  /boot/efi
├─nvme0n1p2                                   259:2    0    16M  0 part  
├─nvme0n1p3                                   259:3    0 100.7G  0 part  
├─nvme0n1p4                                   259:4    0     1G  0 part  
└─nvme0n1p5                                   259:5    0 136.7G  0 part  /

This is root@machine:/# df -h

Filesystem      Size  Used Avail Use% Mounted on
udev            9.7G     0  9.7G   0% /dev
tmpfs           2.0G   35M  1.9G   2% /run
/dev/nvme0n1p5  135G  134G     0 100% /
tmpfs           9.7G  611M  9.1G   7% /dev/shm
tmpfs           5.0M  8.0K  5.0M   1% /run/lock
tmpfs           9.7G     0  9.7G   0% /sys/fs/cgroup
/dev/nvme0n1p1   96M   31M   66M  32% /boot/efi
/dev/sda2       822G  443G  338G  57% /home
/dev/loop0      128K  128K     0 100% /snap/bare/5
/dev/loop1      9.0M  9.0M     0 100% /snap/canonical-livepatch/146
/dev/loop2      115M  115M     0 100% /snap/core/14056
/dev/loop3      8.9M  8.9M     0 100% /snap/pdfmixtool/864
/dev/loop5       92M   92M     0 100% /snap/gtk-common-themes/1535
/dev/loop4       18M   18M     0 100% /snap/pdftk/9
/dev/loop6       56M   56M     0 100% /snap/core18/2632
/dev/loop7       56M   56M     0 100% /snap/core18/2620
/dev/loop8      261M  261M     0 100% /snap/kde-frameworks-5-core18/32
/dev/loop9      115M  115M     0 100% /snap/core/13886
tmpfs           2.0G  388K  2.0G   1% /run/user/1000
tmpfs           2.0G   20K  2.0G   1% /run/user/125
/dev/dm-0       1.8T  1.6T  192G  89% /media/user/WebMade

This root@machine:/# sudo bash -c 'du -xh / | sort -rhk 1 | head -n 50'

du: cannot access '/tmp/.mount_Nextcl0CMkgy': Permission denied
86G /
59G /root
58G /root/VDI-Files
16G /usr
11G /usr/lib
4.2G    /usr/lib/modules
3.7G    /var
3.7G    /opt
3.1G    /usr/share
2.8G    /usr/lib/x86_64-linux-gnu
1.6G    /var/log
1.5G    /var/lib
1.2G    /var/log/journal/989dbf66577645979de1e1ceab4cc143
1.2G    /var/log/journal
897M    /usr/lib/modules/5.15.0-53-generic
897M    /usr/lib/modules/5.15.0-52-generic
896M    /usr/lib/modules/5.15.0-46-generic
888M    /usr/lib/modules/5.15.0-53-generic/kernel
888M    /usr/lib/modules/5.15.0-52-generic/kernel
888M    /usr/lib/modules/5.15.0-46-generic/kernel
874M    /usr/lib/modules/5.15.0-43-generic
867M    /usr/lib/modules/5.15.0-43-generic/kernel
861M    /usr/src
847M    /opt/baidunetdisk
746M    /usr/bin
731M    /var/lib/snapd
730M    /var/lib/snapd/snaps
712M    /usr/lib/firmware
683M    /root/.config/VirtualBox
683M    /root/.config
678M    /usr/lib/i386-linux-gnu
638M    /usr/share/atom
606M    /opt/apps
528M    /opt/deepinwine
525M    /opt/deepinwine/runtime-i386
489M    /boot
487M    /opt/deepin-wine6-stable
484M    /opt/deepinwine/runtime-i386/usr/lib
484M    /opt/deepinwine/runtime-i386/usr
460M    /opt/zoom
446M    /usr/share/atom/resources
443M    /usr/lib/x86_64-linux-gnu/wine-development
435M    /usr/lib/modules/5.15.0-53-generic/kernel/nvidia-470
435M    /usr/lib/modules/5.15.0-52-generic/kernel/nvidia-470
435M    /usr/lib/modules/5.15.0-46-generic/kernel/nvidia-470
434M    /usr/lib/modules/5.15.0-43-generic/kernel/nvidia-470
390M    /usr/lib/modules/5.13.0-52-generic
387M    /opt/deepinwine/runtime-i386/usr/lib/i386-linux-gnu
384M    /usr/lib/modules/5.13.0-52-generic/kernel
373M    /usr/lib/chromium

This is root@machine:/# tune2fs -l /dev/nvme0n1p5

tune2fs 1.45.5 (07-Jan-2020)
Filesystem volume name:   <none>
Last mounted on:          /
Filesystem UUID:          06ed954b-3750-49d3-b02d-43943788697f
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum
Filesystem flags:         signed_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              8962048
Block count:              35840256
Reserved block count:     1792011
Free blocks:              11323026
Free inodes:              8420105
First block:              0
Block size:               4096
Fragment size:            4096
Group descriptor size:    64
Reserved GDT blocks:      1018
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Flex block group size:    16
Filesystem created:       Tue Jan 25 02:18:18 2022
Last mount time:          Tue Nov 29 04:42:51 2022
Last write time:          Tue Nov 29 04:42:50 2022
Mount count:              142
Maximum mount count:      -1
Last checked:             Fri Jul 29 15:02:27 2022
Check interval:           0 (<none>)
Lifetime writes:          5873 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:           256
Required extra isize:     32
Desired extra isize:      32
Journal inode:            8
First orphan inode:       3276903
Default directory hash:   half_md4
Directory Hash Seed:      2fd3d5f1-ff71-4868-be52-17ed7551c56a
Journal backup:           inode blocks
Checksum type:            crc32c
Checksum:                 0x9ad82476

This is root@machine:/home/user# df -i

Filesystem        Inodes   IUsed     IFree IUse% Mounted on
udev             2522846     637   2522209    1% /dev
tmpfs            2532540    1315   2531225    1% /run
/dev/nvme0n1p5   8962048  650714   8311334    8% /
tmpfs            2532540    1322   2531218    1% /dev/shm
tmpfs            2532540       7   2532533    1% /run/lock
tmpfs            2532540      19   2532521    1% /sys/fs/cgroup
/dev/nvme0n1p1         0       0         0     - /boot/efi
/dev/sda2       54763520  910197  53853323    2% /home
/dev/loop0            29      29         0  100% /snap/bare/5
/dev/loop1            24      24         0  100% /snap/canonical-livepatch/146
/dev/loop2         12848   12848         0  100% /snap/core/14056
/dev/loop3           238     238         0  100% /snap/pdfmixtool/864
/dev/loop5         76208   76208         0  100% /snap/gtk-common-themes/1535
/dev/loop4           370     370         0  100% /snap/pdftk/9
/dev/loop6         10873   10873         0  100% /snap/core18/2632
/dev/loop7         10873   10873         0  100% /snap/core18/2620
/dev/loop8         35005   35005         0  100% /snap/kde-frameworks-5-core18/32
/dev/loop9         12845   12845         0  100% /snap/core/13886
tmpfs            2532540     166   2532374    1% /run/user/1000
tmpfs            2532540      45   2532495    1% /run/user/125
/dev/dm-0      122093568 2735363 119358205    3% /media/user/WebMade

This is root@machine:/home/user# cat /etc/fstab

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/nvme0n1p5 during installation
UUID=06ed954b-3750-49d3-b02d-43943788697f /               ext4    errors=remount-ro 0       1
# /boot/efi was on /dev/nvme0n1p1 during installation
UUID=1C0E-0739  /boot/efi       vfat    umask=0077      0       1
# swap was on /dev/nvme0n1p6 during installation
#UUID=5fc4b948-a7d5-454c-a0d8-7160f260d09f none            swap    sw              0       0

#DP Edits:  Move Home to partition
UUID=b659bc0b-8105-4e5f-8402-89208c74620c /home     ext4    defaults    0   2

#DP Edits: 4GB swapfile
/swapfile none swap sw 0 0

Even More Output

I booted into a LiveUSB in order to check my disk for bad blocks and repair stuff... Found nothing... but all my space has been used:

root@ubuntu:/dev# e2fsck -fc /dev/nvme0n1p5
e2fsck 1.45.5 (07-Jan-2020)
Checking for bad blocks (read-only test):   0.00% done, 0:00 elapsed. (0/0/0 errdone                                                 
/dev/nvme0n1p5: Updating bad block inode.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

/dev/nvme0n1p5: ***** FILE SYSTEM WAS MODIFIED *****
/dev/nvme0n1p5: 650571/8962048 files (1.7% non-contiguous), 35823756/35840256 blocks
root@ubuntu:/dev# e2fsck -vfcp /dev/nvme0n1p5
/dev/nvme0n1p5: Updating bad block inode.

      650571 inodes used (7.26%, out of 8962048)
       10434 non-contiguous files (1.6%)
         903 non-contiguous directories (0.1%)
             # of inodes with ind/dind/tind blocks: 0/0/0
             Extent depth histogram: 603342/1743
    35823756 blocks used (99.95%, out of 35840256)
           0 bad blocks
           3 large files

      525726 regular files
       74110 directories
          10 character device files
           1 block device file
           0 fifos
          74 links
       50668 symbolic links (45420 fast symbolic links)
          47 sockets
------------
      650636 files

Does Anyone know how to recover my 54GB of space?

in flag
Could the amount of reserved space for the root partition been changed from 5% to something larger? What is the terminal output of `tune2fs -l /dev/nvme0n1p5`?
nightwatch avatar
tr flag
@matigo added above... any idea? I am afraid to boot liveUSB until I have a strategy to fix this because I don't know if I will be able to reboot the main system if my liveUSB session fails to resolve the problem
nightwatch avatar
tr flag
@matigo added more outputs.. could it be the /tmp ?
nightwatch avatar
tr flag
@matigo booted into LiveUSB still not finding the where 54GB went... and disk shows as healthy
cc flag
If you umount /home, do you see (hidden by the mount) files in /home? Those could account for the missing space.
Score:0
tr flag

I found the problem... Another "ghost mount"...

I boot into a LiveUSB session and ran the above commands to check my disk... all commands said the disk was "happy" but full.

While still in the liveUSB session, I mounted my disk via nautilus, but in terminal I again ran

root@ubuntu:/dev# du -xh /media/ubuntu/06ed954b-3750-49d3-b02d-43943788697f | sort -rhk 1 | head -n 100

(with 06ed954b-3750-49d3-b02d-43943788697f being my mount root partition / in the liveUSB session.

This time it showed root@ubuntu:/dev# du -xh /media/ubuntu/06ed954b-3750-49d3-b02d-43943788697f | sort -rhk 1 | head -n 100

86G /media/ubuntu/06ed954b-3750-49d3-b02d-43943788697f
59G /media/ubuntu/06ed954b-3750-49d3-b02d-43943788697f/root
58G /media/ubuntu/06ed954b-3750-49d3-b02d-43943788697f/root/VDI-Files
49G /media/ubuntu/06ed954b-3750-49d3-b02d-43943788697f/media/user/external_drive
16G /media/ubuntu/06ed954b-3750-49d3-b02d-43943788697f/usr
11G /media/ubuntu/06ed954b-3750-49d3-b02d-43943788697f/usr/lib
4.2G    /media/ubuntu/06ed954b-3750-49d3-b02d-43943788697f/usr/lib/modules
3.7G    /media/ubuntu/06ed954b-3750-49d3-b02d-43943788697f/var
3.7G    /media/ubuntu/06ed954b-3750-49d3-b02d-43943788697f/opt
3.1G    /media/ubuntu/06ed954b-3750-49d3-b02d-43943788697f/usr/share
2.8G    /media/ubuntu/06ed954b-3750-49d3-b02d-43943788697f/usr/lib/x86_64-linux-gnu
1.7G    /media/ubuntu/06ed954b-3750-49d3-b02d-43943788697f/var/log
1.5G    /media/ubuntu/06ed954b-3750-49d3-b02d-43943788697f/var/lib
1.2G    /media/ubuntu/06ed954b-3750-49d3-b02d-43943788697f/var/log/journal/989dbf66577645979de1e1ceab4cc143
1.2G    /media/ubuntu/06ed954b-3750-49d3-b02d-43943788697f/var/log/journal

The 49G /media/ubuntu/06ed954b-3750-49d3-b02d-43943788697f/media/user/external_drive was impossible because I did not mount it during the liveUSB session.

so I rm -rf /media/ubuntu/06ed954b-3750-49d3-b02d-43943788697f/media/user/external_drive

and presto!

49GB of space has returned to me and I can finally get some real work done.

So there seems to be two possible types of "ghost mounts" after unrecommended hard reboots.

A "path error" ghost mount that I ran into here, and now A "disk space" ghost mount (note, none of my scripts freaked out because the paths to my external drive as both normal and root user were correct even though the "ghost mount" existed)

In liveUSB session gparted and other apps now confirm my free disk space.

in flag
Glad you could work it out. I used to have similar issues with web servers using remote storage locations. You may want to add a script that runs at shutdown to unmount locations and report on any files/directories that are found.
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.