Join Log in
Score:1
l k avatar Ubuntu

ZFS bpool is almost full; how can I free up space so I can keep updating my system without errors?

lk flag
l k

I am running Ubuntu with ZFS on root, and my bpool seems to be almost full.

Output of zpool list bpool:

NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
bpool  1.88G  1.69G   191M        -         -    60%    90%  1.00x    ONLINE  -

Output of df -h /boot:

Filesystem                                        Size  Used Avail Use% Mounted on
bpool/BOOT/ubuntu_s2c09x                          367M  303M   64M  83% /boot

This is causing problems with completing the installation of kernel updates, e.g. running sudo apt upgrade:

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
2 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] 
Requesting to save current system state
ERROR couldn't save system state: Minimum free space to take a snapshot and preserve ZFS performance is 20%.
Free space on pool "bpool" is 10%.
Please remove some states manually to free up space. 
Setting up linux-firmware (1.197.5) ...
update-initramfs: Generating /boot/initrd.img-5.11.0-41-generic
I: The initramfs will attempt to resume from /dev/nvme0n1p2
I: (UUID=abdb40da-2d0b-4a45-b024-1c7245ea27ea)
I: Set the RESUME variable to override this.
Error 24 : Write error : cannot write compressed block 
E: mkinitramfs failure cpio 141 lz4 -9 -l 24
update-initramfs: failed for /boot/initrd.img-5.11.0-41-generic with 1.
dpkg: error processing package linux-firmware (--configure):
 installed linux-firmware package post-installation script subprocess returned error exit status 1
Setting up linux-image-5.11.0-34-generic (5.11.0-34.36) ...
Processing triggers for linux-image-5.11.0-34-generic (5.11.0-34.36) ...
/etc/kernel/postinst.d/dkms:
 * dkms: running auto installation service for kernel 5.11.0-34-generic
Error! Your kernel headers for kernel 5.11.0-34-generic cannot be found.
Please install the linux-headers-5.11.0-34-generic package,
or use the --kernelsourcedir option to tell DKMS where it's located
   ...done.
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-5.11.0-34-generic
I: The initramfs will attempt to resume from /dev/nvme0n1p2
I: (UUID=abdb40da-2d0b-4a45-b024-1c7245ea27ea)
I: Set the RESUME variable to override this.
Error 24 : Write error : cannot write compressed block 
E: mkinitramfs failure cpio 141 lz4 -9 -l 24
update-initramfs: failed for /boot/initrd.img-5.11.0-34-generic with 1.
run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1
dpkg: error processing package linux-image-5.11.0-34-generic (--configure):
 installed linux-image-5.11.0-34-generic package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 linux-firmware
 linux-image-5.11.0-34-generic
ZSys is adding automatic system snapshot to GRUB menu

I've tried destroying some bpool snapshots, but zfs list -t snapshot | grep bpool doesn't list any over 100K. I have also tried removing some kernels, see output of dpkg -l linux-{image,headers}-"[0-9]*":

un  linux-headers-3.0               <none>         <none>       (no description available)
un  linux-headers-5.10.0-1045-oem   <none>         <none>       (no description available)
un  linux-headers-5.11.0-34-generic <none>         <none>       (no description available)
un  linux-headers-5.11.0-35-generic <none>         <none>       (no description available)
un  linux-headers-5.11.0-36-generic <none>         <none>       (no description available)
un  linux-headers-5.11.0-37-generic <none>         <none>       (no description available)
un  linux-headers-5.11.0-38-generic <none>         <none>       (no description available)
ii  linux-headers-5.11.0-40         5.11.0-40.44   all          Header files related to Linux kernel version 5.11.0
ii  linux-headers-5.11.0-40-generic 5.11.0-40.44   amd64        Linux kernel headers for version 5.11.0 on 64 bit x86 SMP
ii  linux-headers-5.11.0-41         5.11.0-41.45   all          Header files related to Linux kernel version 5.11.0
ii  linux-headers-5.11.0-41-generic 5.11.0-41.45   amd64        Linux kernel headers for version 5.11.0 on 64 bit x86 SMP
un  linux-headers-686-pae           <none>         <none>       (no description available)
ic  linux-image-5.10.0-1045-oem     5.10.0-1045.47 amd64        Signed kernel image oem
un  linux-image-5.11.0-27-generic   <none>         <none>       (no description available)
iF  linux-image-5.11.0-34-generic   5.11.0-34.36   amd64        Signed kernel image generic
rc  linux-image-5.11.0-35-generic   5.11.0-35.37   amd64        Signed kernel image generic
rc  linux-image-5.11.0-36-generic   5.11.0-36.40   amd64        Signed kernel image generic
rc  linux-image-5.11.0-37-generic   5.11.0-37.41   amd64        Signed kernel image generic
rc  linux-image-5.11.0-38-generic   5.11.0-38.42   amd64        Signed kernel image generic
ii  linux-image-5.11.0-40-generic   5.11.0-40.44   amd64        Signed kernel image generic
ii  linux-image-5.11.0-41-generic   5.11.0-41.45   amd64        Signed kernel image generic

The oldest currently installed one seems to be linux-image-5.11.0-34-generic, however sudo apt purge linux-image-5.11.0-34-generic suggests that it wants to also install linux-image-unsigned-5.11.0-34-generic, taking up more space.

What else can I try? Is it conceivably an option to resize/grow my bpool? I am pretty new to using ZFS.

Edit: I did some more investigating. zfs list bpool/BOOT/ubuntu_s2c09x -o space -r reports

NAME                                                             AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
bpool/BOOT/ubuntu_s2c09x                                          229M  1.51G     1.22G    303M             0B         0B
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_weekly-2021-11-19-1344        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-21-1339         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-22-1336         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-23-1340         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-24-1335         -    56K         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-25-1339         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-26-1338         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_weekly-2021-11-26-1343        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-27-1338         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-28-1336         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-29-1338         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-30-1340         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-01-1340         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-02-1340         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-03-1339         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_weekly-2021-12-03-1344        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-04-1337         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-05-1338         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-06-1338         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-07-1339         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-08-1337         -    80K         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-09-0645         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@autozsys_a2u0ly                             -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-10-1336         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_weekly-2021-12-10-1341        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-11-1339         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-12-1335         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@autozsys_omna1o                             -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@autozsys_5q7106                             -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-13-1336         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@autozsys_fwalez                             -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-14-1336         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@autozsys_dsm4th                             -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-15-1335         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@autozsys_4t769s                             -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-16-1336         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@autozsys_z8aa09                             -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-17-1339         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_weekly-2021-12-17-1344        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@autozsys_dqsnib                             -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@autozsys_7w0l6s                             -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@autozsys_323q08                             -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@autozsys_gqbsa5                             -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-18-1338         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-19-1339         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-0217        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-0317        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-0417        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-0517        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-0617        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-0717        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-0817        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-0917        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1017        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1117        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1217        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1317        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-21-1337         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1417        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1517        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1617        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1717        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1817        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1917        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-2017        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-2117        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-2217        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-2317        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-22-0017        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_frequent-2021-12-22-0100      -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_frequent-2021-12-22-0115      -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-22-0117        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_frequent-2021-12-22-0130      -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_frequent-2021-12-22-0145      -     0B         -       -              -          -

I don't understand how it can claim that 1.22 GB are used by snapshots, but none of the listed bpool snapshots are using any space?

417
0 + 0
apt
zfs
Ciro Santilli 新疆再教育营六四事件法轮功郝海东 avatar
uz flag
Ciro Santilli 新疆再教育营六四事件法轮功郝海东
Related: https://askubuntu.com/questions/1246232/ubuntu-20-04-thinks-its-out-of-free-space-but-it-isnt
0
Reply
Score:3
avatar Ubuntu
ru flag
Alex

Uninstalling old and unused kernels and associated files is the correct action to take; to avoid the suggestion to install an unsigned kernel, remove packages that depend on linux-image-5.11.0-34-generic via:

sudo apt purge linux*5.11.0-34*

The second step required to free up space is to remove old snapshots referencing the files that were removed; list the snapshots via:

zfs list -r -t snapshot -o name,used,referenced,creation bpool/BOOT

You can see in the REFER column how much data is referenced. Identical numbers often coincide with minimal USED values; the snapshots are effectively copies of previous snapshots.

Manually remove the five oldest snapshots via:

zfs list -r -t snapshot -o name,used,referenced,creation bpool/BOOT | head -n 5 | cut -c 35-40 | xargs -n 1 zsysctl state remove --system

Keep removing snapshots until you see space being freed up.

(Source of the last one-liner and more info: https://github.com/ubuntu/zsys/issues/155)

Notes:

  1. Growing a ZFS pool is possible if there is unallocated space, but Ubuntu allocates the remaining space to rpool. Shrinking a ZFS pool in-place is apparently not possible.

  2. I see auto-snapshotting is turned on for bpool, but unless you expect any files to change outside of APT, ZSys snapshotting should be sufficient and would reduce the number of snapshots you have to consider.

  3. To avoid this problem in the future, stay on top of kernel upgrades and remove old ones regularly. Also consider reducing the ZSys garbage cleaning defaults in /etc/zsys.conf. For an example, see: https://github.com/ubuntu/zsys/issues/155#issuecomment-758902487

0 + 0
Fakhamatia avatar
cn flag
Fakhamatia
How remove only large snapshots?
0
Reply
ru flag
Alex
@Fakhamatia, I manually check whether the value in the `USED` column is large, but generally assume that any snapshot after it will reference much of the same data. Therefore, only removing snapshots that appear large may not free much space, since they build on each others' data.
0
Reply
l k avatar
lk flag
l k
auto-snapshotting on bpool does seem to be a part of the problem. Deleting older snapshots did indeed free up space, despite their minimal USED values. FYI, the one-liner to remove the 5 oldest snapshots breaks when auto-snapshots are present, but I could delete them all leaving only the ZSys ones via `zfs list -r -t snapshot -o name bpool/BOOT | grep auto-snap | xargs -n 1 sudo zfs destroy`.
1
Reply
ru flag
Alex
`USED` only shows the size of data unique to that snapshot. Via `-o referenced` you can see `REFER` values, which capture all the data referenced by a snapshot, providing a total snapshot size (that often overlaps with that of other snapshots, most obviously when the `REFER` value is greater than the `USED` value). The `REFER` values should show that auto-snapshots reference a lot of old files, preventing disk space from being freed up until the last snapshot referencing those old files is destroyed. Thank you for sharing how you got rid of the auto-snapshots!
2
Reply
cn flag
Michael
`To avoid this problem in the future, stay on top of kernel upgrades and remove old ones regularly` I don't understand, to my mind this is a bug that I can't just install Ubuntu and have it take care of all this crap automatically... why am I being forced to dig into something that manifests itself cryptically and requires all this work???
0
Reply
cn flag
Michael
to put it another way, I see 27 kernels installed... just... why? Why can't it get rid of the old ones and limit it to some manageable number at a time like say, 5?
0
Reply
cn flag
Michael
so I deleted 26 kernels (somehow... not sure how that many as there are now 14 kernels still installed) and found that the oldest pool claimed to be taking 168M... with snapshots as recent as today listed via the command you gave but none of them taking any significant space. Destroyed the 168M one and... `zpool list` says the *exact* same amount of space is being taken?!
0
Reply
ru flag
Alex
@Michael: 1. See e.g., https://askubuntu.com/q/590673 2. You can, just not by default because of reasons outlined above: https://askubuntu.com/q/563483 3. When deleting snapshots, make sure all snapshots that reference the files you want to remove are gone; usually done by deleting ones with the same value under the `REFER` column - `USED` only shows what data is specific to the particular snapshot, not data referenced by multiple snapshots unless the current one is the first to reference it (AFAIK).
0
Reply
Ciro Santilli 新疆再教育营六四事件法轮功郝海东 avatar
uz flag
Ciro Santilli 新疆再教育营六四事件法轮功郝海东
The command works. But it asks for my password five times :-(
0
Reply
ru flag
Alex
@CiroSantilliOurBigBook.com I have edited my answer to include `sudo` so that it won't keep asking for a password five times.
1
Reply
LiveWireBT avatar
fr flag
LiveWireBT
The statement "remove the five oldest snapshots " does not seem correct to me and I feel a bit uneasy about using `head -n 5`. From what I see on my system it also outputs the header, so the correct statement would be that you remove the oldest 4 snapshots. To be in line again with the statement, and to avoid passing unintended content to programs further down the chain I would suggest using `head -n 6 | tail -n -5` instead. It worked fine except for the first snapshot. But thank you so much, it helped me a bit cleaning up a messy system in the aftermath of Zsys.
0
Reply
ru flag
Alex
Thank you, @LiveWireBT - you are correct! I have added the `-H` parameter to omit headers from the output, achieving the same as your `head -n 6 | tail -n 5`.
0
Reply
LiveWireBT avatar
fr flag
LiveWireBT
@Alex Thank you again, I knew that there must have been an option to supress teh header but I was unable to find it yesterday.
0
Reply
Score:0
avatar Ubuntu
us flag
Chelmite

This will remove the oldest 5 kernels from zfs. It will not remove the most recent 5 if there are 5 or fewer kernels installed.

# Run "zfs list" to list all the installed kernels.
# Tail removes the header line
# head -n -5 preserves the last 5 in the list, listing all others
# head -5 removes only the first 5 (if at least that many exist)
zfs list -r -t snapshot -o name,used,referenced,creation bpool/BOOT | tail -n +2 | head -n -5 | head -5 | cut -c 35-40 | xargs -n 1 /usr/bin/sudo zsysctl state remove --system
# List the amount of space remaining on /boot
df -h /boot
+ 1
us flag
lindhe
That command seems broken to me.
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.