Removed files are still referenced by past snapshots, so what you describe is expected behaviour.
As you already observed, the space remains used due to snapshots; only after deleting the last snapshot that references the deleted files will the corresponding space be available again for new files.
Note that df
does not provide accurate values for free space in ZFS file systems. ZFS uses one or more disks (or partitions) to store pools that can contain one or more datasets (interpreted by df
as filesystems), all of which share the pool's free space. Snapshots also fill up the pool (as part of datasets), further affecting the 'Size' df
reports for an individual file system. ZFS is both a file system, a volume manager, and optionally manages RAID arrays; it would be analogous to, e.g., ext4 on LVM optionally on RAID.
To see how pools correspond to disks or partitions, you can use zpool status
. In Ubuntu, you can find the listed disks or partitions under /dev/disk/by-uuid/
and /dev/disk/by-partuuid/
respectively. To see the free space in pools, you can list all pools via zpool list
. You can list all ZFS datasets ("file systems") via zfs list
, where the USED
column indicates the space used by the dataset, which includes the files, child datasets and snapshots belonging to a specific dataset; AVAIL
indicates the remaining space in the pool, and REFER
indicates the size of the files specific to this file system, i.e., not snapshots or child filesystems.