Score:6

Strange file issues on Ubuntu 21.10 with ZFS

bd flag

After installing or upgrading to Ubuntu 21.10, with the ZFS option, I started to experience strange issues.

For example, on the command line, I have started to see errors like this:

Cannot access 'foobar': No such file or directory

When running ls -l in some directory, I get question marks, like this:

-????????? ? ? ? ? ? foobar.txt
-????????? ? ? ? ? ? foobar2.txt

Eventually, I couldn't even boot Ubuntu 21.10 (which uses ZFS).

Score:15
bd flag

This is a known issue. The release notes for Ubuntu 21.10 say:

Known Issues

Linux kernel

  • The version of the ZFS driver included in the 5.13.0-19 kernel contains a bug that can result in filesystem corruption. Users of ZFS are advised to wait until the first Stable Release Update of the kernel in 21.10 before upgrading.

I recommend this course of action:

  • If you are installing Ubuntu 21.10, do not select the ZFS option. This will result in filesystem corruption, which you do not want.
  • If you are upgrading to Ubuntu 21.10, and you're using ZFS, don't, until the fix has been released and users are no longer been reporting issues. (As of writing, the fix hasn't been released yet.)

If you have already upgraded to Ubuntu 21.10, you can revert to your old filesystem by choosing the "Revert" option in the boot menu (grub). Press del while booting up to display this boot menu.

us flag
This filesystem corruption bug needs a big fat RED warning on top of the Ubuntu release page. Leaving this up is a complete joke. Canonical
Ned Burgher avatar
br flag
Thank you @Flimm - I would've dived right in if I hadn't seen your post.
Score:3
cn flag

FWIW I have just successfully upgraded from 21.04 to 21.10 after checking the bug referenced in the Flimm's post and seeing some indications (not as clear as I would have liked...) that it was fixed in kernel version 5.13.0-20. The bug appears to be present in all previous 5.13 kernel versions, up to and including 5.13.0-19, which is the one that was fatally released with 21.10 initially.

My system is running a custom-deployed ZFS on root. It was deployed from scratch with the stock version that came with Hirsute (ZFS v2.0.3). I am not running Ubuntu's Zsys but I don't think that's relevant.

Procedure

  1. Made sure I had ample backups available and on a separate system; clearly relying on ZFS snapshots isn't a suitable strategy here.
  2. Ran the usual do-release-upgrade procedure. Everything went through flawlessly; actually one of the smoothest Ubuntu upgrades I've experienced.
  3. After the upgrade completed but before rebooting (ie: while still running the "safe" 5.11 Hirsute kernel), I double-checked /boot to make sure the kernel about to be booted was newer than 5.13.0-19; I got 5.13.0-22:
$ ll /boot/vmlinuz
lrwxrwxrwx 1 root root 25 Dec 27 16:59 /boot/vmlinuz -> vmlinuz-5.13.0-22-generic

More information

The below advice is only relevant until a stable version of 22.04 ("Jammy") comes out.

Live image

As of this writing, Canonical's 21.10 Live image still comes with the buggy kernel 5.13.0-19. Therefore, do not under any circumstance use that image to access/deploy ZFS filesystems.

Should you need to perform such maintenance, you should probably use the 21.04 Live image rather than 21.10; the ZFS versions are pretty close (2.0.3/2.0.6).

New 21.10 deployments with ZFS on root

I have confirmed that it's possible to use the 21.04 Live image to perform a 21.10 ZFS-on-root deployment. Simply follow the ZFS on root guide, and specify the desired release when invoking debootstrap (at step 3.4), as in:

debootstrap impish /mnt

You'll also need to specify the same version (impish) in /mnt/etc/apt/sources.list (step 4.3).

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.