zfs managed disks are completely gone after upgrading ubuntu 20.04 to ubuntu 22.04

I installed ubuntu 20.04 on Supermicro X9DRW-3LN4F+/X9DRW-3TF+ with 2 NMVe drive and 4 SATA drive. I setup zfs pool on the 4 SATA drive like below:

$ sudo zpool status
[sudo] password for igdvs: 
  pool: data
 state: ONLINE
  scan: none requested
config:

    NAME                                         STATE     READ WRITE CKSUM
    data                                         ONLINE       0     0     0
      scsi-SATA_Hitachi_HUA72303_MK0371YHK2E0XA  ONLINE       0     0     0
      scsi-SATA_Hitachi_HUA72303_MK0371YHK2L7NA  ONLINE       0     0     0
      scsi-SATA_Hitachi_HUA72303_MK0371YHK1ME0A  ONLINE       0     0     0
      scsi-SATA_Hitachi_HUA72303_MK0331YHGZHD7A  ONLINE       0     0     0

errors: No known data errors

since my lxc container runs on zfs pool and I want to upgrade ubuntu 20.04 host kernel so one of my lxc container can benefit new host kernel feature. so I decided to install 5.15.0-69-generic package, then reboot the host, but after reboot, 5.15.0-69-generic fail to detect zfs managed drives.

then I decided to upgrade host to Ubuntu 22.04, but after upgrade, host still failed to detect zfs managed drives, the SATA drive are completely gone, not showing up in fdisk -l, lsblk, not in /dev/disk/by-id.

$ ls -l /dev/disk/by-id/
total 0
lrwxrwxrwx 1 root root  9 Apr 22 20:48 ata-BP5_FEB40771110806776224 -> ../../sda
lrwxrwxrwx 1 root root 10 Apr 22 20:48 ata-BP5_FEB40771110806776224-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Apr 22 20:48 ata-BP5_FEB40771110806776224-part2 -> ../../sda2
lrwxrwxrwx 1 root root  9 Apr 22 20:48 md-name-ubuntu-server:0 -> ../../md0
lrwxrwxrwx 1 root root 11 Apr 22 20:48 md-name-ubuntu-server:0-part1 -> ../../md0p1
lrwxrwxrwx 1 root root  9 Apr 22 20:48 md-uuid-b99cfa1b:4631aff7:91f67a2d:dc09cb51 -> ../../md0
lrwxrwxrwx 1 root root 11 Apr 22 20:48 md-uuid-b99cfa1b:4631aff7:91f67a2d:dc09cb51-part1 -> ../../md0p1
lrwxrwxrwx 1 root root 13 Apr 22 20:48 nvme-BPX_8B71076C020835003325 -> ../../nvme0n1
lrwxrwxrwx 1 root root 15 Apr 22 20:48 nvme-BPX_8B71076C020835003325-part1 -> ../../nvme0n1p1
lrwxrwxrwx 1 root root 15 Apr 22 20:48 nvme-BPX_8B71076C020835003325-part2 -> ../../nvme0n1p2
lrwxrwxrwx 1 root root 13 Apr 22 20:48 nvme-nvme.1987-3842373130373643303230383335303033333235-425058-00000001 -> ../../nvme0n1
lrwxrwxrwx 1 root root 15 Apr 22 20:48 nvme-nvme.1987-3842373130373643303230383335303033333235-425058-00000001-part1 -> ../../nvme0n1p1
lrwxrwxrwx 1 root root 15 Apr 22 20:48 nvme-nvme.1987-3842373130373643303230383335303033333235-425058-00000001-part2 -> ../../nvme0n1p2
lrwxrwxrwx 1 root root  9 Apr 22 20:48 scsi-0ATA_BP5_FEB40771110806776224 -> ../../sda
lrwxrwxrwx 1 root root 10 Apr 22 20:48 scsi-0ATA_BP5_FEB40771110806776224-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Apr 22 20:48 scsi-0ATA_BP5_FEB40771110806776224-part2 -> ../../sda2
lrwxrwxrwx 1 root root  9 Apr 22 20:48 scsi-1ATA_BP5_FEB40771110806776224 -> ../../sda
lrwxrwxrwx 1 root root 10 Apr 22 20:48 scsi-1ATA_BP5_FEB40771110806776224-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Apr 22 20:48 scsi-1ATA_BP5_FEB40771110806776224-part2 -> ../../sda2
lrwxrwxrwx 1 root root  9 Apr 22 20:48 scsi-SATA_BP5_FEB40771110806776224 -> ../../sda
lrwxrwxrwx 1 root root 10 Apr 22 20:48 scsi-SATA_BP5_FEB40771110806776224-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Apr 22 20:48 scsi-SATA_BP5_FEB40771110806776224-part2 -> ../../sda2
lrwxrwxrwx 1 root root  9 Apr 22 20:48 usb-Generic_Flash_Disk_C8A76757-0:0 -> ../../sdb
lrwxrwxrwx 1 root root 10 Apr 22 20:48 usb-Generic_Flash_Disk_C8A76757-0:0-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Apr 22 20:48 usb-Generic_Flash_Disk_C8A76757-0:0-part9 -> ../../sdb9

from the BIOS, I can see the 4 SATA drive.

the problem is after upgrade, I have the kernel grub boot order:

$ sudo grub-mkconfig | grep -iE "menuentry 'Ubuntu, with Linux" | awk '{print i++ " : "$1, $2, $3, $4, $5, $6, $7}'
Sourcing file `/etc/default/grub'
Sourcing file `/etc/default/grub.d/init-select.cfg'
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.15.0-70-generic
Found initrd image: /boot/initrd.img-5.15.0-70-generic
Found linux image: /boot/vmlinuz-5.15.0-69-generic
Found initrd image: /boot/initrd.img-5.15.0-69-generic
Found linux image: /boot/vmlinuz-5.4.0-147-generic
Found initrd image: /boot/initrd.img-5.4.0-147-generic
Warning: os-prober will not be executed to detect other bootable partitions.
Systems on them will not be added to the GRUB boot configuration.
Check GRUB_DISABLE_OS_PROBER documentation entry.
Adding boot menu entry for UEFI Firmware Settings ...
done
0 : menuentry 'Ubuntu, with Linux 5.15.0-70-generic' --class ubuntu
1 : menuentry 'Ubuntu, with Linux 5.15.0-70-generic (recovery mode)'
2 : menuentry 'Ubuntu, with Linux 5.15.0-69-generic' --class ubuntu
3 : menuentry 'Ubuntu, with Linux 5.15.0-69-generic (recovery mode)'
4 : menuentry 'Ubuntu, with Linux 5.4.0-147-generic' --class ubuntu
5 : menuentry 'Ubuntu, with Linux 5.4.0-147-generic (recovery mode)'

and the /etc/default/grub has:

$ cat /etc/default/grub
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

#GRUB_DEFAULT=0
GRUB_DEFAULT="1>2"
GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=0
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX=""

so the 5.15.0-69-generic becomes the default boot kernel, note this is not the fault of Ubuntu OS upgrade, it is me changed the /etc/default/grub default boot kernel without realizing it. so the lesson is not to install new kernel package alone when you have zfs managed disks, and check your default boot kernel after upgrading the OS and if you changed the default boot kernel previously. this is unusual and should rarely happen, but people does make mistakes