Kernel panic output image
I have a very specific and strange problem and ita kinda urgent to solve it in a timely manner. Yesterday I encountered a kernel panic on my servers
looking at server management logs apparently there was a power outage and UPS was out of service for maintenance
booting to the default kernel:
cannot open "scripts/zfs" no such file or directory kernel panic "not syncing attempted to kill init"
I was able to boot from recovery mode in grub to a previous kernel then after checking system journal I deleted the problem a malfunctioning virtual network card causing problems so I removed the ovswitch, did an apt update and fix broken packages and install the new available kernel. After reboot I still get the kernel panic, and because I updated grub and installed new kernel, the previous kernel which was working is no longer available so im no longer able to boot the system.
Reinstalling system is out of the question so I need help to repair the kernel.
this is what I've tried
So far I booted a live os, imported the zfs pool, I can see the pool is mounted new directory appears in the root directory of the live system and with 'zfs list' command I can see that the root directory of the zfs pool is mounted to / but I cant read the data when I cd to the directory its empty
I did "zpool import rpool" then 2 directories appeared rpool/DATA and rpool/ROOT/pve-1 but they are both empty.
with lsblk I can see most of the guest VMs partitions but not main system's.
also after importing the zfs pool I did "mdadm --assemble --scan" two additional partitions appeared and I dont know what they are ( the reason I assembled raid arrays after importing zfs is because before importing the rpool mdadm wont recognize any raid arrays)
I believe I need to chroot to the storage with live os and repair zfs scripts
its worth mentioning that I have selinux on permissive mode, I i believe this is what caused the problem isn the first place.
A clean install would be my last option.
So, Is there anything I should try before rebuilding?
And, if I must rebuild, Can someone provide guidance on migrating my data?
system configuration:
software RAID 5
filesystem ZFS
storage : rpool zfs / rpool.