After reboot of Ubuntu-2110 on my Hetzner VPS I found myself in a no boot situation.
Based on journalctl I had an issue starting a target for swap and boot partitions. It takes 1.5min and fails leaving me in emergency mode.
I checked UUIDs in /etc/fstab against lsblk, and they match.
I tried to change UUID to /dev/md/0 but it didn't help either.
My fstab looks like that:
proc /proc proc defaults 0 0
#/dev/md/0 none swap sw 0 0
UUID=7b5a72f5-8576-4989-acc3-de6f15e14f46 none swap sw 0 0
# /dev/md/1
UUID=716fc135-1944-4959-9652-c5105c037ec9 /boot ext3 defaults 0 1
# /dev/md/2
UUID=5644440c-f53c-4a4c-998d-dd9777c2c879 / ext4 defaults 0 0
My lsblk look like this
NAME UUID
loop0 1aee0e3e-0e1e-4686-a9a9-61e7152e0209
nvme1n1
├─nvme1n1p1 0648d0f8-4623-db7a-8fe6-48eee7572c5c
│ └─md0 7b5a72f5-8576-4989-acc3-de6f15e14f46
├─nvme1n1p2 58e884a5-30cd-1afa-4b73-0f2f50f09009
│ └─md1 716fc135-1944-4959-9652-c5105c037ec9
└─nvme1n1p3 586d3257-b7e1-9214-d4cc-6b5f1198c2bf
└─md2 5644440c-f53c-4a4c-998d-dd9777c2c879
nvme0n1
├─nvme0n1p1 0648d0f8-4623-db7a-8fe6-48eee7572c5c
│ └─md0 7b5a72f5-8576-4989-acc3-de6f15e14f46
├─nvme0n1p2 58e884a5-30cd-1afa-4b73-0f2f50f09009
│ └─md1 716fc135-1944-4959-9652-c5105c037ec9
└─nvme0n1p3 586d3257-b7e1-9214-d4cc-6b5f1198c2bf
└─md2 5644440c-f53c-4a4c-998d-dd9777c2c879
I have no problem mounting for example /boot in emergency mode, and my / is even auto-mounted properly, yet if I would try to systemctl default it falls right back into emergency mode.
Some highlights from journal:
---This one may be related to what causes it (more of it later)
Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: Kernel command line: BOOT_IMAGE=/vmlinuz-5.13.0-21-generic root=UUID=5644440c-f53c-4a4c-998d-dd9777c2c879 ro consoleblank=>
Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: You have booted with nomodeset. This means your GPU drivers are DISABLED
Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: Any video related functionality will be severely degraded, and you may not even be able to suspend the system properly
Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: Unless you actually understand what nomodeset does, you should reboot without enabling it
---Googling says that part should be harmless
Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: platform eisa.0: Probing EISA bus 0
Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: platform eisa.0: EISA: Cannot allocate resource for mainboard
Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: platform eisa.0: Cannot allocate resource for EISA slot 1
Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: platform eisa.0: Cannot allocate resource for EISA slot 2
Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1
Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: platform eisa.0: Cannot allocate resource for EISA slot 3
Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: platform eisa.0: Cannot allocate resource for EISA slot 4
Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: platform eisa.0: Cannot allocate resource for EISA slot 5
Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: platform eisa.0: Cannot allocate resource for EISA slot 6
Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: platform eisa.0: Cannot allocate resource for EISA slot 7
Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: platform eisa.0: Cannot allocate resource for EISA slot 8
Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: platform eisa.0: EISA: Detected 0 cards
---This part seems to be related to floppy only and harmless, but the last string bothers me.
Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: floppy0: disk absent or changed during operation
Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: blk_update_request: I/O error, dev fd0, sector 0 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: floppy0: disk absent or changed during operation
Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: blk_update_request: I/O error, dev fd0, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: Buffer I/O error on dev fd0, logical block 0, async page read
Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base kernel: EXT4-fs (md2): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
Mar 25 08:20:53 Ubuntu-2110-impish-amd64-base systemd[1]: Inserted module 'autofs4'
---And here is where it seems for me to fail
Mar 25 08:22:23 Ubuntu-2110-impish-amd64-base systemd[1]: dev-disk-by\x2duuid-716fc135\x2d1944\x2d4959\x2d9652\x2dc5105c037ec9.device: Job dev-disk-by\x2duuid-716fc135\x2d1944\>
Mar 25 08:22:23 Ubuntu-2110-impish-amd64-base systemd[1]: Timed out waiting for device /dev/disk/by-uuid/716fc135-1944-4959-9652-c5105c037ec9.
Mar 25 08:22:23 Ubuntu-2110-impish-amd64-base systemd[1]: Dependency failed for File System Check on /dev/disk/by-uuid/716fc135-1944-4959-9652-c5105c037ec9.
Mar 25 08:22:23 Ubuntu-2110-impish-amd64-base systemd[1]: Dependency failed for /boot.
Mar 25 08:22:23 Ubuntu-2110-impish-amd64-base systemd[1]: Dependency failed for Local File Systems.
Mar 25 08:22:23 Ubuntu-2110-impish-amd64-base systemd[1]: local-fs.target: Job local-fs.target/start failed with result 'dependency'.
Mar 25 08:22:23 Ubuntu-2110-impish-amd64-base systemd[1]: local-fs.target: Triggering OnFailure= dependencies.
Mar 25 08:22:23 Ubuntu-2110-impish-amd64-base systemd[1]: boot.mount: Job boot.mount/start failed with result 'dependency'.
Mar 25 08:22:23 Ubuntu-2110-impish-amd64-base systemd[1]: systemd-fsck@dev-disk-by\x2duuid-716fc135\x2d1944\x2d4959\x2d9652\x2dc5105c037ec9.service: Job systemd-fsck@dev-disk-b>
Mar 25 08:22:23 Ubuntu-2110-impish-amd64-base systemd[1]: dev-disk-by\x2duuid-716fc135\x2d1944\x2d4959\x2d9652\x2dc5105c037ec9.device: Job dev-disk-by\x2duuid-716fc135\x2d1944\>
Mar 25 08:22:23 Ubuntu-2110-impish-amd64-base systemd[1]: dev-disk-by\x2duuid-7b5a72f5\x2d8576\x2d4989\x2dacc3\x2dde6f15e14f46.device: Job dev-disk-by\x2duuid-7b5a72f5\x2d8576\>
Mar 25 08:22:23 Ubuntu-2110-impish-amd64-base systemd[1]: Timed out waiting for device /dev/disk/by-uuid/7b5a72f5-8576-4989-acc3-de6f15e14f46.
Mar 25 08:22:23 Ubuntu-2110-impish-amd64-base systemd[1]: Dependency failed for /dev/disk/by-uuid/7b5a72f5-8576-4989-acc3-de6f15e14f46.
Mar 25 08:22:23 Ubuntu-2110-impish-amd64-base systemd[1]: Dependency failed for Swap.
Mar 25 08:22:23 Ubuntu-2110-impish-amd64-base systemd[1]: swap.target: Job swap.target/start failed with result 'dependency'.
Mar 25 08:22:23 Ubuntu-2110-impish-amd64-base systemd[1]: dev-disk-by\x2duuid-7b5a72f5\x2d8576\x2d4989\x2dacc3\x2dde6f15e14f46.swap: Job dev-disk-by\x2duuid-7b5a72f5\x2d8576\x2>
Mar 25 08:22:23 Ubuntu-2110-impish-amd64-base systemd[1]: dev-disk-by\x2duuid-7b5a72f5\x2d8576\x2d4989\x2dacc3\x2dde6f15e14f46.device: Job dev-disk-by\x2duuid-7b5a72f5\x2d8576\>
As for what probably caused it all to happen I was trying to run a headless server for dominions5, so I installed it, tried to run, it missed some libs, so without second thought I just installed a couple of packets namely libglu1 and libsdl2-dev, and only after that I realized that I'm trying to start the server without propper option, and without them, I just trying to run a graphical binary, and I don't really need those libs on VDS. It may be the cause, but I'm not sure, something else may be the cause as well the server had a long uptime.
I removed those packets, and it didn't help, I also switched default systemd target from graphical.target to multi-user.target but it didn't change much.
I also checked SMART and all disks are fine.
So If anyone have any Ideas about it - I would be very grateful.
