fsck failed on all (unrelated) servers

Zak

12/31/22, 9:33 PM

I am curios how this happens? I came in from the weekend and had to reboot two unrelated servers, exhibiting the same similar message:

fsck from util-linux 2.26.2
/dev/sda1 contains a file system with errors, check forced.
/dev/sda1: Inodes that were part of a corrupted orphan linked list found.

/dev/sda1: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
        (i.e., without -a or -p options)
fsck exited with status code 4
The root filesystem on /dev/sda1 requires a manual fsck

Busybox v1.22.1 (Ubuntu 1:1.22.0-15ubuntu1) built in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs) _

Both servers I had to run fsck manually, and both were easily repaired. The question is, one being a database server (production), and the other is a web development (non production) server. How does this happen? Both were running when I came into work .. And both failed at reboot. These are virtual drives btw ... Using VMWare ESXI Hypervisor.

I thought maybe an "unclean" shut down (power off) but the machines were running, and simply would not reboot until the fsck was completed.

UPDATE I just ran fsck on another non-production server in the same Hypervisor Application -- This makes 3 servers:

fsck from util-linux 2.27.1
e2fsck 1.42.13 (17-May-2015)
Warning!  /dev/mapper/ywpData--vg-root is mounted.
Warning: skipping journal recovery because doing a read-only filesystem check.
/dev/mapper/ywpData--vg-root: clean, 384516/6266880 files, 16531831/25039872 blocks
e2fsck 1.42.13 (17-May-2015)
Warning!  /dev/sda1 is mounted.
/dev/sda1 was not cleanly unmounted, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong (100531, counted=295192).
Fix? no

Free inodes count wrong (124599, counted=124616).
Fix? no

/dev/sda1: 329/124928 files (16.1% non-contiguous), 398157/498688 blocks

0 + 3

boot

fsck

20.04

user535733

12/31/22, 9:39 PM

Since these are all VM Guests, are they all on the same Host?

Zak

12/31/22, 9:43 PM

ESXI is a cloud of hosts -- So technically, yes .. although the hardware is "fluid"

Zak

12/31/22, 9:45 PM

Can strange happenings in physical drives (hot swaps etc) cause this sort of error on virtual drives? I though virtual drives were impervious to physical hardware failure.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: fsck failed on all (unrelated) servers

TH: fsck ล้มเหลวบนเซิร์ฟเวอร์ทั้งหมด (ที่ไม่เกี่ยวข้อง)

RO: fsck a eșuat pe toate serverele (fără legătură).

RU: Ошибка fsck на всех (не связанных) серверах

VI: fsck không thành công trên tất cả các máy chủ (không liên quan)

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.