Penguin server won't boot after swapping failed hard drive in RAID 6

Watchdogs1499

2/9/24, 6:02 PM

So I have a Penguin Relion 1900 1U rack mounted server running CentOS 7 that recently had a hard disk failure. It had 4 hard drives configured in RAID 6. To replace the failed HDD, I powered off the machine, swapped the failed one with a new drive of the same size (4TB) and powered on the machine again.

The system booted into the EFI shell instead of loading the OS. I exited from the shell and went to BIOS and noticed the status of the RAID array was rebuilding under the RAID menu in advanced setting of the BIOS.

6-7 hours later, I noticed the rebuild had completed and that the RAID status was in 'Optimal' condition with everything looking good. I saved and quit out of the BIOS, but it went into the EFI shell instead of loading the OS again.

I rebooted the machine and under the BIOS boot sequence priority list could not see the virtual drive provided by the RAID. But instead there was a "SCSI Hard Drive, ..." option.

My booting is set to UEFI mode, so I switched to Legacy under the CSM configuration settings in BIOS and when I reboot, it goes into network booting and the "SCSI Hard Drive ..." disappears from the boot order sequence list.

When in Legacy boot, I do get the option to load into the RAID BIOS, which I did to take a look. All status shows optimal and I can see the Virtual Drive created and present in the RAID BIOS. It's just not being picked up by the system BIOS and loading the OS.

I have attached pictures of my BIOS settings.

RAID setting in the System BIOS (only appears when UEFI booting, also notice boot device set to [None])

map -r output on UEFI shell

Boot order in BIOS

CSM config

Any help would be much appreciated. Already spent a whole day try to fix it!

UPDATE

So I tried booting off a CentOS USB and saw that the RAID volume is visible, but not mounted and not set as the boot drive. I cannot seem to mount it either and getting errors as seen in pics below.

fdisk -l

lsblk

1 + 2

linux

raid

hard-drive

boot

drive-failure

djdomi

2/9/24, 7:31 PM

restore from backup. No backups? Question is offtopic then

Watchdogs1499

2/10/24, 5:04 PM

@djdomi We do have a backup, but it is a month old. We'd prefer trying to restore our current configuration. As far as I can say, all our data is intact in the volume. It's just that the OS cannot see it or boot into it, and that's what I'm trying to figure out.

Score:0

Server

Watchdogs1499

2/21/24, 5:55 PM

UPDATE:

So after talking to manufacturer support and sending them the logs, they concluded that the boot loader got corrupted somehow or was overwritten. This does not normally happen and a simple swap of hard drives should not cause this problem, so it was a one off scenario.

HOW I FIXED IT:

I did have a backup of all the data. Bought fresh hard drives, rebuilt the array, reinstalled the OS and restored from backup.

+ 0

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Penguin server won't boot after swapping failed hard drive in RAID 6

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.