Running xfs_repair on hardware raid while still rebuilding?

Jebediah Pennywhistle

10/26/23, 3:21 PM

Should I allow a hardware raid5 array rebuild to complete after swapping out a drive PRIOR to running an xfs_repair on the volume?

Currently xfs_repair keeps failing in Phase 7 at the same spot: Phase 7 - verify and correct link counts... Metadata corruption detected at 0x45bf78, xfs_dir3_block block 0x6a945ef98/0x1000 libxfs_bwrite: write verifier failed on xfs_dir3_block bno 0x6a945ef98/0x8 xfs_repair: Releasing dirty buffer to free list! xfs_repair: Refusing to write a corrupt buffer to the data device! xfs_repair: Lost a write to the data device!

Basically: Am I being stupid? Is this an instance of:

"Of COURSE I'm seeing data weirdness while the array is being rebuilt" and should just leave it TF alone and let it finish... probably sometime tomorrow or the day after... before I then get started on the xfs_repair.

eg: We'll wait until you're ready.

-OR- 2) I can continue to work on the XFS filesystem WHILE the raid5 hardware array rebuild is also going on, and once the XFS filesystem repair is complete mount the volume, rsync the backed up data over to the primary partition, and get on with my life while the hardware raid rebuild continues to potato chug it's way along but IDGAF because the system is back up and it can take however long it wants so long as it eventually completes.

I Hate Waiting!

Background info: I have a 16TB XFS filesystem running on a Hardware RAID5 in a HP DL380 with a Smart Array P410i controller. I am consistently running into filesystem corruption issues with an XFS filesystem. I've swapped out the controller, and swapped out one of the 6 4TB drives which I suspect might have been the culprit.

The array rebuild is taking quite a long time, and I'd like to get this system back up and running fairly quickly. The hardware recovery / rebuild is at 56% now, after running for a day. This isn't hugely abnormal for this controller and an array this size. The issue I'm facing however is I want to get the xfs_repair started. When I attempted the first xfs_repair, it said the filesystem was dirty and needed to be mounted again. Fine. Mount / umount was done. No drama there. Kicked off the xfs_repair and it started working it's way through the inodes and metadata corruption.

It then occurred to me, that maybe running the xfs_repair while the rebuild was still ongoing might be a... let's say not a great idea. I have all the data backed up on a large drive shelf backup, so I'm not hugely concerned about files being shuffled off to the nomans land of lost+found. I'll rsync the (slower 63TB) drive shelf backup array over to the (faster 16TB) "production" array once the array rebuild completes. That in theory would still be faster than nuking the 16TB array and doing a full restore from backup... but:

Should I step back, stuff my hands in my pockets, glower at the % complete S_L_O_W_L_Y claw it's way to 100%, allow the hardware raid5 array to complete the RAID5 array rebuild PRIOR to running the xfs_repair -OR- can I continue trying to complete the xfs_repair while the array rebuild slowly, painfully, continues to chug along?

218

0 + 4

linux

raid

xfs

filesystems

hardware-raid

doneal24

10/26/23, 5:02 PM

Running `xfs_repair` while the raid is rebuild should not be a problem as the file system and the underlying hardware are mostly independent of each other. One possible concern after reading your post is the use of raid 5. HPE recommends this only if the disks are SSDs. If they're spinning rust you have a good chance of a second disk failure during a rebuild. Any large array of spinning disks really need to be raid 6.

Jebediah Pennywhistle

10/26/23, 7:45 PM

I'm just going to toss the xfs partition. It's still refusing to repair. At this point I've lost ALL faith in xfs, so back to ext4! I've lost so many xfs filesystems at this point, on so many different systems, I don't trust it anymore. So, I'll burn the partition and do a complete restore from the backup.

Jebediah Pennywhistle

10/26/23, 7:47 PM

Yes, I'm aware of the "lose two drives and lose all your data" problems with RAID5. For this array, cost, speed, and capacity are more important than multi drive fault tolerance. If I drive drops, I can always swap it out. Many times pulling the drive, wait and throw in back in the cage works just fine. These HP raid controllers are SO quick to toss a drive out of the array it's ridiculous. The drives I'm running are enterprise drives listed on the supported config list from HP, but looking at the logs, if they are even just slow to respond BONK it's thrown out for a single timeout error.

doneal24

10/26/23, 7:53 PM

HPE iLO diagnostics will mark a disk as failing very quickly. Not completely awful if you have good hardware support and appropriate hardware redundancy but still a pain to receive the disk, replace the disk, and send the "failing" disk back to HPE. Even with HPE's paranoia, it would be a hard sell to use a raid 5. The overhead difference between it and raid 6 is small enough that I would sleep better at night.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Running xfs_repair on hardware raid while still rebuilding?

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.