Repeated corruption issues after expanding Raid6

Oa10712

5/1/24, 2:57 AM

I have a Raid6 array that has been functioning perfectly well for the past several years, but out of a desire to keep path names organized I expanded the array to include 2 additional devices. While the expansion itself went well and all data seemed accessible at first, after about a week I encountered an error while accessing a file stating that the filesystem needed to be checked. Running e3fsck showed several errors stating that various inodes "seem to contain garbage". While I had an extra backup and was able to restore the corrupted files, this corruption seems to recur once or twice a week on different locations and files each time.

While I do have backups, I'd like to at least identify the cause of this corruption and fix it so I don't need to take my systems offline to repair and restore files.

Running 18.04.6 LTS (GNU/Linux 4.15.0-210-generic x86_64), filesystem is ext4 and consists of 6 main and 2 backup 8 terabyte drives in a raid 6 format. The array prior to expansion had 4 main drives.

Drives are 4x HUH728080ALE601 and 4x HGST HUS728T8TAL

Initial error before running recovery tools is cannot access '*****': Structure needs cleaning

0 + 4

raid

mdadm

data-corruption

guiverc

5/1/24, 3:19 AM

FYI: Please don't forget that Ubuntu 18.04 LTS was released in 2018-April (*thus the 18.04*) with five years of *supported* life, thus is currently at the end of that *supported* life with mere hours/days remaining. I'd suggest planning your *release-upgrade* path asap, as once it's declared EOSS (*end of standard support*) it'll be off-topic on this site. You can still use 18.04 with ESM which has support options elsewhere; though your release is still on-topic here for now.

Oa10712

5/1/24, 3:24 AM

Will execute the release upgrade this week, though I don't have high expectations that will help.

matigo

5/1/24, 4:40 AM

Are there disk caching tools running on the volume? I've seen something similar in the past with Apache disk caching where files that go beyond the original volume size become corrupted because the caching tool considers the space beyond the original partition boundary as out of bounds. If any caching systems are in place, it may be worth examining how they are configured.

Oa10712

5/2/24, 11:04 PM

I haven't enabled anything like that to my knowledge, all of the Apache stuff on this server are on a different drive.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Repeated corruption issues after expanding Raid6

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.