Join Log in
Score:1
tjgrillz avatar Server

md Raid6 array disapeered on reboot

aq flag
tjgrillz

my raid6 array disappeared on reboot after I grew it. I believe the issue was growing 2x with the full disk not partition. It has been suggested a possible other reason the drives were not correctly recognized is I didn't zero the superblocks before readding to a new array. It could be a combination of both? Here are the commands issued (pulled from history, formatted to have consistent drive letters):

mdadm --create --verbose /dev/md0 --level=0 --raid-devices=2 /dev/sd[b-c]1

#Full backup of ROC raid 10 onto these drives, after having copied most files to other drives, check to make sure worked through reboot.

mdadm --create /dev/md1 --level=6 --raid-devices=4 /dev/sd[d-g]1

#Time passed to sync drives, and then rsync data from md0, reboots fine.

mdadm -S /dev/md0
mdadm /dev/md0 -r /dev/sd[b-c]

#NOTICE THE MISSING PARTITION NUMBER BELOW.

mdadm /dev/md1 --add /dev/sdb
mdadm /dev/md1 --add /dev/sdc
mdadm -list
mdadm --detail /dev/md1
mdadm --grow --raid-devices=6 --backup-file=/media/FastRaid/md1_grow.bak /dev/md1

After a reboot, the raid6 disapeered and is replaced by 2 raid0 arrays, one active (sdb/sdc) and one inactive (sdd-sdg). The the following is what I get from examining the superblocks:

/dev/sdb1:
        Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
    Array UUID : 501c08da:5069a3d8:b2982a5d:ab56c37c
        Name : tim-server:0  (local to host tim-server)
Creation Time : Tue Dec 13 22:01:10 2022
    Raid Level : raid0
Raid Devices : 2

Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
    Data Offset : 264192 sectors
Super Offset : 8 sectors
Unused Space : before=264112 sectors, after=0 sectors
        State : clean
    Device UUID : e8db27d6:0dbd1ac5:4456c304:0b43f09c

    Update Time : Tue Dec 13 22:01:10 2022
Bad Block Log : 512 entries available at offset 8 sectors
    Checksum : dfd187c0 - correct
        Events : 0

    Chunk Size : 512K

Device Role : Active device 0
Array State : AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
        Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
    Array UUID : 501c08da:5069a3d8:b2982a5d:ab56c37c
        Name : tim-server:0  (local to host tim-server)
Creation Time : Tue Dec 13 22:01:10 2022
    Raid Level : raid0
Raid Devices : 2

Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
    Data Offset : 264192 sectors
Super Offset : 8 sectors
Unused Space : before=264112 sectors, after=0 sectors
        State : clean
    Device UUID : 3ce84b05:607f8565:456e7f83:88b83052

    Update Time : Tue Dec 13 22:01:10 2022
Bad Block Log : 512 entries available at offset 8 sectors
    Checksum : e35ce3e5 - correct
        Events : 0

    Chunk Size : 512K

Device Role : Active device 1
Array State : AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd1:
        Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
    Array UUID : 929a14c9:adaf502a:53658e03:90a19fce
        Name : tim-server:0  (local to host tim-server)
Creation Time : Wed Dec 14 11:18:57 2022
    Raid Level : raid6
Raid Devices : 6

Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
    Array Size : 15627540480 (14903.58 GiB 16002.60 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
    Data Offset : 264192 sectors
Super Offset : 8 sectors
Unused Space : before=264112 sectors, after=655 sectors
        State : clean
    Device UUID : eaf10189:940aeaf8:947efe82:5d0e4aea

Internal Bitmap : 8 sectors from superblock
    Update Time : Sun Dec 18 06:31:11 2022
Bad Block Log : 512 entries available at offset 24 sectors
    Checksum : e38a1bd9 - correct
        Events : 26630

        Layout : left-symmetric
    Chunk Size : 512K

Device Role : Active device 1
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde1:
        Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
    Array UUID : 929a14c9:adaf502a:53658e03:90a19fce
        Name : tim-server:0  (local to host tim-server)
Creation Time : Wed Dec 14 11:18:57 2022
    Raid Level : raid6
Raid Devices : 6

Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
    Array Size : 15627540480 (14903.58 GiB 16002.60 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
    Data Offset : 264192 sectors
Super Offset : 8 sectors
Unused Space : before=264112 sectors, after=655 sectors
        State : clean
    Device UUID : 5c34a9c7:bcc3f190:d1719a9c:8aa2b722

Internal Bitmap : 8 sectors from superblock
    Update Time : Sun Dec 18 06:31:11 2022
Bad Block Log : 512 entries available at offset 24 sectors
    Checksum : c429edf - correct
        Events : 26630

        Layout : left-symmetric
    Chunk Size : 512K

Device Role : Active device 3
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf1:
        Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
    Array UUID : 929a14c9:adaf502a:53658e03:90a19fce
        Name : tim-server:0  (local to host tim-server)
Creation Time : Wed Dec 14 11:18:57 2022
    Raid Level : raid6
Raid Devices : 6

Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
    Array Size : 15627540480 (14903.58 GiB 16002.60 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
    Data Offset : 264192 sectors
Super Offset : 8 sectors
Unused Space : before=264112 sectors, after=655 sectors
        State : clean
    Device UUID : 12d1e3a8:b8749f59:654bcca4:4f4750df

Internal Bitmap : 8 sectors from superblock
    Update Time : Sun Dec 18 06:31:11 2022
Bad Block Log : 512 entries available at offset 24 sectors
    Checksum : 7af56ae7 - correct
        Events : 26630

        Layout : left-symmetric
    Chunk Size : 512K

Device Role : Active device 0
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg1:
        Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
    Array UUID : 929a14c9:adaf502a:53658e03:90a19fce
        Name : tim-server:0  (local to host tim-server)
Creation Time : Wed Dec 14 11:18:57 2022
    Raid Level : raid6
Raid Devices : 6

Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
    Array Size : 15627540480 (14903.58 GiB 16002.60 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
    Data Offset : 264192 sectors
Super Offset : 8 sectors
Unused Space : before=264112 sectors, after=655 sectors
        State : clean
    Device UUID : 72085967:835efe92:cb268a64:4d192b52

Internal Bitmap : 8 sectors from superblock
    Update Time : Sun Dec 18 06:31:11 2022
Bad Block Log : 512 entries available at offset 24 sectors
    Checksum : a5623977 - correct
        Events : 26630

        Layout : left-symmetric
    Chunk Size : 512K

Device Role : Active device 2
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)

I had inactivated md0 at some point, so I recreated with mdadm -A -o /dev/md0 /dev/sdb1 /dev/sdc1. This is a /proc/mdstat now:

cat /proc/mdstat

Personalities : [raid0] [linear] [multipath] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active (read-only) raid0 sdb1[0] sdc1[1]
  7813770240 blocks super 1.2 512k chunks

md1 : inactive sdf1[0](S) sde1[3](S) sdd1[1](S) sdg1[2](S)
    15627541790 blocks super 1.2

unused devices: <none>

If I try to mount /dev/md0 /media/tmp_md_raid I get: mount: /media/tmp_md_raid: wrong fs type, bad option, bad superblock on /dev/md126, missing codepage or helper program, or other error.. If I try: mdadm -A -o /dev/md1 /dev/sdf1 /dev/sde1 /dev/sdd1 /dev/sdg1 I get:

mdadm: /dev/sdf1 is busy - skipping
mdadm: /dev/sde1 is busy - skipping
mdadm: /dev/sdd1 is busy - skipping
mdadm: /dev/sdg1 is busy - skipping

All of the smartctl's say all the drives are fine. I am not sure if I should try an mdadm --assemble --force first or an mdadm --create --assume-clean first. Should I try the 2nd with -o set to see if I can recreate the array and view the data without possibly destroying the recovery? Thanks for any advice.

102
1 + 6
ng flag
Halfgaar
Did `mdadm /dev/md0 -r /dev/sd[b-c]` work? Can one remove the whole drive from an array made of partitions? And did `mdadm /dev/md1 --add /dev/sdb` work? Can one add drives to a RAID6 array without issuing any kind of grow command first? What I would have done, is zero the `sda` and `sdb`; completely or otherwise with `--zero-superblock`. Because of all these uncertainties, I don't have any advice on how to procede. You can start anew with the original ROC, or clone the component drives to back them up somewhere if you need to experiment.
2
Reply
Nikita Kipriyanov avatar
za flag
Nikita Kipriyanov
The first command should have been failed. You can't remove "good" drives from the array. You have to fail (mdadm /dev/mdX -f /dev/sdX) them first. Then, the second command could not complete because the devices were busy (in the old array). // It is okay to put /dev/sdXN and /dev/sdY into the same array. Should be no problems. What I see is that second (RAID6) array looks fully assembled (all "A"'s, meaning all devices are there). What means "inactive" then? Where is the `/proc/mdstat` contents? Is it where you see "inactive"? Array will be made active automatically once you interact with it.
1
Reply
Nikita Kipriyanov avatar
za flag
Nikita Kipriyanov
Also, it is pointless to remove devices from RAID0 array. Just destroy it altogether: mdadm --stop /dev/md0 and when this succeeds do with the devices whatever you want, probably begin with mdadm --zero-superblock /dev/sd[bc]1. Also, devices will be added into new array as *spares*, to make them fully participate in it (and increase the capacity) you need to grow the array *after* addtition of new (spare) devices.
0
Reply
tjgrillz avatar
aq flag
tjgrillz
I added /proc/mdstat.
1
Reply
tjgrillz avatar
aq flag
tjgrillz
I added the stop command, so the /dev/md0 array was successfully stopped and taken apart. I didn't zero them though before adding to the /dev/md1 array as was suggested. Would it make sense (to check/freeze data) to issue following commands (with no possible loss of future recovery/ie no disk writes): `mdadm -A /dev/sd[b-c] -o` and `mdadm -A /dev/sd[d-g] -o`
0
Reply
Nikita Kipriyanov avatar
za flag
Nikita Kipriyanov
Well. All participating devices have correct checksums for superblocks and all experienced the same number of events, so they are in sync. But it shows all devices as spares and says it is "busy". What it says about `md1` in `dmesg`? What is in `/sys/block/sdb/sd[d-g]1/holders` (e.g. who could be making them busy)? It may be worth looking at `lsblk`.
0
Reply
Score:0
Nikita Kipriyanov avatar Server
za flag
Nikita Kipriyanov

It seems you have 6-device array (AAAAAA), but only 4 component devices are available (/dev/sd[defg]1). The capacity calculation confirms this: one needs 6 disks of 4TB to create 16TB RAID6 array.

Since this is RAID6 and all 4 available devices seem to be in sync, it can be run, but only in so called fully degraded mode. In this mode to read any block it needs to read a stripe from all drives (which is I/O intensive) and perform a reconstruction (it uses both parity syndroms which involves CPU-intensive Galois field calculations), and to write a block it needs to do read the whole stripe, calculate new parity syndroms and write to at least three devices (which is even more I/O-intensive overall).

Linux has no other way to go but fall back to this if the array was running and some device failed in the middle of the use, that's the whole point of having a RAID array. As you may have guessed, the performance in this state is very bad and the risk of losing data is very high, which is why you shouldn't run an array in this state for long stretches of time. Ideally you supply a hot spare in addition to working devices so it will be able to start reconstruction immediately it detects the failure of any component.

But during the boot it doesn't know are some devices permanently missing or are they just not available yet due to staggered spin-up or other initialization delays. Activating array early will kick late devices out of sync and force a time-lengthy resync, during which the array will experience the worst performance characteristic as described above. This motivates to wait for late appearing devices. Linux will not activate partially-available array by default automatically, even if there are enough devices to run it at least in some degraded mode.

But you, an administrator, can force it to do so. For this, reassemble the array with --force:

mdadm --stop /dev/md1
mdadm --force --assemble /dev/md1 /dev/sd[defg]1

More precisely, it won't automatically assemble the array if there are less devices available than are recorded in superblocks of present devices (in your case it is recorded that all devices were available last time); when you remove a device properly with mdadm -f/ mdadm -r sequence, or when force assembling it, it records that and the array will then be auto-assembled in same degraded state automatically.

If this array doesn't contain valuable data, better re-create it. Initialization feels faster than adding devices and suffering reconstruction.

+ 2
tjgrillz avatar
aq flag
tjgrillz
Would you recommend to guarentee no data loss first running `mdadm --force --assemble -o /dev/md1 /dev/sd[defg]1`, duplicating any stuff not already backed up then trying to re add the drives?
0
Reply
Nikita Kipriyanov avatar
za flag
Nikita Kipriyanov
Generally speaking, when all drives are good (no bad blocks/suspicious behaviour) and synched (as they are in your case), force assembly is safe operation. If in doubt, work out the process using overlays to protect actual drives from any writes [using qemu-nbd](https://serverfault.com/questions/1101514/unable-to-mount-an-xfs-filesystem-from-linux-raid6-array-log-inconsistent/1101520#1101520) as described in my other answer or [device mapper managed sparse overlay file](https://raid.wiki.kernel.org/index.php/Recovering_a_damaged_RAID) as described in the RAID wiki, then assemble upper devices.
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.