mdadm - cant read superblock - can I zero the superblocks without erasing data and recreate a raid?

Question

Score:1

Server

mdadm - cant read superblock - can I zero the superblocks without erasing data and recreate a raid?

G4nja Wizard

12/23/22, 3:11 PM

Today I had to fix a Server Fan by disassembling and rebuild it. After booting the server I had the problem that all disks where not at the same location they have been yesterday. So the system couldnt mount my array. Now all disks have the old path as it was before... But my system is unable to mount them on boot, giving me the error "cant read superblock" I am not that familiar with raid configurations. Thats why I am always scared of running the wrong command and wiping all disks.

Thats why I am asking, is it safe to zero the superblocks and recreate a "new" raid5 array?

cat /proc/mdstat 
Personalities : 
md127 : inactive sda[0](S) sdb[1](S) sdd[4](S) sdc[2](S)
      7813529952 blocks super 1.2
       
unused devices: <none>

I tried the following:

# mdadm --stop /dev/md127
mdadm: stopped /dev/md127

# mdadm --assemble --scan -v
mdadm: /dev/sdc is identified as a member of /dev/md/vo1, slot 2.
mdadm: /dev/sdd is identified as a member of /dev/md/vo1, slot 3.
mdadm: /dev/sdb is identified as a member of /dev/md/vo1, slot 1.
mdadm: /dev/sda is identified as a member of /dev/md/vo1, slot 0.
mdadm: added /dev/sda to /dev/md/vo1 as 0 (possibly out of date)
mdadm: added /dev/sdb to /dev/md/vo1 as 1 (possibly out of date)
mdadm: added /dev/sdd to /dev/md/vo1 as 3
mdadm: added /dev/sdc to /dev/md/vo1 as 2
mdadm: /dev/md/vo1 assembled from 2 drives - not enough to start the array.

Details show me that it has set Raid0 instead of Raid5.

# mdadm --detail /dev/md127
/dev/md127:
           Version : 1.2
        Raid Level : raid0
     Total Devices : 4
       Persistence : Superblock is persistent

             State : inactive
   Working Devices : 4

              Name : ncloud:vo1  (local to host ncloud)
              UUID : b8ecad1a:56e6c31c:35bb6532:3dd2f9c7
            Events : 5102

    Number   Major   Minor   RaidDevice

       -       8       32        -        /dev/sdc
       -       8        0        -        /dev/sda
       -       8       48        -        /dev/sdd
       -       8       16        -        /dev/sdb

There are 2 disks that have each 1 missing state.

Edit #2

root@ncloud:~# mdadm --examine /dev/sd[abcd]
/dev/sda:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : b8ecad1a:56e6c31c:35bb6532:3dd2f9c7
           Name : ncloud:vo1  (local to host ncloud)
  Creation Time : Wed Dec  9 13:01:02 2020
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)
     Array Size : 5860147200 (5588.67 GiB 6000.79 GB)
  Used Dev Size : 3906764800 (1862.89 GiB 2000.26 GB)
    Data Offset : 264192 sectors
   Super Offset : 8 sectors
   Unused Space : before=264112 sectors, after=176 sectors
          State : clean
    Device UUID : c0e60330:de836b95:5dd932a0:fe90c3dd

Internal Bitmap : 8 sectors from superblock
    Update Time : Mon Aug 23 00:45:53 2021
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : 96c1485f - correct
         Events : 5102

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 0
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : b8ecad1a:56e6c31c:35bb6532:3dd2f9c7
           Name : ncloud:vo1  (local to host ncloud)
  Creation Time : Wed Dec  9 13:01:02 2020
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)
     Array Size : 5860147200 (5588.67 GiB 6000.79 GB)
  Used Dev Size : 3906764800 (1862.89 GiB 2000.26 GB)
    Data Offset : 264192 sectors
   Super Offset : 8 sectors
   Unused Space : before=264112 sectors, after=176 sectors
          State : clean
    Device UUID : 27b3f2cd:ca5d1594:625c4478:48e2a2d6

Internal Bitmap : 8 sectors from superblock
    Update Time : Mon Aug 23 00:45:53 2021
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : 44ac303 - correct
         Events : 5102

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 1
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : b8ecad1a:56e6c31c:35bb6532:3dd2f9c7
           Name : ncloud:vo1  (local to host ncloud)
  Creation Time : Wed Dec  9 13:01:02 2020
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)
     Array Size : 5860147200 (5588.67 GiB 6000.79 GB)
  Used Dev Size : 3906764800 (1862.89 GiB 2000.26 GB)
    Data Offset : 264192 sectors
   Super Offset : 8 sectors
   Unused Space : before=264112 sectors, after=176 sectors
          State : clean
    Device UUID : 5e2a55e9:81dce8d4:8e4f5dce:e270f7a5

Internal Bitmap : 8 sectors from superblock
    Update Time : Mon Aug 23 09:48:16 2021
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : 85efb9d6 - correct
         Events : 5102

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 2
   Array State : .AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : b8ecad1a:56e6c31c:35bb6532:3dd2f9c7
           Name : ncloud:vo1  (local to host ncloud)
  Creation Time : Wed Dec  9 13:01:02 2020
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)
     Array Size : 5860147200 (5588.67 GiB 6000.79 GB)
  Used Dev Size : 3906764800 (1862.89 GiB 2000.26 GB)
    Data Offset : 264192 sectors
   Super Offset : 8 sectors
   Unused Space : before=264112 sectors, after=176 sectors
          State : clean
    Device UUID : 22630b19:7930354b:7e3784a8:4f7396b7

Internal Bitmap : 8 sectors from superblock
    Update Time : Mon Aug 23 09:48:16 2021
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : 17b830f0 - correct
         Events : 5102

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 3
   Array State : .AAA ('A' == active, '.' == missing, 'R' == replacing)

EDIT #3

OK root@ncloud:~# mdadm --stop /dev/md127
mdadm: stopped /dev/md127
OK root@ncloud:~# mdadm --assemble --force /dev/md127 /dev/sd[abcd]
mdadm: /dev/md127 has been started with 4 drives.
OK root@ncloud:~# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] 
md127 : active (auto-read-only) raid5 sda[0] sdd[4] sdc[2] sdb[1]
      5860147200 blocks super 1.2 level 5, 128k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/15 pages [0KB], 65536KB chunk

unused devices: <none>

root@ncloud:~# mdadm --detail /dev/md127 
/dev/md127:
           Version : 1.2
     Creation Time : Wed Dec  9 13:01:02 2020
        Raid Level : raid5
        Array Size : 5860147200 (5588.67 GiB 6000.79 GB)
     Used Dev Size : 1953382400 (1862.89 GiB 2000.26 GB)
      Raid Devices : 4
     Total Devices : 4
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Mon Aug 23 00:45:53 2021
             State : clean 
    Active Devices : 4
   Working Devices : 4
    Failed Devices : 0
     Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 128K

Consistency Policy : bitmap

              Name : ncloud:vo1  (local to host ncloud)
              UUID : b8ecad1a:56e6c31c:35bb6532:3dd2f9c7
            Events : 5102

    Number   Major   Minor   RaidDevice State
       0       8        0        0      active sync   /dev/sda
       1       8       16        1      active sync   /dev/sdb
       2       8       32        2      active sync   /dev/sdc
       4       8       48        3      active sync   /dev/sdd

Edit 4

After scaning details to mdadm.conf, uncommenting my line in fstab (UUID of array was the same), also update-initramfs -u and rebooting. I saw the following in dmesg:

Error root@ncloud:~# dmesg | grep block
[   11.326178] sd 0:0:0:0: [sda] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[   11.328529] sd 0:0:0:1: [sdb] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[   12.198036] sd 1:0:0:0: [sdc] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[   12.200996] sd 1:0:0:1: [sdd] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[  144.497538] EXT4-fs (md127): unable to read superblock

Personalities : 
md127 : inactive sdd[4](S) sdc[2](S) sda[0](S) sdb[1](S)
      7813529952 blocks super 1.2
       
unused devices: <none>

EDIT 5 DMESG

I just booted up the system and none of the HDD's came up. I always shut down systems when I go to bed.

[   44.331998] sd 0:0:0:1: [sdb] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[   44.332253] sd 0:0:0:1: [sdb] Write Protect is off
[   44.332262] sd 0:0:0:1: [sdb] Mode Sense: 67 00 10 08
[   44.333022] sd 0:0:0:1: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
[   44.333748] sd 0:0:0:1: [sdb] Optimal transfer size 33553920 bytes
[   44.335655] sd 0:0:0:0: [sda] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[   44.335907] sd 0:0:0:0: [sda] Write Protect is off
[   44.335916] sd 0:0:0:0: [sda] Mode Sense: 67 00 10 08
[   44.336385] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
[   44.337327] sd 0:0:0:0: [sda] Optimal transfer size 33553920 bytes
[   44.574978] usb 2-2: USB disconnect, device number 15
[   44.635129] sd 0:0:0:0: [sda] Read Capacity(16) failed: Result: hostbyte=0x07 driverbyte=0x00
[   44.635152] sd 0:0:0:0: [sda] Sense not available.
[   44.635234] sd 0:0:0:0: [sda] Read Capacity(10) failed: Result: hostbyte=0x07 driverbyte=0x00
[   44.635253] sd 0:0:0:0: [sda] Sense not available.
[   44.635322] sd 0:0:0:0: [sda] 0 512-byte logical blocks: (0 B/0 B)
[   44.635366] sd 0:0:0:0: [sda] Write Protect is on
[   44.635386] sd 0:0:0:0: [sda] Mode Sense: 5c 10 8b c4
[   44.636524] sd 0:0:0:0: [sda] Read Capacity(16) failed: Result: hostbyte=0x07 driverbyte=0x00
[   44.636544] sd 0:0:0:0: [sda] Sense not available.
[   44.636625] sd 0:0:0:0: [sda] Read Capacity(10) failed: Result: hostbyte=0x07 driverbyte=0x00
[   44.636643] sd 0:0:0:0: [sda] Sense not available.
[   44.636865] sd 0:0:0:0: [sda] Attached SCSI disk
[   44.815456] sd 0:0:0:1: [sdb] Read Capacity(16) failed: Result: hostbyte=0x07 driverbyte=0x00
[   44.815467] sd 0:0:0:1: [sdb] Sense not available.
[   44.815500] sd 0:0:0:1: [sdb] Read Capacity(10) failed: Result: hostbyte=0x07 driverbyte=0x00
[   44.815507] sd 0:0:0:1: [sdb] Sense not available.
[   44.815535] sd 0:0:0:1: [sdb] 0 512-byte logical blocks: (0 B/0 B)
[   44.816034] sd 0:0:0:1: [sdb] Read Capacity(16) failed: Result: hostbyte=0x07 driverbyte=0x00
[   44.816042] sd 0:0:0:1: [sdb] Sense not available.
[   44.816075] sd 0:0:0:1: [sdb] Read Capacity(10) failed: Result: hostbyte=0x07 driverbyte=0x00
[   44.816082] sd 0:0:0:1: [sdb] Sense not available.
[   44.816132] sd 0:0:0:1: [sdb] Attached SCSI disk
[   46.625314] usb 2-2: Device not responding to setup address.
[   46.845292] usb 2-2: Device not responding to setup address.
[   47.064953] usb 2-2: device not accepting address 16, error -71
[   47.295335] usb 2-2: Device not responding to setup address.
[   47.515295] usb 2-2: Device not responding to setup address.
[   47.734964] usb 2-2: device not accepting address 17, error -71
[   47.743443] usb usb2-port2: attempt power cycle
[   48.505310] usb 2-2: Device not responding to setup address.
[   48.725233] usb 2-2: Device not responding to setup address.
[   48.944928] usb 2-2: device not accepting address 18, error -71
[   49.175256] usb 2-2: Device not responding to setup address.
[   49.395303] usb 2-2: Device not responding to setup address.
[   49.614958] usb 2-2: device not accepting address 19, error -71
[   49.623420] usb usb2-port2: unable to enumerate USB device
[   63.844995] usb 1-1.1: new high-speed USB device number 4 using xhci_hcd
[   64.054967] usb 2-1: USB disconnect, device number 3
[   64.068262] sd 1:0:0:0: [sdc] Synchronizing SCSI cache
[   64.685012] sd 1:0:0:0: [sdc] Synchronize Cache(10) failed: Result: hostbyte=0x07 driverbyte=0x00
[   64.767524] sd 1:0:0:1: [sdd] Synchronizing SCSI cache
[   65.365009] sd 1:0:0:1: [sdd] Synchronize Cache(10) failed: Result: hostbyte=0x07 driverbyte=0x00
[   65.755417] usb 2-1: Device not responding to setup address.
[   65.975280] usb 2-1: Device not responding to setup address.
[   66.194952] usb 2-1: device not accepting address 20, error -71
[   66.425315] usb 2-1: Device not responding to setup address.
[   66.645324] usb 2-1: Device not responding to setup address.
[   66.864943] usb 2-1: device not accepting address 21, error -71
[   66.873409] usb usb2-port1: attempt power cycle
[   67.645288] usb 2-1: Device not responding to setup address.
[   67.865275] usb 2-1: Device not responding to setup address.
[   68.084979] usb 2-1: device not accepting address 22, error -71
[   68.315282] usb 2-1: Device not responding to setup address.
[   68.535306] usb 2-1: Device not responding to setup address.
[   68.754993] usb 2-1: device not accepting address 23, error -71
[   68.763463] usb usb2-port1: unable to enumerate USB device
[  267.345137] usb 2-2: Device not responding to setup address.
[  267.565113] usb 2-2: Device not responding to setup address.
[  267.784800] usb 2-2: device not accepting address 24, error -71
[  268.895181] usb 2-2: Device not responding to setup address.
[  269.115322] usb 2-2: Device not responding to setup address.
[  269.334860] usb 2-2: device not accepting address 25, error -71
[  269.335263] usb usb2-port2: attempt power cycle
[  270.545312] usb 2-2: Device not responding to setup address.
[  270.765244] usb 2-2: Device not responding to setup address.
[  270.984922] usb 2-2: device not accepting address 26, error -71
[  271.215265] usb 2-2: Device not responding to setup address.
[  271.435271] usb 2-2: Device not responding to setup address.
[  271.654947] usb 2-2: device not accepting address 27, error -71
[  271.655343] usb usb2-port2: unable to enumerate USB device
[  272.085001] usb 1-1.1: new high-speed USB device number 5 using xhci_hcd
[  273.145469] usb 2-2: Device not responding to setup address.
[  273.365365] usb 2-2: Device not responding to setup address.
[  273.585013] usb 2-2: device not accepting address 28, error -71
[  273.815367] usb 2-2: Device not responding to setup address.
[  274.035413] usb 2-2: Device not responding to setup address.
[  274.255037] usb 2-2: device not accepting address 29, error -71
[  274.263447] usb usb2-port2: attempt power cycle
[  275.025412] usb 2-2: Device not responding to setup address.
[  275.245477] usb 2-2: Device not responding to setup address.
[  275.465093] usb 2-2: device not accepting address 30, error -71
[  275.695434] usb 2-2: Device not responding to setup address.
[  275.915439] usb 2-2: Device not responding to setup address.
[  276.135142] usb 2-2: device not accepting address 31, error -71
[  276.143608] usb usb2-port2: unable to enumerate USB device
[  277.225494] usb 2-1: Device not responding to setup address.
[  277.445492] usb 2-1: Device not responding to setup address.
[  277.665166] usb 2-1: device not accepting address 32, error -71
[  278.705210] usb 1-1.1: new high-speed USB device number 6 using xhci_hcd
[  279.875710] usb 2-1: Device not responding to setup address.
[  280.095620] usb 2-1: Device not responding to setup address.
[  280.315260] usb 2-1: device not accepting address 34, error -71
[  280.545634] usb 2-1: Device not responding to setup address.
[  280.765611] usb 2-1: Device not responding to setup address.
[  280.985280] usb 2-1: device not accepting address 35, error -71
[  280.993740] usb usb2-port1: attempt power cycle
[  281.755646] usb 2-1: Device not responding to setup address.
[  281.975646] usb 2-1: Device not responding to setup address.
[  282.195368] usb 2-1: device not accepting address 36, error -71
[  282.425667] usb 2-1: Device not responding to setup address.
[  282.645665] usb 2-1: Device not responding to setup address.
[  282.865366] usb 2-1: device not accepting address 37, error -71
[  282.873826] usb usb2-port1: unable to enumerate USB device
[  283.105694] usb 2-2: Device not responding to setup address.
[  283.325685] usb 2-2: Device not responding to setup address.
[  283.545360] usb 2-2: device not accepting address 38, error -71
[  283.775712] usb 2-2: Device not responding to setup address.
[  283.995714] usb 2-2: Device not responding to setup address.
[  284.215384] usb 2-2: device not accepting address 39, error -71
[  284.223859] usb usb2-port2: attempt power cycle
[  284.985762] usb 2-2: Device not responding to setup address.
[  285.205835] usb 2-2: Device not responding to setup address.
[  285.425431] usb 2-2: device not accepting address 40, error -71
[  285.655770] usb 2-2: Device not responding to setup address.
[  285.875801] usb 2-2: Device not responding to setup address.
[  286.095503] usb 2-2: device not accepting address 41, error -71
[  286.103979] usb usb2-port2: unable to enumerate USB device

EDIT 6

For me dmesg's errors seemed like my pi suffers from power. I have an Allnet NAS with a Rock-Pi SATA head. It will be powered by a 45W power supply via USB-C. The HDD's itself are connected via SATA and GPIO Ports on the Raspberry, but the HDD's are additionally connected via USB3<->USB3 port connection from the PI4 board to Rock-Pi SATA port. I unplugged this. Booted the system and after booting adding the USB3 Ports back. After this dmesg is silent about read errors. IT seems fine as nothing has happened... really strange.

261

2 + 7

raid

hard-drive

mdadm

software-raid

mdadm - cant read superblock - can I zero the superblocks without erasing data and recreate a raid?

Post an answer