Join Log in
Score:1
Charles Chou avatar Server

Restore 4-Drive RAID5 array after accidentally reinitializing last 2-Drive as RAID0

us flag
Charles Chou

I have an Asustor NAS with RAID 5 4-Drive running, after system update it reboot into the initialize page in web console, I think it's part of the upgrade process so I started the initialize progress, after few minute later I felt something wrong and unplug the power, the NAS boot into a clean OS, all setting has GONE and unable to mount the RAID.

after checking mdadm and fdisk in terminal, I found out that the last 2 drive has been reinitialize to RAID 0 array(sdc4, sdd4).

I have tried to assemble the original RAID but with no luck

# mdadm --assemble /dev/mdx /dev/sd*4
mdadm: superblock on /dev/sdc4 doesn't match others - assembly aborted

here is the result of mdadm --examine /dev/sd* the original RAID should be [sda4,sdb4,sdc4,sdd4] UUID: 1ba5dfd1:e861b791:eb307ef1:4ae4e4ad 8T.
the accidentally created raid0 is [sdc4,sdd4] UUID: 06b57325:241ba722:6dd303af:baaa5e4e

/dev/sda:
   MBR Magic : aa55
Partition[0] :       522240 sectors at         2048 (type 83)
Partition[3] :         2047 sectors at            1 (type ee)
mdadm: No md superblock detected on /dev/sda1.
/dev/sda2:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 1c90030d:10445d9f:d39fc32a:06d4b79a
           Name : AS1004T-7CBC:0  (local to host AS1004T-7CBC)
  Creation Time : Sun Jun 11 10:56:28 2017
     Raid Level : raid1
   Raid Devices : 4

 Avail Dev Size : 4190208 (2046.34 MiB 2145.39 MB)
     Array Size : 2095104 (2046.34 MiB 2145.39 MB)
    Data Offset : 4096 sectors
   Super Offset : 8 sectors
   Unused Space : before=4008 sectors, after=0 sectors
          State : active
    Device UUID : cca1545a:14112668:0ebd0ed3:df55018d

    Update Time : Sun Oct 13 01:05:27 2019
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 95866108 - correct
         Events : 228987


   Device Role : Active device 3
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sda3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 8c3ca866:3e6b6804:32f2955e:1b955d76
           Name : AS1004T-7CBC:126  (local to host AS1004T-7CBC)
  Creation Time : Sun May 14 09:50:45 2023
     Raid Level : raid1
   Raid Devices : 4

 Avail Dev Size : 4190208 (2046.34 MiB 2145.39 MB)
     Array Size : 2095104 (2046.34 MiB 2145.39 MB)
    Data Offset : 4096 sectors
   Super Offset : 8 sectors
   Unused Space : before=4008 sectors, after=0 sectors
          State : clean
    Device UUID : f3836318:4899a170:a0018b8b:1aa428ab

    Update Time : Sun May 14 14:40:28 2023
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 48f1cfbb - correct
         Events : 92


   Device Role : Active device 3
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sda4:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 1ba5dfd1:e861b791:eb307ef1:4ae4e4ad
           Name : AS1004T-7CBC:1  (local to host AS1004T-7CBC)
  Creation Time : Sun Jun 11 10:56:51 2017
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 5851357184 (2790.14 GiB 2995.89 GB)
     Array Size : 8777035776 (8370.43 GiB 8987.68 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=0 sectors
          State : clean
    Device UUID : 6a18260d:f0d1b882:5608a7e4:8eeabe1f

    Update Time : Sun May 14 09:31:25 2023
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 6e46beec - correct
         Events : 213501

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 3
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)

/dev/sdb:
   MBR Magic : aa55
Partition[0] :       522240 sectors at         2048 (type 83)
Partition[3] :         2047 sectors at            1 (type ee)
mdadm: No md superblock detected on /dev/sdb1.
/dev/sdb2:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 1c90030d:10445d9f:d39fc32a:06d4b79a
           Name : AS1004T-7CBC:0  (local to host AS1004T-7CBC)
  Creation Time : Sun Jun 11 10:56:28 2017
     Raid Level : raid1
   Raid Devices : 4

 Avail Dev Size : 4190208 (2046.34 MiB 2145.39 MB)
     Array Size : 2095104 (2046.34 MiB 2145.39 MB)
    Data Offset : 4096 sectors
   Super Offset : 8 sectors
   Unused Space : before=4008 sectors, after=0 sectors
          State : active
    Device UUID : 648f0d6d:967f432c:3b9e1ceb:d15959c2

    Update Time : Sun Oct 13 01:05:27 2019
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : b9c2a23f - correct
         Events : 228987


   Device Role : Active device 1
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 8c3ca866:3e6b6804:32f2955e:1b955d76
           Name : AS1004T-7CBC:126  (local to host AS1004T-7CBC)
  Creation Time : Sun May 14 09:50:45 2023
     Raid Level : raid1
   Raid Devices : 4

 Avail Dev Size : 4190208 (2046.34 MiB 2145.39 MB)
     Array Size : 2095104 (2046.34 MiB 2145.39 MB)
    Data Offset : 4096 sectors
   Super Offset : 8 sectors
   Unused Space : before=4008 sectors, after=0 sectors
          State : clean
    Device UUID : 8adc82c0:010edc11:5702a9f6:7287da86

    Update Time : Sun May 14 14:40:28 2023
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : d91b8119 - correct
         Events : 92


   Device Role : Active device 2
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb4:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 1ba5dfd1:e861b791:eb307ef1:4ae4e4ad
           Name : AS1004T-7CBC:1  (local to host AS1004T-7CBC)
  Creation Time : Sun Jun 11 10:56:51 2017
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 5851357184 (2790.14 GiB 2995.89 GB)
     Array Size : 8777035776 (8370.43 GiB 8987.68 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=0 sectors
          State : clean
    Device UUID : 15bd0bdb:b5fdcfaf:94729f61:ed9e7bea

    Update Time : Sun May 14 09:31:25 2023
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : b0f8adf8 - correct
         Events : 213501

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 1
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)

/dev/sdc:
   MBR Magic : aa55
Partition[0] :       522240 sectors at         2048 (type 83)
Partition[3] :         2047 sectors at            1 (type ee)
mdadm: No md superblock detected on /dev/sdc1.
/dev/sdc2:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 14d010c5:aaed7a5c:30956792:cfd0c452
           Name : AS1004T-7CBC:0  (local to host AS1004T-7CBC)
  Creation Time : Sun May 14 09:50:35 2023
     Raid Level : raid1
   Raid Devices : 4

 Avail Dev Size : 4190208 (2046.34 MiB 2145.39 MB)
     Array Size : 2095104 (2046.34 MiB 2145.39 MB)
    Data Offset : 4096 sectors
   Super Offset : 8 sectors
   Unused Space : before=4008 sectors, after=0 sectors
          State : clean
    Device UUID : 373358f6:76ca625d:e9193081:216676cb

    Update Time : Sun May 14 14:37:42 2023
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : ba188081 - correct
         Events : 880


   Device Role : Active device 1
   Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 8c3ca866:3e6b6804:32f2955e:1b955d76
           Name : AS1004T-7CBC:126  (local to host AS1004T-7CBC)
  Creation Time : Sun May 14 09:50:45 2023
     Raid Level : raid1
   Raid Devices : 4

 Avail Dev Size : 4190208 (2046.34 MiB 2145.39 MB)
     Array Size : 2095104 (2046.34 MiB 2145.39 MB)
    Data Offset : 4096 sectors
   Super Offset : 8 sectors
   Unused Space : before=4008 sectors, after=0 sectors
          State : clean
    Device UUID : 737541e2:f5a3673d:8db35b12:2db86324

    Update Time : Sun May 14 14:40:28 2023
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : dfa191e3 - correct
         Events : 92


   Device Role : Active device 1
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc4:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 06b57325:241ba722:6dd303af:baaa5e4e
           Name : AS1004T-7CBC:1  (local to host AS1004T-7CBC)
  Creation Time : Sun May 14 09:51:00 2023
     Raid Level : raid0
   Raid Devices : 2

 Avail Dev Size : 5851357184 (2790.14 GiB 2995.89 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=0 sectors
          State : clean
    Device UUID : d73a946c:9aa8e26e:c4388d7a:566dcf90

    Update Time : Sun May 14 09:51:00 2023
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 9bd7221c - correct
         Events : 0

     Chunk Size : 64K

   Device Role : Active device 1
   Array State : AA ('A' == active, '.' == missing, 'R' == replacing)

/dev/sdd:
   MBR Magic : aa55
Partition[0] :       522240 sectors at         2048 (type 83)
Partition[3] :         2047 sectors at            1 (type ee)
mdadm: No md superblock detected on /dev/sdd1.
/dev/sdd2:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 14d010c5:aaed7a5c:30956792:cfd0c452
           Name : AS1004T-7CBC:0  (local to host AS1004T-7CBC)
  Creation Time : Sun May 14 09:50:35 2023
     Raid Level : raid1
   Raid Devices : 4

 Avail Dev Size : 4190208 (2046.34 MiB 2145.39 MB)
     Array Size : 2095104 (2046.34 MiB 2145.39 MB)
    Data Offset : 4096 sectors
   Super Offset : 8 sectors
   Unused Space : before=4008 sectors, after=0 sectors
          State : clean
    Device UUID : acfa8c63:b226e810:3640a42a:9f8b72b1

    Update Time : Sun May 14 14:37:42 2023
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 6a42effb - correct
         Events : 880


   Device Role : Active device 0
   Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 8c3ca866:3e6b6804:32f2955e:1b955d76
           Name : AS1004T-7CBC:126  (local to host AS1004T-7CBC)
  Creation Time : Sun May 14 09:50:45 2023
     Raid Level : raid1
   Raid Devices : 4

 Avail Dev Size : 4190208 (2046.34 MiB 2145.39 MB)
     Array Size : 2095104 (2046.34 MiB 2145.39 MB)
    Data Offset : 4096 sectors
   Super Offset : 8 sectors
   Unused Space : before=4008 sectors, after=0 sectors
          State : clean
    Device UUID : 1dd56ce1:770fa0d6:13127388:46c0d14f

    Update Time : Sun May 14 14:40:28 2023
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 198ac3af - correct
         Events : 92


   Device Role : Active device 0
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd4:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 06b57325:241ba722:6dd303af:baaa5e4e
           Name : AS1004T-7CBC:1  (local to host AS1004T-7CBC)
  Creation Time : Sun May 14 09:51:00 2023
     Raid Level : raid0
   Raid Devices : 2

 Avail Dev Size : 7804860416 (3721.65 GiB 3996.09 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=0 sectors
          State : clean
    Device UUID : 1dece618:58743ad6:9f56922c:fa500120

    Update Time : Sun May 14 09:51:00 2023
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 6528b89e - correct
         Events : 0

     Chunk Size : 64K

   Device Role : Active device 0
   Array State : AA ('A' == active, '.' == missing, 'R' == replacing)

fdisk one of the disk, all disk missing sd2 and sd3, but /dev/sd*[1-4] exist and readable.

root@AS1004T-7CBC:~ # fdisk /dev/sdd
fdisk: device has more than 2^32 sectors, can't use all of them

The number of cylinders for this disk is set to 267349.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/sdd: 2199.0 GB, 2199023255040 bytes
255 heads, 63 sectors/track, 267349 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks  Id System
/dev/sdd1               1          33      261120  83 Linux
Partition 1 does not end on cylinder boundary
/dev/sdd4               1           1        1023+ ee EFI GPT
Partition 4 does not end on cylinder boundary

Partition table entries are not in disk order

Command (m for help): q

I have following questions:

  • Did reinitialize of RAID 0 array overrided my data?
  • Should I just zero-superblock the 3rd drive and reassemble the first 3 drive?
  • Since the first 2 drive looks fine, could I recover superblock of last 2 drive from first 2 drive?
  • I want to recover the RAID 5 data

========================
Update 2023-05-16:
With help of qemu-nbd, i am able to mount [dbca] and [cbda] at the same time as device md1 and md2, but

  • both device unable to find superblock of ext4, tried backup block from mke2fs -n using e2fsck -b not work
  • hexdump of 0~100MB having "1 line random data, 2 zero line,1 line random data, 2 zero line ... repeat"
  • both device able to find readable text with hexdump after 100MB.
  • have no idea which one is correct order( [dbca] or [cbda] ), will try search a file > 64KB later.

one of device after recreate using mdadm --create -e1.2 --chunk=64 /dev/md1 --assume-clean --raid-devices=4 --level=5 /dev/nbd{3,2,1,0}
the different part:

  • Feature Map : 0x1(instead of 0.0)
  • Internal Bitmap : 8 sectors from superblock(new line)
  • forgot to mention one drive is 4Tb other is 3Tb
/dev/nbd3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 1539136c:7e081bd8:e244150d:69900af6
           Name : osboxes:111  (local to host osboxes)
  Creation Time : Mon May 15 12:36:32 2023
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 7804858368 (3721.65 GiB 3996.09 GB)
     Array Size : 8777032704 (8370.43 GiB 8987.68 GB)
  Used Dev Size : 5851355136 (2790.14 GiB 2995.89 GB)
    Data Offset : 264192 sectors
   Super Offset : 8 sectors
   Unused Space : before=264112 sectors, after=1953503232 sectors
          State : clean
    Device UUID : 03f5a8c0:2c7adf91:9e3acb0f:1ef7590b

Internal Bitmap : 8 sectors from superblock
    Update Time : Mon May 15 12:36:32 2023
  Bad Block Log : 512 entries available at offset 24 sectors
       Checksum : fefc213c - correct
         Events : 0

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 0
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)

I create a new 1-drive raid0 on NAS for comparation:

  • the file system of data volumn is ext4, without lvm. (sda4->md1->ext4)
  • some info of the new disk:
root@AS1004T-7CBC:/volume1/home/admin # df -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           250M  8.0K  250M   1% /tmp
/dev/md0        2.0G  296M  1.6G  16% /volume0
/dev/loop0      951K   12K  919K   2% /share
/dev/md1        3.6T   89M  3.6T   1% /volume1
root@AS1004T-7CBC:/volume1/home/admin # mount
rootfs on / type rootfs (rw)
proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /tmp type tmpfs (rw,relatime)
/dev/md0 on /volume0 type ext4 (rw,relatime,data=ordered,jqfmt=vfsv1,usrjquota=aquota.user,grpjquota=aquota.group)
/dev/loop0 on /share type ext4 (rw,relatime)
/dev/md1 on /volume1 type ext4 (rw,relatime,data=ordered,jqfmt=vfsv1,usrjquota=aquota.user,grpjquota=aquota.group)
/dev/md1 on /share/Public type ext4 (rw,relatime,data=ordered,jqfmt=vfsv1,usrjquota=aquota.user,grpjquota=aquota.group)
/dev/md1 on /share/home type ext4 (rw,relatime,data=ordered,jqfmt=vfsv1,usrjquota=aquota.user,grpjquota=aquota.group)
/dev/md1 on /share/Web type ext4 (rw,relatime,data=ordered,jqfmt=vfsv1,usrjquota=aquota.user,grpjquota=aquota.group)
root@AS1004T-7CBC:/volume1/home/admin # mdadm --examine /dev/sda*
/dev/sda:
   MBR Magic : aa55
Partition[0] :       522240 sectors at         2048 (type 83)
Partition[3] :         2047 sectors at            1 (type ee)
mdadm: No md superblock detected on /dev/sda1.
/dev/sda2:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : f1ee1039:f9c14088:cffa3471:aa3e90ae
           Name : AS1004T-7CBC:0  (local to host AS1004T-7CBC)
  Creation Time : Mon May 15 23:05:28 2023
     Raid Level : raid1
   Raid Devices : 4

 Avail Dev Size : 4188160 (2045.00 MiB 2144.34 MB)
     Array Size : 2094080 (2045.00 MiB 2144.34 MB)
    Data Offset : 6144 sectors
   Super Offset : 8 sectors
   Unused Space : before=6064 sectors, after=0 sectors
          State : clean
    Device UUID : 9e92e90d:da2823cc:a37fd324:f8afc5b3

    Update Time : Mon May 15 23:13:59 2023
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : a6bb9ed2 - correct
         Events : 170


   Device Role : Active device 0
   Array State : A... ('A' == active, '.' == missing, 'R' == replacing)
/dev/sda3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 7fbd2329:0372140e:b608f289:fc20681e
           Name : AS1004T-7CBC:126  (local to host AS1004T-7CBC)
  Creation Time : Mon May 15 23:05:38 2023
     Raid Level : raid1
   Raid Devices : 4

 Avail Dev Size : 4188160 (2045.00 MiB 2144.34 MB)
     Array Size : 2094080 (2045.00 MiB 2144.34 MB)
    Data Offset : 6144 sectors
   Super Offset : 8 sectors
   Unused Space : before=6064 sectors, after=0 sectors
          State : clean
    Device UUID : 0f7afe79:862435fc:fb725dcf:ac326f8c

    Update Time : Mon May 15 23:06:31 2023
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : fac36572 - correct
         Events : 6


   Device Role : Active device 0
   Array State : A... ('A' == active, '.' == missing, 'R' == replacing)
/dev/sda4:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 93f70cdf:afa455d9:e6d74e16:dc0b718f
           Name : AS1004T-7CBC:1  (local to host AS1004T-7CBC)
  Creation Time : Mon May 15 23:05:43 2023
     Raid Level : raid1
   Raid Devices : 1

 Avail Dev Size : 7804858368 (3721.65 GiB 3996.09 GB)
     Array Size : 3902429184 (3721.65 GiB 3996.09 GB)
    Data Offset : 264192 sectors
   Super Offset : 8 sectors
   Unused Space : before=263912 sectors, after=0 sectors
          State : active
    Device UUID : 8d4012c7:d65e50fc:77afded9:e64fcfe1

    Update Time : Mon May 15 23:06:00 2023
  Bad Block Log : 512 entries available at offset 264 sectors
       Checksum : 921f67a4 - correct
         Events : 3


   Device Role : Active device 0
   Array State : A ('A' == active, '.' == missing, 'R' == replacing)

Update 2023-05-17:(skip to 05-18) try to find the ext4 superbokck from array

root@osboxes:/home/osboxes# hexdump   /dev/md1 |grep 'ffff ef53'
7d00030 9bf5 6460 006b ffff ef53 0000 0001 0000
bd00030 af1c 6460 0000 ffff ef53 0000 0001 0000
11c7b550 ef20 0012 ffff ffff ef53 0012 0000 0000
1347b550 ef20 0012 ffff ffff ef53 0012 0000 0000
^C
root@osboxes:/home/osboxes# hexdump   /dev/md2 |grep 'ffff ef53'
7[ce]0030 9bf5 6460 006b ffff ef53 0000 0001 0000
bd[2]0030 af1c 6460 0000 ffff ef53 0000 0001 0000
11c7b550 ef20 0012 ffff ffff ef53 0012 0000 0000
1347b550 ef20 0012 ffff ffff ef53 0012 0000 0000
^C

is this look like ext4 usperblock?

root@osboxes:/home/osboxes# hexdump -s 0xbd00000 -n 0x100  /dev/md1
bd00000 f000 0cb7 2b00 65bf fffb 000b f6f2 64f0
bd00010 eff5 0cb7 0000 0000 0002 0000 0002 0000
bd00020 8000 0000 8000 0000 1000 0000 0000 0000
bd00030 af1c 6460 0000 ffff ef53 0000 0001 0000
bd00040 af05 6460 0000 0000 0000 0000 0001 0000
bd00050 0000 0000 000b 0000 0100 0001 003c 0000
bd00060 02c2 0000 007b 0000 b5d9 42ac 7544 a848
bd00070 df96 1606 2700 6edf 0000 0000 0000 0000
bd00080 0000 0000 0000 0000 0000 0000 0000 0000
*
bd000c0 0000 0000 0000 0000 0000 0000 0000 0400
bd000d0 0000 0000 0000 0000 0000 0000 0000 0000
bd000e0 0008 0000 0000 0000 0000 0000 215a a8e3
bd000f0 17b0 154b e18b 37e5 99bd e3cf 0101 0040
bd00100
root@osboxes:/home/osboxes# hexdump -s 0xbd20000 -n 0x100  /dev/md2
bd20000 f000 0cb7 2b00 65bf fffb 000b f6f2 64f0
bd20010 ... data is same except address

0xbd00000/4096=0xbd00 or 48384

root@osboxes:/home/osboxes# e2fsck -b 48384 /dev/md1
e2fsck 1.46.2 (28-Feb-2021)
Superblock has an invalid journal (inode 8).
Clear<y>?

dumpe2fs -o superblock=48384 /dev/md1 dumpe2fs -o superblock=$((48384+131072/4096)) /dev/md2 showing the same result, Block count: 1707027200 tell it is 6.98 terabytes.

root@osboxes:/home/osboxes# dumpe2fs -o superblock=48384 /dev/md1
dumpe2fs 1.46.2 (28-Feb-2021)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          d9b5ac42-4475-48a8-96df-06160027df6e
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype extent 64bit flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         unsigned_directory_hash
Default mount options:    user_xattr acl
Filesystem state:         not clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              213381120
Block count:              1707027200
Reserved block count:     786427
Free blocks:              1693513458
Free inodes:              213381109
First block:              0
Block size:               4096
Fragment size:            4096
Group descriptor size:    64
Reserved GDT blocks:      1024
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         4096
Inode blocks per group:   256
RAID stride:              16
RAID stripe width:        32
Flex block group size:    16
Filesystem created:       Sun May 14 05:51:01 2023
Last mount time:          n/a
Last write time:          Sun May 14 05:51:24 2023
Mount count:              0
Maximum mount count:      -1
Last checked:             Sun May 14 05:51:01 2023
Check interval:           0 (<none>)
Lifetime writes:          145 MB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      5a21e3a8-b017-4b15-8be1-e537bd99cfe3
Journal backup:           inode blocks
Journal superblock magic number invalid!

the superblock position is wired (compare to mke2fs -n 48384-32768=0x8000), is the new array have wrong offset of ext4 file system?

root@osboxes:/home/osboxes# e2fsck -p -b $((48384)) /dev/md1
/dev/md1: Superblock has an invalid journal (inode 8).
CLEARED.
*** journal has been deleted ***

/dev/md1: Block bitmap for group 960 is not in group.  (block 8305676774422481356)


/dev/md1: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
        (i.e., without -a or -p options)
root@osboxes:/home/osboxes# e2fsck -p -b $((48384+131072/4096)) /dev/md2
/dev/md2: Superblock has an invalid journal (inode 8).
CLEARED.
*** journal has been deleted ***

/dev/md2: Block bitmap for group 960 is not in group.  (block 1961591391969192664)


/dev/md2: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
        (i.e., without -a or -p options)

Update 2023-05-18 :
last night I was recover using the wrong superblock, there are two set of superblock on array one created 2017 and one created last week.. I filtered only the 2017 one, found they have same relative position compare to normal ext4

superblock on md1 +offset= normal superblock
not found +0x300000 0000430
7d00030 +0x300000 8000030
17d00030 +0x300000 18000030
27d00030 +0x300000 28000030
37d00030 +0x300000 38000030
47d00030 +0x300000 48000030
c7d00030 +0x300000 c8000030
d7d00030 +0x300000 d8000030
187d00030 +0x300000 88000030
287d00030 +0x300000 e8000030
3e7d00030 +0x300000 98000030

... Look like first 0x2ffbd0+0x430=0x300000 byte= 3MB truncated.

Update 2023-05-19:
after prepend 3MB to the md1 block, then with a simple e2fsck -p, im able to mount the file system.

enter image description here

I have wrote a summary in answer.

200
2 + 3
br flag
Chopper3
Please don't use RAID5 - it's dangerous.
1
Reply
Nikita Kipriyanov avatar
za flag
Nikita Kipriyanov
Wrong offset = wrong order of blocks, i.e. wrong order of disks.
0
Reply
Charles Chou avatar
us flag
Charles Chou
i bet the 1/2 chances. md1 and md2 should have one correct order. now I worry about the original ext4 partition is not starting at 1024Kb. since superblock position of both array is not normal(48384 for md1 and 48416 for md2).
0
Reply
Score:3
Nikita Kipriyanov avatar Server
za flag
Nikita Kipriyanov

If you don't have spare space resources to store data at least two times larger the raw size of your array, e.g. if you don't have at least 3 TB * 4 drives * 2 = 24 TB of free space to dedicate to the recovery operation, stop and lend this whole job to a professional data recovery service.

Now, answers.

  1. If it ran mdadm --create without --assume-clean, yes, data was overwritten with zeros.

  2. No. You must not change anything on drives. Your first required step is making dumps (images) of all four members of the RAID.

  3. No. These superblocks are different. Something in them should be the same, while something differs. In particular, each superblock records the role (an ordered position) of this device in the array.

As outlined in (1), most likely the data that was located in the beginning of the array is irreversibly destroyed (as if two drives from RAID5 were lost simultaneously). It might be possible to recover the "tail" which lies where you stopped the process. This doesn't necessarily mean you are able to recover the user data which is there, because the file system structures which live there presumably depend on the blocks that lie in the destroyed area too. But decent filesystem will have many replicas of the superblock which may happen to be in the non-damaged area so there is still a hope. You may try to reveal this non-damaged tail and recover what possible from it.

  1. Begin with taking required backup of all four devices as outlined in (2) (use e.g. dd or ddrescue). This will use half of your spare space.

  2. Then, you may proceed with re-creating the array with mdadm --create -e1.2 -l5 -n4 --assume-clean /dev/sd[abcd]4. Pay attention to the order of the drives, as I presented them in above command most likely in an incorrect order. You'll have to play a bit to guess the correct order; probably the correct order should be [cbda] or [dbca], because surviving devices have orders: sda4=3, sdb4=1 (taken from Device Role property). If you guessed it wrong, you'll have to copy the dumps back to drives and start over; this is what dumps were for, but see the tip below. Ideally this will require no more than 2 guesses. At most there are 4! = 1 * 2 * 3 * 4 = 24 different orderings of four drives.

What you should expect is that the data in the end of the array happens to be clean. How to check that, depends on what you had there. Your array uses chunk size 64KiB, so you have to check that 64KiB stretches of data on the array turned out standing in a correct order. Again, see the tip to ease the guessing process.

  1. When the correct order is found, you dump the image of the assembled array to the remaining spare free space. Now you'll be carrying out the file system recovery operation on the array. If that's ext4, you might try to run e2fsck -b <superblock>, specifying the superblock which is in the non-damaged area; which one, you might guess by running mke2fs -n which simulate the creation of the file system without actually writing anything. Basically, what you'll get after this step is what was possible to recover.

Tip. After taking required full dumps you may speed up guessing process by instantiating the read-write overlays so not to change the data on drives. You'll be only in the need to recreate overlay images instead of copying dumps back to drives in case of wrong guess, which is much faster than copying 12 TB. I described this in the other answer, but for your problem you make overlay not for the assembled array but for four individual devices and then build the array from layered nbdX devices.

This also will let you skip dumping the file system image. You can make all 2 or even all 24 possible orders in these dumps simultaneously (which will require 8 or 96 overlay images and NBDs, respectively, but all those images will contain only changes and tend to not grow very much during recovery operations like this). Then try to recover filesystem on each one and see which one is correct. Then you remove all incorrect attempts, copy the contents of the file system onto the spare free space, remove the array on devices and re-create them anew and then copy survived data back.

+ 6
Charles Chou avatar
us flag
Charles Chou
Thank you for your detailed response! I just did an experiment to prove if mdadm zero new array, i was able to recover the data if I just recreate the array(without create lvm and filesystem inside), I hope the install program just wipe the system partition, I will try to restore the raid after I have backup my first 3 drive. overlay the block storage is a good idea but I have problem to mount 4 drive on my PC, I will try to do this on my NAS first.
0
Reply
Nikita Kipriyanov avatar
za flag
Nikita Kipriyanov
You can dump them sequentially onto 1 or 2 large enough drives, and then carry out all operations out of those images, leaving original drives alone. This way you will never need to connect 4 drives simultaneously to the PC.
0
Reply
Charles Chou avatar
us flag
Charles Chou
Another thing you point out is the ordering, do you mean the `Device Role` field? so the sda4 and sdb4 is 4th and 2nd drive? so I only have 2 guesses left? `?B?A`
0
Reply
Charles Chou avatar
us flag
Charles Chou
I will borrow some Hard Drive Dock to mount all drive on PC since I dont have 9Tb spare space, that will take some time.
0
Reply
Nikita Kipriyanov avatar
za flag
Nikita Kipriyanov
Yes, in this case Device Role only specifies order of drives in the array. I outlined that (see my point 2). Likely you only need to check `[dbca]` and `[cbda]`, one of them should be correct (sorry, I misread that in your question, now I should fix it in the answer). Also note, that you need to back up **all four** drives, even 9 TB is clearly not sufficient for this.
0
Reply
Charles Chou avatar
us flag
Charles Chou
I mounted 2 array [dbca] and [cbda] same time as md1&2, I am able to found the superblock(at 48384 and 48416),recover superblock using `e2fsck -b` , supplied dumpe2fs result in question, is it possible just `e2fsck -y` to fix? How to ignore lost file?
0
Reply
Score:1
Charles Chou avatar Server
us flag
Charles Chou

TL;DR
Only work if you overwritten different RAID config / filesystem (because the backup superblock location is different, old superblock was not overwritten), and your old file system was EXT4/3. and no physical damage. the key to recover is the backup superblock exist.

If only the first part of EXT4 corrupt, should be able use this method recover its file strcuture

Situation: 2 of 4 RAID 5 drive was overwritten their header(by reinit new raid and create new ext4 inside), the other 2 remain untouched.

  1. Backup first, or mount these 4 drive on PC as read-only. the old raid is /dev/sd{b,c,d,e}4 for me.
blockdev --setro /dev/sd{b,c,d,e}4 # check the device using lsblk
  1. setting up overlay block device for them, (mention by @Nikita Kipriyanov, it really make the operations more efficient)
modprobe nbd
apt update && apt install qemu-utils -y
# create tmp file to hold changes
qemu-img create -f qcow2 -b /dev/sdb4 -F raw /tmp/sdb4.qcow2
qemu-img create -f qcow2 -b /dev/sdc4 -F raw /tmp/sdc4.qcow2
qemu-img create -f qcow2 -b /dev/sdd4 -F raw /tmp/sdd4.qcow2
qemu-img create -f qcow2 -b /dev/sde4 -F raw /tmp/sde4.qcow2
# mount nbd[1-4] as an overlay device of `/dev/sd[bcde]4`
qemu-nbd -c /dev/nbd0 /tmp/sdb4.qcow2 
qemu-nbd -c /dev/nbd1 /tmp/sdc4.qcow2 
qemu-nbd -c /dev/nbd2 /tmp/sdd4.qcow2 
qemu-nbd -c /dev/nbd3 /tmp/sde4.qcow2

you can start over by recreating nbd device

mdadm --stop /dev/md1
qemu-nbd -d /dev/nbd0
qemu-nbd -d /dev/nbd1
qemu-nbd -d /dev/nbd2
qemu-nbd -d /dev/nbd3
# run step 2 again
  1. create new mdadm array Note you have to find the correct order (change 3,2,1,0),I was able to find by mdadm --examine /dev/sdx4 by Device Role field(the 2 drive that not affected).
    (thanks @Nikita Kipriyanov point out)
# Note you must use --assume-clean to avoid mdadm to destroy the data inside old array
# mdadm --create -e1.2  --chunk=64 /dev/md1 --assume-clean --raid-devices=4 --level=5  /dev/nbd{3,2,1,0}
mdadm: /dev/nbd3 appears to be part of a raid array:
       level=raid0 devices=2 ctime=Sun May 14 05:51:00 2023
mdadm: /dev/nbd2 appears to be part of a raid array:
       level=raid5 devices=4 ctime=Sun Jun 11 06:56:51 2017
mdadm: /dev/nbd1 appears to be part of a raid array:
       level=raid0 devices=2 ctime=Sun May 14 05:51:00 2023
mdadm: /dev/nbd0 appears to be part of a raid array:
       level=raid5 devices=4 ctime=Sun Jun 11 06:56:51 2017
mdadm: largest drive (/dev/nbd3) exceeds size (2925677568K) by more than 1%
Continue creating array? y

the 0 and 2nd drive was accidentally reinitializing, so I force recreate a new array using old config(--chunk=64) bd20030 4. New I have /dev/md1, assume it has no lvm, no partition table, only a EXT4 fs inside. (know by creating a new NAS OS on new drive)
Searching for ext4 superblock backup.

# hexdump /dev/md1 | awk '$6 == "ef53"'
7ce0030 9bf5 6460 006b ffff ef53 0000 0001 0000
bd20030 af1c 6460 0000 ffff ef53 0000 0001 0000
11c7b550 ef20 0012 ffff ffff ef53 0012 0000 0000
1347b550 ef20 0012 ffff ffff ef53 0012 0000 0000
17d20030 9bf5 6460 006b ffff ef53 0000 0001 0000
23d20030 af1c 6460 0000 ffff ef53 0000 0001 0000
37ce0030 9bf5 6460 006b ffff ef53 0000 0001 0000
3bd20030 af1c 6460 0000 ffff ef53 0000 0001 0000
47d20030 9bf5 6460 006b ffff ef53 0000 0001 0000
53d20030 af1c 6460 0000 ffff ef53 0000 0001 0000
6bd20030 af1c 6460 0000 ffff ef53 0000 0001 0000
95395060 ef52 e732 fffd ffff ef53 e731 5f5f 5757

note there is both ext4 fs superblock at different offset, we only need the old one, verify by dumpe2fs

# dumpe2fs -o superblock=$(((0xbd20030-0x30)/4096)) /dev/md1
dumpe2fs 1.46.2 (28-Feb-2021)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          d9b5ac42-4475-48a8-96df-06160027df6e
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype extent 64bit flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         unsigned_directory_hash
Default mount options:    user_xattr acl
Filesystem state:         not clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              213381120
Block count:              1707027200
Reserved block count:     786427
Free blocks:              1693513458
Free inodes:              213381109
First block:              0
Block size:               4096
Fragment size:            4096
Group descriptor size:    64
Reserved GDT blocks:      1024
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         4096
Inode blocks per group:   256
RAID stride:              16
RAID stripe width:        32
Flex block group size:    16
Filesystem created:       Sun May 14 05:51:01 2023
Last mount time:          n/a
Last write time:          Sun May 14 05:51:24 2023
Mount count:              0
Maximum mount count:      -1
Last checked:             Sun May 14 05:51:01 2023
Check interval:           0 (<none>)
Lifetime writes:          145 MB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      5a21e3a8-b017-4b15-8be1-e537bd99cfe3
Journal backup:           inode blocks
Journal superblock magic number invalid!

note the Filesystem created field, If it is the time you accidentally formatted, then this is not superblock you looking for. found out 2 byte before magicbyte the old one is 006b the new one is 0000.

after filter correct signature, the result more make sense.

# hexdump /dev/md1 | awk '$6 == "ef53" && $4 == "006b"'
7d00030 9bf5 6460 006b ffff ef53 0000 0001 0000
17d00030 9bf5 6460 006b ffff ef53 0000 0001 0000
27d00030 9bf5 6460 006b ffff ef53 0000 0001 0000
37d00030 9bf5 6460 006b ffff ef53 0000 0001 0000
47d00030 9bf5 6460 006b ffff ef53 0000 0001 0000
c7d00030 9bf5 6460 006b ffff ef53 0000 0001 0000
d7d00030 9bf5 6460 006b ffff ef53 0000 0001 0000
187d00030 9bf5 6460 006b ffff ef53 0000 0001 0000
287d00030 9bf5 6460 006b ffff ef53 0000 0001 0000
3e7d00030 9bf5 6460 006b ffff ef53 0000 0001 0000

If I compare to normal EXT4 layout, they have same relative position compare to normal ext4.

Superblock on md1 +Offset= Normal EXTT4 superblock
not found +0x300000 0000430
7d00030 +0x300000 8000030
17d00030 +0x300000 18000030
27d00030 +0x300000 28000030
37d00030 +0x300000 38000030
47d00030 +0x300000 48000030
c7d00030 +0x300000 c8000030
d7d00030 +0x300000 d8000030
187d00030 +0x300000 88000030
287d00030 +0x300000 e8000030
3e7d00030 +0x300000 98000030

Look like first 0x2ffbd0+0x430 = 0x300000 byte = 3MB was truncated. Not sure why 3MB was truncated, I guess 3 Mb is the header of each mdadm member [3 (data member) * 1M (mdadm header)], so 3MB was taken by new created array, may be the partition table on disk has problem too.

the 3MB offset may be cause by wrong data offset? https://raid.wiki.kernel.org/index.php/A_guide_to_mdadm

These superblocks also define a "data offset". This is the gap between the start of the device, and the start of the data. This means that v1.2 must always have at least 4K per device, although it's normally several megabytes. This space can be used for all sorts of things, typically the write-intent bitmap, the bad blocks log, and a buffer space when reshaping an array. It is usually calculated automatically, but can be over-ridden.
  1. prepend 3 MB before the block then I have the correct position of the original file system.
# dd if=/dev/zero of=/root/header bs=1M count=$((3+2)) # 1M mdadm header for each mdadm member block
# losetup /dev/loop13 /root/header
# mdadm --create /dev/md111 --level=linear  --assume-clean --raid-devices=2 /dev/loop13 /dev/md1
# hexdump /dev/md1 | awk '$6 == "ef53" && $5 == "ffff" && $4 == "006b"'
8000030 9bf5 6460 006b ffff ef53 0000 0001 0000
18000030 9bf5 6460 006b ffff ef53 0000 0001 0000
28000030 9bf5 6460 006b ffff ef53 0000 0001 0000
38000030 9bf5 6460 006b ffff ef53 0000 0001 0000
48000030 9bf5 6460 006b ffff ef53 0000 0001 0000
c8000030 9bf5 6460 006b ffff ef53 0000 0001 0000
d8000030 9bf5 6460 006b ffff ef53 0000 0001 0000
88000030 9bf5 6460 006b ffff ef53 0000 0001 0000
e8000030 9bf5 6460 006b ffff ef53 0000 0001 0000

only the first superblock is missing, It is easy to fix now.

# e2fsck -p -b $(((0x8000030-0x30)/4096)) /dev/md111
/dev/md111: Superblock needs_recovery flag is clear, but journal has data.
/dev/md111: Recovery flag not set in backup superblock, so running journal anyway.
/dev/md111: recovering journal
/dev/md111: Resize inode not valid.

/dev/md111: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
        (i.e., without -a or -p options)
# mount -o ro /dev/md111 /mnt
  1. mount without error message, and filesystem structure still here. I open some file and play a video did not found any problem(yet). enter image description here

refs:
https://raid.wiki.kernel.org/index.php/Advanced_data_recovery https://raid.wiki.kernel.org/index.php/A_guide_to_mdadm https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout#Layout

+ 4
Nikita Kipriyanov avatar
za flag
Nikita Kipriyanov
could you explain how your filesystem appear shifted 3 mb? Something important on what structure it actually was is ommited?
0
Reply
Charles Chou avatar
us flag
Charles Chou
I dont know, I guess 3 Mb is the header of each mdadm member [3 (data member) * 1M (mdadm header)], so 3MB was taken by new created array, may be the partition table on disk has problem too (fdisk -l only show sdx1 and sdx4, I will update in question). I know some file may corrupt(bytes overwritten by new superblock and journals), I open some file and play a video did not found any problem(yet).
0
Reply
Charles Chou avatar
us flag
Charles Chou
or bay be the old array's superblock is not at the beginning of device, see `Data Offset : 262144 sectors`, its much larger then default. so the new create array may overwritten the data, the `--data-offset` argument of mdadm might affect superblock location
0
Reply
Charles Chou avatar
us flag
Charles Chou
my be the default `data offset` is not same compare to original array, I will try recreate the array have smaller `data offset` to see if the position of superblock change later.
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.