I ran into an issue with a degraded mdadm RAID1 array on a server I inherited:
The server has two identical M.2 SSD drives with 256GB capacity each in a RAID1 config via mdadm for the operating system (Debian 11). The partitions on the drive are:
/boot -> /dev/md0 (512mb)
swap -> /dev/md1 (2gb)
/ -> /dev/md2 (232gb)
The server was last rebooted in September 2021 and has been running fine since. After inheriting it from the previous admin, I found that the daily LOGWATCH report has been reporting mdadm: cannot open /dev/md/0: No such file or directory in the mdadm-section since February 2022, so I checked all three devices on the server with mdadm --detail /dev/mdX:
/dev/md0:
Version : 1.2
Creation Time : Mon Jan 25 08:51:55 2021
Raid Level : raid1
Array Size : 523264 (511.00 MiB 535.82 MB)
Used Dev Size : 523264 (511.00 MiB 535.82 MB)
Raid Devices : 2
Total Devices : 1
Persistence : Superblock is persistent
Update Time : Sun Jan 1 00:57:01 2023
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0
Consistency Policy : resync
Number Major Minor RaidDevice State
- 0 0 0 removed
1 259 8 1 active sync
/dev/md1:
Version : 1.2
Creation Time : Mon Jan 25 08:51:56 2021
Raid Level : raid1
Array Size : 2094080 (2045.00 MiB 2144.34 MB)
Used Dev Size : 2094080 (2045.00 MiB 2144.34 MB)
Raid Devices : 2
Total Devices : 1
Persistence : Superblock is persistent
Update Time : Mon Jan 2 07:20:43 2023
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0
Consistency Policy : resync
Name : storage01:1 (local to host storage01)
UUID : 9b0721ae:6b5c0c39:02a8f4b1:4c56e931
Events : 3998
Number Major Minor RaidDevice State
- 0 0 0 removed
1 259 13 1 active sync /dev/nvme3n1p2
/dev/md2:
Version : 1.2
Creation Time : Mon Jan 25 08:51:56 2021
Raid Level : raid1
Array Size : 247302464 (235.85 GiB 253.24 GB)
Used Dev Size : 247302464 (235.85 GiB 253.24 GB)
Raid Devices : 2
Total Devices : 1
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Mon Jan 2 09:02:16 2023
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0
Consistency Policy : bitmap
Name : storage01:2 (local to host storage01)
UUID : dbc01a28:9f934ddd:3dbdb4bd:b84256d2
Events : 22088209
Number Major Minor RaidDevice State
0 259 14 0 active sync /dev/nvme3n1p3
- 0 0 1 removed
All three show status as degraded with one drive from the mirror marked as "removed". For the swap (md1) and root (md2) partitions, device nvme3n1 is shown as the remaining active drive while for the boot partition (md0) no device is shown and there is also no Name and UUID for md0.
Running lsblk shows that device nvme3n1 is alive and present and contains md1 and md2 and a 512mb partition that should be md0 but is not marked as such in lsblk:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
... HDDs omitted
nvme3n1 259:7 0 238.5G 0 disk
├─nvme3n1p1 259:12 0 512M 0 part
├─nvme3n1p2 259:13 0 2G 0 part
│ └─md1 9:1 0 2G 0 raid1 [SWAP]
├─nvme3n1p3 259:14 0 236G 0 part
│ └─md2 9:2 0 235.9G 0 raid1 /
└─nvme3n1p4 259:15 0 1M 0 part
lsblk also does not show device nvme2n1 which should be the mirror to nvme3n1.
Digging through /var/mail/root I found messages from February 17, 2022 which report Fail events for md2:
- cut -
A Fail event had been detected on md device /dev/md/2.
It could be related to component device /dev/nvme2n1p3.
Faithfully yours, etc.
P.S. The /proc/mdstat file currently contains the following:
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md2 : active raid1 nvme2n1p3[1](F) nvme3n1p3[0]
247302464 blocks super 1.2 [2/1] [U_]
bitmap: 2/2 pages [8KB], 65536KB chunk
md1 : active (auto-read-only) raid1 nvme3n1p2[1] nvme2n1p2[0]
2094080 blocks super 1.2 [2/2] [UU]
md0 : active raid1 nvme2n1p1[1]
523264 blocks super 1.2 [2/1] [_U]
and md1:
- cut -
A Fail event had been detected on md device /dev/md/1.
It could be related to component device /dev/nvme2n1p2.
Faithfully yours, etc.
P.S. The /proc/mdstat file currently contains the following:
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md2 : active raid1 nvme3n1p3[0]
247302464 blocks super 1.2 [2/1] [U_]
bitmap: 2/2 pages [8KB], 65536KB chunk
md1 : active (auto-read-only) raid1 nvme3n1p2[1] nvme2n1p2[0](F)
2094080 blocks super 1.2 [2/1] [_U]
md0 : active raid1 nvme2n1p1[1]
523264 blocks super 1.2 [2/1] [_U]
As the mails show, md0 was already marked as degraded at this point in time. I traced the failure of md0 back to March 21, 2021 in the root mails:
Date: Sun, 21 Mar 2021 04:56:59 +0100 (CET)
This is an automatically generated mail message from mdadm
running on storage01
A Fail event had been detected on md device /dev/md/0.
It could be related to component device /dev/nvme2n1p1.
Faithfully yours, etc.
P.S. The /proc/mdstat file currently contains the following:
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md2 : active raid1 nvme3n1p3[1]
247302464 blocks super 1.2 [2/1] [_U]
bitmap: 1/2 pages [4KB], 65536KB chunk
md1 : active (auto-read-only) raid1 nvme2n1p2[1](F) nvme3n1p2[0]
2094080 blocks super 1.2 [2/1] [U_]
resync=PENDING
md0 : active raid1 nvme2n1p1[0](F) nvme3n1p1[1]
523264 blocks super 1.2 [2/1] [_U]
Directly preceeding this mail on March 21, 2021 were again Fail events for md1 and md2 (not included for some brevity), all three mails show nvme2n1 as the failed device and nvme3n1 as the active one.
Going further chronologically through the root mails, I found that md1 automatically recovered on March 30, 2021 and md2 followed one day later, but md0 is still showing as degraded and, strangely, only present on nvme2n1, the previously failed device:
Date: Wed, 31 Mar 2021 06:25:01 +0200 (CEST)
This is an automatically generated mail message from mdadm
running on storage01
A DegradedArray event had been detected on md device /dev/md/0.
Faithfully yours, etc.
P.S. The /proc/mdstat file currently contains the following:
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md2 : active raid1 nvme3n1p3[0] nvme2n1p3[1]
247302464 blocks super 1.2 [2/2] [UU]
bitmap: 1/2 pages [4KB], 65536KB chunk
md0 : active raid1 nvme2n1p1[1] <---------------- !!!
523264 blocks super 1.2 [2/1] [_U]
md1 : active (auto-read-only) raid1 nvme3n1p2[1] nvme2n1p2[0]
2094080 blocks super 1.2 [2/2] [UU]
resync=PENDING
The root mails from that point on report a degraded array only for md0 every day (with md0 on nvme2n1) until a server reboot in August 2021, after which md0 is still degraded but now shown as located on nvme3n1:
Date: Fri, 6 Aug 2021 09:49:47 +0200 (CEST)
This is an automatically generated mail message from mdadm
running on storage01
A DegradedArray event had been detected on md device /dev/md/0.
Faithfully yours, etc.
P.S. The /proc/mdstat file currently contains the following:
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md0 : active (auto-read-only) raid1 nvme3n1p1[1] <---------------- suddenly on nvme3
523264 blocks super 1.2 [2/1] [_U]
md2 : active raid1 nvme2n1p3[0] nvme3n1p3[1]
247302464 blocks super 1.2 [2/2] [UU]
bitmap: 1/2 pages [4KB], 65536KB chunk
md1 : active (auto-read-only) raid1 nvme3n1p2[0] nvme2n1p2[1]
2094080 blocks super 1.2 [2/2] [UU]
The reporting continues like this every day until another server reboot in September 2021 after which md0 switches back to be on nvme2n1:
Date: Tue, 21 Sep 2021 08:42:03 +0200 (CEST)
This is an automatically generated mail message from mdadm
running on storage01
A DegradedArray event had been detected on md device /dev/md/0.
Faithfully yours, etc.
P.S. The /proc/mdstat file currently contains the following:
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md2 : active raid1 nvme2n1p3[1] nvme3n1p3[0]
247302464 blocks super 1.2 [2/2] [UU]
bitmap: 1/2 pages [4KB], 65536KB chunk
md1 : active (auto-read-only) raid1 nvme3n1p2[1] nvme2n1p2[0]
2094080 blocks super 1.2 [2/2] [UU]
md0 : active (auto-read-only) raid1 nvme2n1p1[1] <---------------- suddenly nvme2 again
523264 blocks super 1.2 [2/1] [_U]
The report stays this way up until the second Fail event for md1 and md2 on February 17, 2022 that is shown at the very beginning. Since then, root mail reports degraded arrays for all three: md0, md1 and md2 daily.
I'm not sure what happened here, since I wasn't involved in anything that happened with the server up until a few months ago. I can only reconstruct from the logs and root mails.
My problem or question is now: How do I repair the array? Can I even reboot the server in this condition? (it is in production, so I can't just try it out)
I would normally try to replace the failed nvme2n1 device and then rebuild it from the active nvme3n1 but since the boot partition md0 is not present (or at least not recognized) on nvme3n1, can I even do that?
I also find it very strange that lsblk shows nvme2 as absent but /proc/mdstat insists that md0 is located there, all while LOGWATCH (and ls /dev/md/ which only shows 1 and 2) both say that md0 isn't there at all and mdadm --detail /dev/md0 is strangely undecided on where md0 is, it's just sure that it's in active sync...
I also logged into the IPMI interface of the server and checked the HDD inventory: The two drives are reported to be physically present at slots "M2_1" and "PCIE_1_B"... which is also strange, since both are M.2 drives and should be present at slots "M2_1" and M2_2" instead.
I'm a little at a loss here on what could have caused this and whether or not I can repair the array without reinstalling the OS and everything?
Any help is appreciated!
