Score:0

Server

Recover a failed Physical Volume (PV) in LVM

Martins Pukitis

3/23/23, 2:15 PM

My motherboard burned. After getting a new motherboard and attaching the disks, recovering a failed PV fails, because the hard disk size has decreased.

# pvcreate --test --uuid "wcUYiJ-ULvA-YcLK-xlLM-qTof-uCD4-u2p0FH" --restorefile /etc/lvm/backup/home-vg /dev/sda
  TEST MODE: Metadata will NOT be updated and volumes will not be (de)activated.
  WARNING: Couldn't find device with uuid wcUYiJ-ULvA-YcLK-xlLM-qTof-uCD4-u2p0FH.
  Device /dev/sda excluded by a filter.

Now the PV doesn't fit the disk any more. I know that the disk has shrunk not only from PVM backup file, but also I have an output from fdisk before the disaster. Earlier fdisk reported disk size was 9.1 TiB, now it shows:

# fdisk -l /dev/sda
Disk /dev/sda: 8,19 TiB, 9000659811328 bytes, 17579413694 sectors
Disk model: WDC WD100EFAX-68
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device     Boot Start        End    Sectors Size Id Type
/dev/sda1           1 4294967295 4294967295   2T ee GPT

Partition 1 does not start on physical sector boundary.

The partiton table of the disk is also corrupt. The correct partiton table should look like this:

Disk /dev/sdd: 9,1 TiB, 10000831348736 bytes, 19532873728 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: E965D649-FADC-4450-BB9A-EA0B7E6191DE

Device         Start         End     Sectors  Size Type
/dev/sdd1       2048   671090687   671088640  320G Linux filesystem
/dev/sdd2  671090688 19532873694 18861783007  8,8T Linux filesystem

# gdisk -l /dev/sda
GPT fdisk (gdisk) version 1.0.5

Warning! Disk size is smaller than the main header indicates! Loading
secondary header from the last sector of the disk! You should use 'v' to
verify disk integrity, and perhaps options on the experts' menu to repair
the disk.
Caution: invalid backup GPT header, but valid main header; regenerating
backup header from main header.

Warning! One or more CRCs don't match. You should repair the disk!
Main header: OK
Backup header: ERROR
Main partition table: OK
Backup partition table: ERROR

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: damaged

****************************************************************************
Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk
verification and recovery are STRONGLY recommended.
****************************************************************************
Disk /dev/sda: 17579413694 sectors, 8.2 TiB
Model: WDC WD100EFAX-68
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): E965D649-FADC-4450-BB9A-EA0B7E6191DE
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 2048, last usable sector is 19532873694
Partitions will be aligned on 2048-sector boundaries
Total free space is 0 sectors (0 bytes)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048       671090687   320.0 GiB   8300  
   2       671090688     19532873694   8.8 TiB     8300

How to proceed? I would like to preserve as much data as possible, of course.

0 + 0

lvm

Nikita Kipriyanov

3/23/23, 3:03 PM

I don't understand how replaced mobo is related to decreased hdd. Why it was decreased, how? Did you replaced only the mobo? // My first thought is to build a block device somehow which could accept a *complete* physical volume. And after that to take LV out from it properly.

Martins Pukitis

3/23/23, 3:11 PM

I don't know why the size of hdd decreased. Just the motherboard was replaced.

Nikita Kipriyanov

3/23/23, 3:24 PM

This is *very* strange. In any case, to not to lose data you have to figure that out first. How disk system was implemented before? Was there any fakeRAID or something like this? Does new MB support drives of that capacity?

Martins Pukitis

3/23/23, 3:43 PM

There was no fakeRAID, it's just a single hard disk. The new motherboard is the same model as the old one.

Martins Pukitis

3/23/23, 6:25 PM

Updated the question - added how the partition table of the driver should look like.

Nikita Kipriyanov

3/23/23, 6:26 PM

The "incorrect" partition table you showed is a MBR protective, it doesn't show anything resembling "real" partitioning and only serves to indicate GPT presence. What `gdisk` shows? (I mean, recent fdisk should correctly recognize it too, but somehow it didn't)

Martins Pukitis

3/23/23, 6:28 PM

Updated with gdisk.

Nikita Kipriyanov

3/23/23, 6:31 PM

Now you see the partitions are there. This is good. But it could mean GPT was corrupt. gdisk should be able to fix it, and then kernel will recognize and PV should work.

Nikita Kipriyanov

3/23/23, 6:32 PM

I suggest to proceed like it is described here: https://askubuntu.com/questions/386752/fixing-corrupt-backup-gpt-table then reload partition table and retry LVM.

Score:0

Server

FooBar

3/23/23, 8:00 PM

That is very odd, the WD 10tb hard drive should not detect as 8.2TiB

I suspect your fried Mobo may have sent a power spike and toasted your drive firmware!

Perhaps try the WD drive diagnostics -

https://support.wdc.com/downloads.aspx?p=2

being very careful to only do the non invasive ones if you have data on the drive!

0 + 0

Score:0

Server

Martins Pukitis

3/24/23, 8:02 AM

I did this:

# hdparm -N /dev/sda
/dev/sda:
 max sectors   = 17579413694/19532873728, HPA is enabled
# hdparm -N p19532873728 /dev/sda
/dev/sda:
 setting max visible sectors to 19532873728 (permanent)
 max sectors   = 19532873728/19532873728, HPA is disabled

After the reboot, however, the disk wasn't detected any more. Then I replaced the SATA cable and now the disk is working fine. All data available.

0 + 0

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Recover a failed Physical Volume (PV) in LVM

TH: กู้คืนฟิสิคัลวอลุ่ม (PV) ที่ล้มเหลวใน LVM

RO: Recuperați un volum fizic (PV) eșuat în LVM

RU: Восстановление неисправного физического тома (PV) в LVM

VI: Khôi phục Khối lượng vật lý (PV) bị lỗi trong LVM

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.