Score:0

LVM disk is corrupt - is any data recoverable?

in flag

Under RHEL7 we have a logical volume consisting of 27 volume groups PVs. Each of these VGsPVs was added sequentially as we ran out of space on the LV.

We do not have a backup of the data. There are no snapshots of the VM.

Recently it refused to mount with a "Remote I/O" error.

I turned this over to our sysadmins, who have basically said there is nothing more that they can do. They specifically told me:

  1. Redhat support has identified one of the disks has got corrupted and that is why this is not loading, and
  2. The storage disks are from our old SAN which is out of support now .
  3. We have run the diagnostics from the VMware side and this could not find and correct the errors.

Is there any way that any of data on the other disks could be recovered ? We understand that the data on the corrupt disk will be lost.

I do have root on this system and am willing to work on this without our sysadmin's support since they have thrown in the towel.

EDIT: Replaced "volume groups" with PVs.

Romeo Ninov avatar
in flag
Do you mean PV by VG? Because AFAIK one LV can reside in only one VG. SO you can't add VG's to LV.
jm flag
If your disks are supplied by VMware, have you considered a snapshot recovery from the hypervisor level? As for the SAN disks, are these generic iSCSI blocks or something more esoteric like EMC vsans?
tmark avatar
in flag
@RomeoNinov you're right. I meant PV where I wrote VG. I will edit.
Romeo Ninov avatar
in flag
@tmark, is it LV just set by default, or it is mirrored, or something else?
tmark avatar
in flag
@doneal24 - One of our sysadmins informs me: "They are a set of ESX vmdk files that have been strung together at the OS level of the server to form an ‘extended’ lun , at least I think that is the correct unix term ".
tmark avatar
in flag
@RomeoNinov I'm not sure how to confirm, but if there's a command I can run to find out, I will do so. I am 99% that they would have just created an LVM with settings as close to default a s possible.
Romeo Ninov avatar
in flag
In such case I suppose the LV is stripped over most (all) of the PVs. Which mean one missing PV is fatal for build the information. Maybe some data restoration companies will be able to extract some part of the information or even entire files. But this service can cost serious amount of money.
A.B avatar
cl flag
A.B
Usually the OS keeps a backup of the LVM metadata in /etc/lvm/backup or alike and then...
tmark avatar
in flag
@RomeoNinov I could well be wrong, but think there might a chance some of the data may not be distributed across multiple PVs. The PVs were added one-by-one as the LV was filled. So in this case, I would (perhaps naively and wrongly) hope that the particular corrupt PV would contain data from one contiguous block of time.
Romeo Ninov avatar
in flag
@tmark, that's right, but do not forget over the LV you have filesystem which need to be reconstructed. And files are allocated based on free blocks and may spread across more than one PV.
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.