Join Log in
Score:0
soulmerge avatar Server

Recovering data from a zpool with a missing device

cn flag
soulmerge

I have a zpool consisting of two mirrored drives, storing all my personal data. I wanted to add another mirror drive, since the other drives have been in use for several years now. Instead, I managed to cripple the pool and cannot import it any longer.

This was the original layout of the pool (called "vault") before I did anything:

vault
  mirror-0
    sdd
    sdh

I tried to add another drive by issuing zpool add vault /dev/sdc, and got a warning from zpool (something about expecting a vdev, but scd being a disk). I ignored the warning, and forced the operation with -f (yes, that was pretty stupid ... )

I ended up with the following pool configuration:

vault
  mirror-0
    sdd
    sdh
  sdc

Noticing that sdc was not part of the mirror, I tried to remove it from the pool again and realized that this is not possible. I ignored the extra drive and rebooted, and assumed that my data is already safe inside the mirror-0 and decided to use repurpose sdc and use another drive instead, overwriting the first 100MB of sdc with zeros: dd if=/dev/zero of=/dev/sdc bs=1MB count=100 (yes, I am particularly stupid today ... )

Now when I try to import my zpool, I get the following error:

root@atlas:~# zpool import                                                                                                                         
   pool: vault                                                                                                                                     
     id: 3553551207657826252                                                                                                                       
  state: UNAVAIL                                                                                                                                   
 status: One or more devices are missing from the system.                                                                                          
 action: The pool cannot be imported. Attach the missing                                                                                           
        devices and try again.                                                                                                                     
   see: http://zfsonlinux.org/msg/ZFS-8000-6X                                                                                                      
 config:                                                                                                                                           
                                                                                                                                                   
        vault        UNAVAIL  missing device                                                                                                       
          mirror-0   ONLINE                                                                                                                        
            sdd      ONLINE                                                                                                                        
            sdh      ONLINE                                                                                                                        
                                                                                                                                                   
        Additional devices are known to be part of this pool, though their                                                                         
        exact configuration cannot be determined.

The data in the mirror on these drives is extremely important to me, and I really hope there is a way of restoring the data from the "partial" pool. Since I had rebooted a few minutes after the bogus add operation, I am assuming that the metadata for the file system was wiped, and that there is a way to access the raw data somehow. But I have no clue where to start.

Can anyone help me with this?

246
0 + 0
zfs
Score:0
soulmerge avatar Server
cn flag
soulmerge

I have managed to recover parts of the data by upgrading debian to bullseye (in order to have access to a more recent version of zdb) and importing the zpool with missing vdevs:

echo 1 >> /sys/module/zfs/parameters/zfs_max_missing_tvds
zpool import -o readonly=on vault

Some of the filesystems on the zpool were corrupt, though. I have copied the intact data onto a new zpool, and probed earlier txg values to find one that did not throw any errors:

zdb -e vault -ul | grep ' txg' | sed 's/.*= //' | sort | head
zdb -deG -o zfs_max_missing_tvds=1 -T 5102203 vault  # <- errors
zdb -deG -o zfs_max_missing_tvds=1 -T 5102202 vault  # <- errors
zdb -deG -o zfs_max_missing_tvds=1 -T 5102201 vault  # <- worked!

I was then able to import the zpool at the intact transaction group id:

zpool import -o readonly=on -T 5102201 vault

This command took about 15 hours for my 2x4TB mirror, but I could access all my files again. The data has since been copied onto a new zpool that I won't ever touch in a sleep-deprived state again.

0 + 0
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.