Score:1

Broken lvm results in read_urandom error

ar flag

I was doing something in a chroot, and unfortunately I broke the host. Now I cannot manage the volume groups:

pvs
  read_urandom: /dev/urandom: open failed: No such file or directory

Same error with diferent commands about lvm.. Trying to reconfigure:

# dpkg-reconfigure linux-image-4.19.0-16-amd64

/etc/kernel/postinst.d/dkms:
dkms: running auto installation service for kernel 4.19.0-16-amd64:/usr/sbin/dkms: line 3345: /dev/fd/62: No such file or directory
.
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-4.19.0-16-amd64
cryptsetup: ERROR: Couldn't resolve device 
    /dev/mapper/rootvg-root--server--alpha--host
cryptsetup: WARNING: Couldn't determine root device
cryptsetup: ERROR: Couldn't resolve device /dev/dm-1 (deleted)
cryptsetup: ERROR: Couldn't resolve device 
    UUID=e9ef352b-a648-4499-ade2-54235f40a3df
W: Couldn't identify type of root file system for fsck hook
I: The initramfs will attempt to resume from /dev/dm-1 (deleted)
I: Set the RESUME variable to override this.
/etc/kernel/postinst.d/zz-update-grub:
/usr/sbin/grub-probe: error: failed to get canonical path of `/dev/mapper/rootvg-root--server--alpha--host'.
run-parts: /etc/kernel/postinst.d/zz-update-grub exited with return code 1

Trying yo update initramfs:

# update-initramfs -u
update-initramfs: Generating /boot/initrd.img-4.19.0-16-amd64
cryptsetup: ERROR: Couldn't resolve device 
    /dev/mapper/rootvg-root--server--alpha--host
cryptsetup: WARNING: Couldn't determine root device
cryptsetup: ERROR: Couldn't resolve device /dev/dm-1 (deleted)
cryptsetup: ERROR: Couldn't resolve device 
    UUID=e9ef352b-a648-4499-ade2-54235f40a3df
W: Couldn't identify type of root file system for fsck hook
I: The initramfs will attempt to resume from /dev/dm-1 (deleted)
I: Set the RESUME variable to override this.

Partition architecture:

# lsblk
NAME                                       MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
nvme1n1                                    259:0    0  1.8T  0 disk  
├─nvme1n1p1                                259:2    0    2M  0 part  
│ └─md1                                      9:1    0    2M  0 raid1 
├─nvme1n1p2                                259:3    0  510M  0 part  
│ └─md2                                      9:2    0  509M  0 raid1 /boot
└─nvme1n1p3                                259:4    0  1.8T  0 part  
  └─md3                                      9:3    0  1.8T  0 raid1 
    └─croot                                253:0    0  1.8T  0 crypt 
      ├─rootvg-swap--server--alpha--host 253:1    0    8G  0 lvm   
      ├─rootvg-root--server--alpha--host 253:2    0  1.5T  0 lvm   /
      ├─rootvg-root--vm1                   253:3    0  100G  0 lvm   
      ├─rootvg-root--vm2                   253:4    0   20G  0 lvm   
      ├─rootvg-root--vm3                   253:5    0   40G  0 lvm   
      └─rootvg-root--vm4                   253:6    0  100G  0 lvm   
nvme0n1                                    259:1    0  1.8T  0 disk  
├─nvme0n1p1                                259:5    0    2M  0 part  
│ └─md1                                      9:1    0    2M  0 raid1 
├─nvme0n1p2                                259:6    0  510M  0 part  
│ └─md2                                      9:2    0  509M  0 raid1 /boot
└─nvme0n1p3                                259:7    0  1.8T  0 part  
  └─md3                                      9:3    0  1.8T  0 raid1 
    └─croot                                253:0    0  1.8T  0 crypt 
      ├─rootvg-swap--server--alpha--host 253:1    0    8G  0 lvm   
      ├─rootvg-root--server--alpha--host 253:2    0  1.5T  0 lvm   /
      ├─rootvg-root--vm1                   253:3    0  100G  0 lvm   
      ├─rootvg-root--vm2                   253:4    0   20G  0 lvm   
      ├─rootvg-root--vm3                   253:5    0   40G  0 lvm   
      └─rootvg-root--vm4                   253:6    0  100G  0 lvm  

The problem was at doing chroot on rootvg-root--vm4. Maybe due to update-initramfs or removing files in /mnt that affected running processes on host. Now I cannot even delete that volume and I'm afraid to lost the server if a server reboot...

waiting for your support, thanks in advance.

edited:

More actions: Trying to restart a VM:

$ sudo virsh start vm2-bastion
sudo virsh start vm2

error: Failed to start domain vm2
error: internal error: Failed to probe QEMU binary with QMP: Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator

Lost the conventional access via SSH:

$ ssh user@ip
PTY allocation request failed on channel 0

...but I managed to login using:

$ ssh user@ip "/bin/bash -i" 

bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
user@server-alpha-host:~$
Michael Hampton avatar
cz flag
What exactly did you do?
ar flag
I did a rm -r /mnt/* and noticed the mistake, Ctrl-C. But removed some host`s chroot mounted /dev or proc...
Score:1
cn flag

Now I cannot even delete that volume and I'm afraid to lost the server if a server reboot...

Backup any important data. If this host is important to you, it must be possible to rebuild and recover it.

Determine if any block devices were mounted where and when the deletion happened, and if you care about data on them. Possibly not, but just because you sent it SIGINT doesn't mean the deletion stopped at device nodes.

Missing /dev/urandom, /dev/kvm, block devices, /proc, and other devices will break many things. Reboot to get them back. devtmpfs and udev recreates device nodes on every boot normally. This reboot is to ensure every misbehaving program is restarted, and must happen eventually.

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.