Exit read-only file system after system crash and hard reset (20.04.2 Server, VM)

Yesterday my system crashed forcing me to do a hard reset on my host machine whilst running 2 virtual machines. The first machine survived and the second has been put into a read only mode. After logging in to the VM I am presented with the following messages.

Unable to setup logging. [Errno 30] Read-only file system: '/var/log/landscape/sysinfo.log' run-parts: /etc/update-mot.d/50-landscape-sysinfo exited with return code 1 /usr/lib/ubuntu-release-upgrader/release-upgrade-motd: 31: cannot create /var/lib/ubuntu-release-upgrader/release-upgrade-available: Read-only file system mktemp: failedd to create file via template '/var/lib/update-notifier/tmp.XXXXXXXXXX': Read-only file system run-parts: /etc/update-motd.d/95-hwe-eol exited with return code 1 /usr/lib/update-notifier/update-motd-fsck-at-reboot: 38: cannot create /var/lib/update-notifier/fsck-at-reboot: Read-only file system

I have checked the disk image file from the host, which in this case is a qcow2 file which I use in tandem with KVM.

sudo qemu-img check ubuntu204.qcow2

Result of which is:

No errors were found on the image. 245760/245760 = 100.00% allocated, 0.00% fragmented, 0.00% compressed clusters Image end offset: 16108814336

I have read that this is a forced state to protect the file system and that if I scan the disk and everything is OK I can just exit read-only mode, How would I do that with the following partition table on tty in read only mode?

loop0                         0
loop1                         0
loop2                         0
loop3                         0
loop4                         0
loop5                         0
|-sda                         0   15G    0 disk
  |-sda1                      0   1M     0 part
  |-sda2                      0   1G     0 part /boot
  |-sda3                      0   14G    0 part
    |-ubuntu--vg-ubuntu--lv   0   14G    0 lvm /
|-sr0                         1   1024M  0 rom

From the warnings I suspect there may be a problem with unattended-upgrades I would also be interested in any further input regarding further checks you would recommend to ensure the integrity of both machines.

How to recover a qcow2 disk image file using fsck from the host

Load network block device module:

sudo modprobe nbd max_part=8

Poweroff machine:

sudo virsh destroy virtual-machine

Connect disk image:

sudo qemu-nbd --connect=/dev/nbd0 /var/lib/libvirt/images/virtual-machine.qcow2

Check disk: (Disk is now mounted on host you can see it in nautilus/explorer)

sudo fsck /dev/nbd0p2

For me i had two extra drives mounted which I just scanned both nbd-dm3 and nbd0p2, here is an example of expected output which I just answered yes to all.

fsck from util-linux 2.25.2
e2fsck 1.42.12 (29-Aug-2014)
/dev/nbd0p1: recovering journal
/dev/nbd0p1 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Inodes that were part of a corrupted orphan linked list found.  Fix<y>? yes
Inode 274 was part of the orphaned inode list.  FIXED.
Inode 132276 was part of the orphaned inode list.  FIXED.
Deleted inode 142248 has zero dtime.  Fix<y>? yes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences:  -603674 -623174 +(689342--689343)
Fix<y>? yes
Free blocks count wrong for group #18 (15076, counted=15077).
Fix<y>? yes
Free blocks count wrong for group #19 (11674, counted=11675).
Fix<y>? yes
Free blocks count wrong (632938, counted=670871).
Fix<y>? yes
Inode bitmap differences:  -274 -132276 -142248
Fix<y>? yes
Free inodes count wrong for group #0 (52, counted=53).
Fix<y>? yes
Free inodes count wrong for group #16 (99, counted=100).
Fix<y>? yes
Free inodes count wrong for group #17 (519, counted=520).
Fix<y>? yes
Free inodes count wrong (204392, counted=204599).
Fix<y>? yes

/dev/nbd0p1: ***** FILE SYSTEM WAS MODIFIED *****
/dev/nbd0p1: 101833/306432 files (0.2% non-contiguous), 553321/1224192 blocks

Disconnect device:

sudo qemu-nbd --disconnect /dev/nbd0

/dev/nbd0 disconnected

Start your machine and check if it worked:

virsh start virtual-machine

Source: https://gist.github.com/islander/8517685e3a9f7e0e1d458033710f0893