I have an Ubuntu 20.04.6 LTS server.
- It suddenly started refusing ssh connections citing
/bin/bash: No such file or directory
.
- Reboot the machine fails with a kernel panic, with the following excerpts:
Begin: Running /scripts/init-bottom ... mkdir: can't create directory '/root/lib/modules': Read-only file system
[...]
run-init: can't execute '/sbin/init', No such file or directory
[...]
run-init, can't execute '/etc/init': Permission denied
[...]
Kernel panic not syncing: Attempted to kill Init! exitcode-Ox00000100`
- Booting from a live USB, and looking inside the root disk.
/sbin/init
is not missing but it is a softlink and the /lib/systemd/systemd
it points to is missing. This leads me to find out that /lib
is a directory containing only a x86_64-linux-gnu
folder, while in other machines, /lib
is usually a softlink to usr/lib
- After backing up
/lib
and replacing it with a softlink to usr/lib
, I am now able to reboot and ssh into the machine!
- However, when trying to run
sudo apt update
and sudo apt upgrade
to make sure my packages are in order, I am now faced with these errors that make apt fail:
/usr/bin/python3: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.35' not found (required by /usr/bin/python3)
/usr/bin/python3: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /usr/bin/python3)
/usr/bin/python3: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /usr/bin/python3)
/usr/bin/python3: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /usr/bin/python3)
How may I recover the system in this situation, short of re-installing Ubuntu?
Also, what may have caused this situation?
Edit 1:
Here is the content of the x86_64-linux-gnu
I found at /lib
originally. I don't think it contains as many files as it's supposed to
$ ls [...]/lib/x86_64-linux-gnu/
libexpat.so.1 libexpat.so.1.8.7 libhistory.so.8 libhistory.so.8.1 libreadline.so.8 libreadline.so.8.1
$ diff [...]/lib/x86_64-linux-gnu /lib/x86_64-linux-gnu | grep -v "Only in /lib/x86_64-linux-gnu"
Binary files [...]/lib/x86_64-linux-gnu/libexpat.so.1 and /lib/x86_64-linux-gnu/libexpat.so.1 differ
Only in [...]/lib/x86_64-linux-gnu: libexpat.so.1.8.7
Binary files [...]/lib/x86_64-linux-gnu/libhistory.so.8 and /lib/x86_64-linux-gnu/libhistory.so.8 differ
Only in [...]/lib/x86_64-linux-gnu: libhistory.so.8.1
Binary files [...]/lib/x86_64-linux-gnu/libreadline.so.8 and /lib/x86_64-linux-gnu/libreadline.so.8 differ
Only in [...]/lib/x86_64-linux-gnu: libreadline.so.8.1
Also, these warnings also show up just when I SSH in:
/usr/bin/xauth: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /usr/bin/xauth)
/usr/bin/xauth: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /usr/bin/xauth)
/usr/bin/xauth: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /lib/x86_64-linux-gnu/libX11.so.6)
/usr/bin/xauth: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /lib/x86_64-linux-gnu/libX11.so.6)
/usr/bin/xauth: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /lib/x86_64-linux-gnu/libXau.so.6)
/usr/bin/xauth: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /lib/x86_64-linux-gnu/libbsd.so.0)
/usr/bin/xauth: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /lib/x86_64-linux-gnu/libmd.so.0)