Score:2

/lib is usually a softlink, but was replaced by a mostly empty directory

ro flag

I have an Ubuntu 20.04.6 LTS server.

  1. It suddenly started refusing ssh connections citing /bin/bash: No such file or directory.
  2. Reboot the machine fails with a kernel panic, with the following excerpts:
Begin: Running /scripts/init-bottom ... mkdir: can't create directory '/root/lib/modules': Read-only file system
[...]
run-init: can't execute '/sbin/init', No such file or directory
[...]
run-init, can't execute '/etc/init': Permission denied
[...]
Kernel panic not syncing: Attempted to kill Init! exitcode-Ox00000100`
  1. Booting from a live USB, and looking inside the root disk. /sbin/init is not missing but it is a softlink and the /lib/systemd/systemd it points to is missing. This leads me to find out that /lib is a directory containing only a x86_64-linux-gnu folder, while in other machines, /lib is usually a softlink to usr/lib
  2. After backing up /lib and replacing it with a softlink to usr/lib, I am now able to reboot and ssh into the machine!
  3. However, when trying to run sudo apt update and sudo apt upgrade to make sure my packages are in order, I am now faced with these errors that make apt fail:
/usr/bin/python3: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.35' not found (required by /usr/bin/python3)
/usr/bin/python3: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /usr/bin/python3)
/usr/bin/python3: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /usr/bin/python3)
/usr/bin/python3: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /usr/bin/python3)

How may I recover the system in this situation, short of re-installing Ubuntu? Also, what may have caused this situation?

Edit 1:

Here is the content of the x86_64-linux-gnu I found at /lib originally. I don't think it contains as many files as it's supposed to

$ ls [...]/lib/x86_64-linux-gnu/
libexpat.so.1  libexpat.so.1.8.7  libhistory.so.8  libhistory.so.8.1  libreadline.so.8  libreadline.so.8.1

$ diff [...]/lib/x86_64-linux-gnu /lib/x86_64-linux-gnu | grep -v "Only in /lib/x86_64-linux-gnu"
Binary files [...]/lib/x86_64-linux-gnu/libexpat.so.1 and /lib/x86_64-linux-gnu/libexpat.so.1 differ
Only in [...]/lib/x86_64-linux-gnu: libexpat.so.1.8.7
Binary files [...]/lib/x86_64-linux-gnu/libhistory.so.8 and /lib/x86_64-linux-gnu/libhistory.so.8 differ
Only in [...]/lib/x86_64-linux-gnu: libhistory.so.8.1
Binary files [...]/lib/x86_64-linux-gnu/libreadline.so.8 and /lib/x86_64-linux-gnu/libreadline.so.8 differ
Only in [...]/lib/x86_64-linux-gnu: libreadline.so.8.1

Also, these warnings also show up just when I SSH in:

/usr/bin/xauth: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /usr/bin/xauth)
/usr/bin/xauth: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /usr/bin/xauth)
/usr/bin/xauth: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /lib/x86_64-linux-gnu/libX11.so.6)
/usr/bin/xauth: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /lib/x86_64-linux-gnu/libX11.so.6)
/usr/bin/xauth: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /lib/x86_64-linux-gnu/libXau.so.6)
/usr/bin/xauth: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /lib/x86_64-linux-gnu/libbsd.so.0)
/usr/bin/xauth: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /lib/x86_64-linux-gnu/libmd.so.0)
Raffa avatar
jp flag
For that one, try copying the directory `x86_64-linux-gnu` that you backed up from `/lib/x86_64-linux-gnu` to `/usr/lib/x86_64-linux-gnu` assuming you have previously moved it from its original location and it's still intact ... You seem to have messed up your filesystem hierarchy quiet a bit.
hr flag
AFAIK Ubuntu 20.04 uses libc version 2.31 - so it suggests your /usr/bin/python3 is not the default version from the Ubuntu repo?
ro flag
@Raffa, I just edited the question to add more details on the contents of the `x86_64-linux-gnu` I backed up. I don't think it has everything it's supposed to, and the few packages still in it don't seem to relate to the warnings.
ro flag
@steeldriver I think you're right. Any way I could downgrade all packages to their default versions in Ubuntu repo, or install a higher version of `libc` & `libm` to match what is now being required? All without being able to use `apt`?
ro flag
@steeldriver, by the way, I can see `/usr/bin/python3 -> python3.10` and there is a `python3.8` available there. Maybe I could try redirecting `/usr/bin/python3` to 3.8?
Raffa avatar
jp flag
This related post might help: [Terminal and other apps suddenly won't open](https://askubuntu.com/q/1448702)
hr flag
@LemmeTestThat yes re-linking `/usr/bin/python3` to `python3.8` should help
ro flag
@steeldriver Linking `python3.8` did seem to get apt going further, but then It still crashed with an error from `/lib/x86_64-linux-gnu/libsqlite3.so.0`
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.