Broken update and package mess: how do I remove old packages manually?

After a do-release-upgrade to switch from 16.04 to 18.04, something failed and machine was left in a "in between" state, with lots of broken dependencies issues. My package tree is a mess, I am desperate... ;-)

I tried lots of things (including dpkg --configure -a, randomly removing packages with dpkg --force depends -P <pkg-name>, plus other things), but I am kinda stuck at present.

So, when I try:

$ sudo apt --fix-broken install
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
   (here 20 lines)
The following additional packages will be installed:
... (here 5 lines)
 Suggested packages:
 ... more
Recommended packages:
 ... more
The following packages will be REMOVED:
 ... more
The following NEW packages will be installed:
 ... more (10 lines)
28 upgraded, 6 newly installed, 1 to remove and 327 not upgraded.
228 not fully installed or removed.
Need to get 0 B/13,3 MB of archives.
After this operation, 2 771 kB of additional disk space will be used.
Do you want to continue? [Y/n] 

Up to here, nothing I am afraid of. But now, entering Y:

Extracting templates from packages: 100%
(Reading database ... 977800 files and directories currently installed.)
Removing ubuntu-advantage-tools (27.0.2~18.04.1) ...
Traceback (most recent call last):
  File "<string>", line 2, in <module>
  File "/usr/lib/python3/dist-packages/uaclient/apt.py", line 9, in <module>
    from uaclient import gpg
  File "/usr/lib/python3/dist-packages/uaclient/gpg.py", line 3, in <module>
    import shutil
ModuleNotFoundError: No module named 'shutil'
dpkg: error processing package ubuntu-advantage-tools (--remove):
 installed ubuntu-advantage-tools package pre-removal script subprocess returned error exit status 1
Errors were encountered while processing:
 ubuntu-advantage-tools
E: Sub-process /usr/bin/dpkg returned an error code (1)

So this sounds like a Python error, from what I understand, it can't find shutil. So I manually downloaded what I thought was the thing: https://packages.ubuntu.com/bionic/python-backports-shutil-get-terminal-size and installed manually with:

dpkg -i sudo dpkg -i python-backports-shutil-get-terminal-size_1.0.0-5_all.deb

But doing apt --fix-broken install again ends up at the same point.

I have limited knowledge with Python, so I can't be sure this is the real issue, but any help is welcome. Else, next step will be probably reinstalling from scratch, but I'd really prefer to avoid that.

At present I am able to manually install packages (download from https://packages.ubuntu.com/ then dpkg -i) but that's about it, everything else fails.

For example, if I try to install libpython3.6-stdlib:

$ sudo apt-get install libpython3.6-stdlib

I get:

libpython3.6-stdlib : Depends: libpython3.6-minimal (= 3.6.9-1~18.04ubuntu1.4) but 3.6.13-1+xenial2 is to be installed

which means that I still have packages from 16.04 (Xenial).

How can I manually remove all the 16.04 packages?

Additional info (machine now boots only with a console):

• $ python --version => Python 3.6.13
• $ lsb_release -a is not available
• $ uname -a tells me its a 4.15 kernel (which seem consistent with 18.04)

FWIW: The key point was in fact here:

dpkg: error processing package ubuntu-advantage-tools (--remove):
 installed ubuntu-advantage-tools package pre-removal script subprocess returned error exit status 1

The whole process failed because of the failure of the "pre-removal" script for the package ubuntu-advantage-tools (no idea what that thing is!).

So I did:

$ sudo mv /var/lib/dpkg/info/ubuntu-advantage-tools.prerm /var/lib/dpkg/info/ubuntu-advantage-tools.prerm_OLD

and I was able to complete the sudo apt --fix-broken install command.

Reboot, then another issue (keyboard unrecognized) that was solved using another question, after booting in recovery mode:

$ sudo apt-get install xserver-xorg-input-all

Was finally able to boot graphically, but... still some issues with package manager still referencing xenial (16.04). I'll post another question.

This sounds like one very sick system; you might want to take it out of its misery, i.e. reinstall. If you don't have a separate /home filesystem already, copy your home directory and everything that you put work into away and start over to get a clean system again.

I'd also copy /etc away just in case some non-trivial configuration file is worthwhile rescuing (but I'd hold back copying it to the new system unless absolutely necessary).

Think of it this way: That system served you well for long years (some time just after 2016 since it's a 16.04); let it rest in peace. ;-) Now it's time for something new and clean.

I once made a checklist for myself what stuff I do to a freshly installed Ubuntu to get it into a shape that I find best usable; maybe you can draw some inspiration from it: https://github.com/shundhammer/huha-linux-tips/blob/master/doc/ubuntu-tips.md (some parts may be slightly outdated; nevermind)