Score:3

apt forgot about more than 1,000 packages, and I'm unable to reinstall

us flag

I'm running 20.04.2 LTS (Focal Fossa)

I hadn't run aptitude for a couple days. It always worked fine, but when I tried it now, it started giving me the "e Examine" dialog each time I run it. When I type "e", it goes into a 10-20 minute run with the following showing at the top of the page:

open: aaa; closed: bbb; defer: ccc; conflict: ddd

Where 'aaa', 'bbb', 'ccc', 'ddd' are increasing numeric values ... and 'aaa' and 'bbb' are increasing rapidly, up to more than 50,000.

After a while, aptitude simply exits.

At this point, if I run dpkg --configure -a, it quickly returns to the command prompt with no error. But then, when I run apt-get install -f, I get a list of more than 1,000 package names followed by this ...

Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Most of these packages (probably all of them, actually) are items that I use from day to day, and they should not be removed.

Oh, also, if I capture the huge list of packages that are shown when I do apt-get install -f, and if I generate a thousand or so apt install PKG commands for each of these package names and then run this, I just get multiple lists of those same thousand-or-so packages for each of these commands, followed by that same "use sudo apt autoremove" message after each apt install PKG run.

Somehow, my apt caches seem to have gotten hosed. I don't know what could have caused this.

However, I don't really want to spend time trying to figure out why this happened. I just want to fix it so that all those packages are once again properly installed and manageable via apt.

Can anyone point me to docs that could help me, or perhaps to suggest a solution?

Thank you very much in advance.

PS: my /etc/apt/sources.list hasn't changed for weeks, nor have any files in /etc/apt/source.list.d changed in a long time. Actually, nothing under /etc/apt seems to have changed since a couple months ago.

PPS*: Here are the last dozen or so lines of /var/log/apt/history.log ...

Start-Date: 2021-07-24  20:01:24
Requested-By: ljz (1000)
Install: linux-headers-5.4.0-80-generic:amd64 (5.4.0-80.90, automatic), linux-modules-5.4.0-80-generic:amd64 (5.4.0-80.90, automatic), linux-headers-5.4.0-80:amd64 (5.4.0-80.90, automatic), linux-image-5.4.0-80-generic:amd64 (5.4.0-80.90, automatic), linux-modules-extra-5.4.0-80-generic:amd64 (5.4.0-80.90, automatic)
Upgrade: networkd-dispatcher:amd64 (2.0.1-1, 2.1-2~ubuntu20.04.1), linux-headers-generic:amd64 (5.4.0.77.80, 5.4.0.80.84), linux-libc-dev:amd64 (5.4.0-77.86, 5.4.0-80.90), libcurl4:amd64 (7.68.0-1ubuntu2.5, 7.68.0-1ubuntu2.6), ruby2.7:amd64 (2.7.0-5ubuntu1.4, 2.7.0-5ubuntu1.5), libsystemd0:amd64 (245.4-4ubuntu3.7, 245.4-4ubuntu3.11), linux-image-generic:amd64 (5.4.0.77.80, 5.4.0.80.84), libsystemd-dev:amd64 (245.4-4ubuntu3.7, 245.4-4ubuntu3.11), python3-pip:amd64 (20.0.2-5ubuntu1.5, 20.0.2-5ubuntu1.6), udev:amd64 (245.4-4ubuntu3.7, 245.4-4ubuntu3.11), ruby2.7-dev:amd64 (2.7.0-5ubuntu1.4, 2.7.0-5ubuntu1.5), ruby2.7-doc:amd64 (2.7.0-5ubuntu1.4, 2.7.0-5ubuntu1.5), libudev1:amd64 (245.4-4ubuntu3.7, 245.4-4ubuntu3.11), systemd-timesyncd:amd64 (245.4-4ubuntu3.7, 245.4-4ubuntu3.11), libxnvctrl0:amd64 (460.73.01-0ubuntu0.20.04.1, 470.57.01-0ubuntu0.20.04.1), libglib2.0-dev-bin:amd64 (2.64.6-1~ubuntu20.04.3, 2.64.6-1~ubuntu20.04.4), libruby2.7:amd64 (2.7.0-5ubuntu1.4, 2.7.0-5ubuntu1.5), systemd-sysv:amd64 (245.4-4ubuntu3.7, 245.4-4ubuntu3.11), libglib2.0-bin:amd64 (2.64.6-1~ubuntu20.04.3, 2.64.6-1~ubuntu20.04.4), libpam-systemd:amd64 (245.4-4ubuntu3.7, 245.4-4ubuntu3.11), libglib2.0-dev:amd64 (2.64.6-1~ubuntu20.04.3, 2.64.6-1~ubuntu20.04.4), libglib2.0-doc:amd64 (2.64.6-1~ubuntu20.04.3, 2.64.6-1~ubuntu20.04.4), systemd:amd64 (245.4-4ubuntu3.7, 245.4-4ubuntu3.11), libnss-systemd:amd64 (245.4-4ubuntu3.7, 245.4-4ubuntu3.11), libglib2.0-data:amd64 (2.64.6-1~ubuntu20.04.3, 2.64.6-1~ubuntu20.04.4), curl:amd64 (7.68.0-1ubuntu2.5, 7.68.0-1ubuntu2.6), linux-generic:amd64 (5.4.0.77.80, 5.4.0.80.84), python-pip-whl:amd64 (20.0.2-5ubuntu1.5, 20.0.2-5ubuntu1.6), libglib2.0-0:amd64 (2.64.6-1~ubuntu20.04.3, 2.64.6-1~ubuntu20.04.4), libcurl3-gnutls:amd64 (7.68.0-1ubuntu2.5, 7.68.0-1ubuntu2.6)
End-Date: 2021-07-24  20:02:46

Start-Date: 2021-07-25  19:31:25
Requested-By: ljz (1000)
Remove: linux-modules-extra-5.4.0-74-generic:amd64 (5.4.0-74.83), linux-modules-extra-5.4.0-73-generic:amd64 (5.4.0-73.82)
End-Date: 2021-07-25  19:31:28

Start-Date: 2021-07-27  18:56:53
Requested-By: ljz (1000)
Upgrade: libaspell15:amd64 (0.60.8-1build1, 0.60.8-1ubuntu0.1), aspell:amd64 (0.60.8-1build1, 0.60.8-1ubuntu0.1), libmysqlclient21:amd64 (8.0.25-0ubuntu0.20.04.1, 8.0.26-0ubuntu0.20.04.2), teamviewer:amd64 (15.19.3, 15.20.3)
End-Date: 2021-07-27  18:57:05

aptitude worked fine around 2021-07-24, so whatever broke my setup must have occurred after that.

Output of sudo aptitude search ?broken

iB  anydesk                                                 - The fastest remote desktop software on the market.                
iBA apt                                                     - commandline package manager                                       
iBA aptitude                                                - terminal-based package manager                                    
iBA base-files                                              - Debian base system miscellaneous files                            
iBA base-passwd                                             - Debian base system master password and group files                
iBA bash                                                    - GNU Bourne Again SHell                                            
iBA bsdutils                                                - basic utilities from 4.4BSD-Lite                                  
iB  chromium-common                                         - web browser - common resources used by the chromium packages      
iB  chromium-sandbox                                        - web browser - setuid security sandbox for chromium                
iB  containerd.io                                           - An open and reliable container runtime                            
iBA coreutils                                               - GNU core utilities                                                
iBA dash                                                    - POSIX-compliant shell                                             
iBA debianutils                                             - Miscellaneous utilities specific to Debian                        
iBA diffutils                                               - File comparison utilities                                         
iBA dpkg                                                    - Debian package management system                                  
iBA e2fsprogs                                               - ext2/ext3/ext4 file system utilities                              
iBA fdisk                                                   - collection of partitioning utilities                              
iBA findutils                                               - utilities for finding files--find, xargs                          
iB  google-chrome-stable                                    - The web browser from Google                                       
iBA grep                                                    - GNU grep, egrep and fgrep                                         
iBA gzip                                                    - GNU compression utilities                                         
iBA hostname                                                - utility to set/show the host name or domain name                  
iBA libc-bin                                                - GNU C Library: Binaries                                           
iBA libcurl3-gnutls                                         - easy-to-use client-side URL transfer library (GnuTLS flavour)     
iBA libglib2.0-0                                            - GLib library of C routines                                        
iBA libnss-systemd                                          - nss module providing dynamic user and group name resolution       
iBA libpam-systemd                                          - system and service manager - PAM module                           
iBA libsystemd0                                             - systemd utility library                                           
iBA libudev1                                                - libudev shared library                                            
iBA linux-base                                              - Linux image base package                                          
iBA linux-headers-5.4.0-80-generic                          - Linux kernel headers for version 5.4.0 on 64 bit x86 SMP          
iBA linux-image-5.4.0-80-generic                            - Signed kernel image generic                                       
iBA linux-image-generic                                     - Generic Linux kernel image                                        
iBA linux-modules-extra-5.4.0-80-generic                    - Linux kernel extra modules for version 5.4.0 on 64 bit x86 SMP    
iBA login                                                   - system login tools                                                
iBA mount                                                   - tools for mounting and manipulating filesystems                   
iBA ncurses-bin                                             - terminal-related programs and man pages                           
iB  net.downloadhelper.coapp                                - Video DownloadHelper companion app                                
iBA networkd-dispatcher                                     - Dispatcher service for systemd-networkd connection status changes 
iB  ocenaudio                                               - Ocenaudio is a fast, cross-platform audio editor.                 
iB  palemoon                                                - Firefox-based, efficient and easy to use web browser              
iBA perl-base                                               - minimal Perl system                                               
iB  rpi-imager                                              - Raspberry Pi imaging utility                                      
iBA sed                                                     - GNU stream editor for filtering/transforming text                 
iB  signal-desktop                                          - Private messaging from your desktop                               
iB  signal-desktop-beta                                     - Private messaging from your desktop                               
iBA systemd                                                 - system and service manager                                        
iBA systemd-timesyncd                                       - minimalistic service to synchronize local time with NTP servers   
iBA sysvinit-utils                                          - System-V-like utilities                                           
iBA tar                                                     - GNU version of the tar archiving utility                          
iBA util-linux                                              - miscellaneous system utilities                                    
iB  vdpau-va-driver                                         - VDPAU-based backend for VA API                                    
iB  zoom                                                    - Zoom, #1 Video Conferencing and Web Conferencing Service          

And here's more:

% sudo apt update
Hit:1 https://linux.teamviewer.com/deb stable InRelease
Hit:2 https://updates.signal.org/desktop/apt xenial InRelease                                     
Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]                         
Hit:4 http://us.archive.ubuntu.com/ubuntu focal InRelease                                                    
Hit:5 http://us.archive.ubuntu.com/ubuntu focal-updates InRelease                                            
Get:6 http://us.archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB]
Hit:7 http://archive.canonical.com/ubuntu focal InRelease                                
Fetched 214 kB in 1s (178 kB/s)
Reading package lists... Done
Building dependency tree       
Reading state information... Done
3 packages can be upgraded. Run 'apt list --upgradable' to see them.

% sudo apt list --upgradeable
Listing... Done
signal-desktop-beta/xenial 5.11.0-beta.3 amd64 [upgradable from: 5.8.0-beta.1]
signal-desktop/xenial 5.10.0 amd64 [upgradable from: 5.9.0]
ubuntu-advantage-tools/focal-updates 27.2.2~20.04.1 amd64 [upgradable from: 27.1~20.0

% sudo apt-get install -f
Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
  fonts-ubuntu fonts-uralic fotoxx fotoxx-common friendly-recovery ftp gdal-data gegl genisoimage geoclue-2.0 geoip-database
  [ ... more than 100 lines containing more package names ... ]
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.

Still more ...

grep -r "^deb" /etc/apt/ --include=*.list
/etc/apt/sources.list:deb http://us.archive.ubuntu.com/ubuntu/ focal main restricted
/etc/apt/sources.list:deb-src http://us.archive.ubuntu.com/ubuntu/ focal restricted main universe multiverse
/etc/apt/sources.list:deb http://us.archive.ubuntu.com/ubuntu/ focal-updates main restricted
/etc/apt/sources.list:deb-src http://us.archive.ubuntu.com/ubuntu/ focal-updates restricted main universe multiverse
/etc/apt/sources.list:deb http://us.archive.ubuntu.com/ubuntu/ focal universe
/etc/apt/sources.list:deb http://us.archive.ubuntu.com/ubuntu/ focal-updates universe
/etc/apt/sources.list:deb http://us.archive.ubuntu.com/ubuntu/ focal multiverse
/etc/apt/sources.list:deb http://us.archive.ubuntu.com/ubuntu/ focal-updates multiverse
/etc/apt/sources.list:deb http://us.archive.ubuntu.com/ubuntu/ focal-backports main restricted universe multiverse
/etc/apt/sources.list:deb-src http://us.archive.ubuntu.com/ubuntu/ focal-backports main restricted universe multiverse
/etc/apt/sources.list:deb http://archive.canonical.com/ubuntu focal partner
/etc/apt/sources.list:deb-src http://archive.canonical.com/ubuntu focal partner
/etc/apt/sources.list:deb http://security.ubuntu.com/ubuntu focal-security main restricted
/etc/apt/sources.list:deb-src http://security.ubuntu.com/ubuntu focal-security restricted main universe multiverse
/etc/apt/sources.list:deb http://security.ubuntu.com/ubuntu focal-security universe
/etc/apt/sources.list:deb http://security.ubuntu.com/ubuntu focal-security multiverse
/etc/apt/sources.list.d/teamviewer.list:deb https://linux.teamviewer.com/deb stable main
/etc/apt/sources.list.d/signal-xenial.list:deb [arch=amd64 signed-by=/usr/share/keyrings/signal-desktop-keyring.gpg] https://updates.signal.org/desktop/apt xenial main

Possible solution ...

What if I do the following? ...

  1. run sudo apt-get install -f >apt-installs
  2. edit that file, removing the lines that precede and follow the package names, and putting "sudo apt install " at the beginning of each line
  3. run "sudo apt autoremove"
  4. run "bash apt-installs" to reinstall each and every package.

Here's what the "apt-installs" file would look like after step 2 ...

sudo apt install accountsservice acpi-support acpid adb alsa-base android-libadb android-libbase android-libboringssl android-libcrypto-utils
sudo apt install android-libcutils android-liblog android-sdk-platform-tools-common apt-rdepends aptdaemon aptdaemon-data apturl apturl-common
sudo apt install aria2 audacious audacious-plugins audacious-plugins-data bind9-dnsutils blueman bluez bluez-cups brltty brltty-x11 brotli
sudo apt install busybox-static byobu catdoc catfish cheese-common cinnamon-desktop-data cinnamon-l10n clang-format-11 clang-tidy-11
sudo apt install clang-tools-11 command-not-found cppcheck cvs dc dctrl-tools debhelper debtree devilspie devilspie2 dh-autoreconf
sudo apt install dh-elpa-helper dh-strip-nondeterminism diffstat distro-info dmz-cursor-theme doc-base docbook-xml dwz efibootmgr elfutils
[ ... etc. ... for more than 100 lines ]

What do you folks think? Could this fix my problem?

More follow-up ...

After more investigation about apt, I now see that the sudo apt autoremove will indeed uninstall all of those packages. That's over 1,000 packages, and the my system will indeed become unusuable after that. And it's unclear to me whether running all of those sudo apt install ... commands will even work, given that so much other software would have been uninstalled.

So, I'm still stuck. :(

I don't want to uninstall any software. I just want to get the apt caches back into a usable state, without changing any existing software installations.

guiverc avatar
cn flag
You've provided no specifics allowing us to see, but it sounds like you've made inappropriate changes to your sources & polluted your originally *focal* system, and either created a condition that requires the removal of loads of packages to correct issues, or you've caused the removal by the addition of a inappropriate package to your system, either way the cause will be found in your apt logs (`/var/log/apt/history.log`) which we cannot view; just as we cannot see the output of `sudo apt update` which shows potential issues.
HippoMan avatar
us flag
I listed every command I ran, verbatim. For example `dpkg configure -a`, which I said produced no output, and `apt-get install -f`, which I mentioned explicitly. I am not going to list the hundreds lines of output that it produced, because each line was simply a list of packages. And I listed the explicit output that it gave at the end, after the 1,000-or-so package names were listed. What could be more specific? That is a real question, not a rhetorical question. Thank you very much.
HippoMan avatar
us flag
I just now added a "PPS" to the end of my post with the contents of `/var/log/apt/history.log`.
N0rbert avatar
zw flag
Please add the output of `aptitude search ?broken` to the question body.
HippoMan avatar
us flag
I have just added the `aptitude search ?broken` output to the question body.
HippoMan avatar
us flag
... and I added even more detail now to the question.
N0rbert avatar
zw flag
Please also share the output of `grep -r "^deb" /etc/apt/ --include=*.list` . Running autoremove by `sudo apt autoremove` would be useful too.
HippoMan avatar
us flag
I added the output of the `grep` command to the question. However, I'm afraid to run `apt autoremove`, because I don't want the 1,000-plus packages to be uninstalled. Right now, my system is running fine, but I'm afraid that will end if I do `apt autoremove`. Or will I still have the same runnable system after the autoremove?
HippoMan avatar
us flag
I posted a "Possible solution" at the end of my question. What do you folks think? I just want to make sure that the "apt autoremove" doesn't leave me with a non-usable system.
HippoMan avatar
us flag
After more investigation about `apt`, I now think that "autoremove" will indeed leave my system in a quite unusable state. Not only will the apt caches be cleared, but each of the more than 1,000 packages will also be uninstalled. So, I still am at a loss as to how to solve this.
HippoMan avatar
us flag
I figured out how to fix this. See my answer below.
Score:1
us flag

I figured out how to fix this problem. I needed to remove and reinstall all of those 1,000-plus packages, and I did so one-by-one. And then I needed to reinstall aptitude. To acomplish this fix, I performed the following steps ...

Step 1:

% sudo apt-get install -f >apt-installs

After running this, the apt-installs file contained this data ...

Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
  activity-log-manager apg audacious audacious-plugins cppcheck cracklib-runtime dh-strip-nondeterminism docbook-xml
  [ ... more than 180 lines containing lists of package names ... ]
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.

Step 2:

I edited this apt-installs file to remove the top 4 lines and the bottom 2 lines, and I reformatted it so that there is one package name per line.

Step 3:

I prepended the string fixit to the beginning of each line.

Step 4:

I created a shell function called fixit to remove and then install the specified package, and I put that function at the top of the apt-installs file. Then, here's what the first dozen or so lines of apt-installs look like ...

fixit() {
  [[ $# -lt 1 ]] && return 0
  package="${1}"
  shift
  echo "### fixing ${package}"
  apt-get -y remove "${package}"
  apt-get -y install "${package}"
  echo ''
}
              
fixit activity-log-manager
fixit apg
fixit audacious
fixit audacious-plugins
fixit cppcheck
fixit cracklib-runtime
fixit dh-strip-nondeterminism
fixit docbook-xml
fixit gkbd-capplet
fixit gnome-bluetooth
fixit gnome-control-center-data
[ ... etc. ... more than 1,500 lines ]

Step 5:

I ran this:

% sudo bash apt-installs

This removed and reinstalled each of the packages.

The advantage of these piecemeal "remove/install" runs for each package is that only one, single package is uninstalled and then reinstalled at any given moment, instead of blowing away more than 1,500 pieces of software via an apt autoremove and thereby rendering my system un-runnable.

Step 6:

I then ran these two commands ...

% sudo apt-get purge aptitude
% sudo apt-get install aptitude

Before I did this, aptitude was still hanging and crashing, even after the reinstalls of all the packages. This purge/install fixed the aptitude crashes.

And now, apt, apt-get, and aptitude are all working fine again, and my system is still usable.

All's well that ends well.

Score:0
zw flag

Usually such issues may be fixed by removing of auto-removable packages by

sudo apt autoremove

and then launching Aptitude in interactive mode by

sudo aptitude

and then hitting F10, ActionsCancel pending actions, then g to ensure that all is well.

HippoMan avatar
us flag
Thank you very much. I just now did that ... first `F10` "Cancel pending actions" and then `F10` again, followed by "g". At this point, I am getting a "Trying to fix broken packages" message, but my system is mostly hung as that message just sits on the screen. My CPU is at 100 percent. And the red "Resolving dependencies" line is at the bottom, and underneath it is another red line with "e Examine". I'll let this run for a while, but I'm not optimistic.
HippoMan avatar
us flag
... and after around 10 minutes, `aptitude` simply crashed.
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.