Score:8

Why is apt so dependent on individual Python versions?

es flag

We get a lot of questions here about people messing with Python versions and ruining their systems, most noticeably that apt fails to function properly. I am aware that many Ubuntu packages require specific versions of Python to function properly.

My question is: what makes a newer/different Python version incompatible with apt? Do these packages depend on language features that only exist in certain versions, or what version-specific Python features does apt depend on that makes it only work with one particular Python version?

cn flag
This is not a q+a kind of question :( I believe https://discourse.ubuntu.com/ might be better. I would not mind if Ubuntu was locked down to a specific python but installing another version would automatically envoke the ENVVARS method . So a setup where python version automatically co-exist. The reason will be: python is easy to code. `apt` code totally be a C or C+ tool or even Perl. But python is a lot easier and is taught more in schools.
Esther avatar
es flag
@Rinzwind my question is less about why a decision was made, and more about technically how it works. Like what makes a newer Python version incompatible with apt? Or is it just because it was decided that it shouldn't work, but technically it could?
Esther avatar
es flag
@Rizwind although thanks for the Discourse link, I will be checking that out as well
raj avatar
cn flag
raj
Just as a side note, it's not only apt, it's the same case with a lot of other Ubuntu tools as well.
mchid avatar
bo flag
You can have different python versions but you can't have a different default python version and you need to keep the python versions that other packages depend on, so those packages do not break. Can you please provide another specific example? The example you provided mentions removing a version of Python and not installing a different version. If and when more than one version exist on the system, you simply have to call the alternate version specifically, like `python3.14` for example, and you can't simply run `python` or `python3` which are linked to the default versions.
mchid avatar
bo flag
Installing an alternative version of Python without knowing what you're doing can be a bit dangerous because you cannot replace, override, or uninstall the default versions without causing problems. So, if you would like to install an alternative version of Python on your system, this would be a good question to ask and completely on topic here on Askubuntu. There's also multiple different ways of installing. You can even install locally. You can also [use different a version for an isolated python virtual environment](https://stackoverflow.com/a/1534343) which is one reason to use a venv.
mchid avatar
bo flag
There again with the virtual environment, you would explicitly call the alternate version of python by specifying the full path to the alternate version. This will tell the system that you want that instead of the default, so anything calling the default remains unchanged but the other version is still available to you.
muru avatar
us flag
`apt` is dependent on Python? Since when? It's a C++ program AFAIK. And we even had a question recently asking why Docker images of Ubuntu don't have Python on them, and those images definitely do have `apt`.
Raffa avatar
jp flag
"*Do these packages depend on language features that only exist in certain versions*" This is hardly the reason since "moving" Ubuntu to Python3 ... "*or what version-specific Python features*" Mainly modules related differences between versions ... "*does apt depend*" Not APT itself but rather its [low level librarries/tools/backends](https://github.com/Debian/apt)
Raffa avatar
jp flag
.. And the majority of Python related errors printed while using APT to e.g. install some packages is mostly generated by those packages install/configure scripts that ether use python directly or call system utilities that rely on Python ... That might cause a general misunderstanding that APT is depending on Python ... I think
Score:6
bo flag

Yes, it usually is specific language features that only exist in a certain Python version.

Depending on the issue, you will often get an error when an application like apt or more likely dpkg calls for something using the wrong syntax that doesn't exist in the newer version.

However apt itself does not depend on python and the same is true for dpkg — but the packages that do depend on python often have pre and post install scripts that invoke python or python scripts.

For example, if you download the deb file for software-properties-gtk, extract that file, and then extract the control.tar.xz file, you will see a prerm script.

The script is an sh script but you can see that it also calls on python:

#!/bin/sh
set -e

# Automatically added by dh_python3:
if which py3clean >/dev/null 2>&1; then
    py3clean -p software-properties-gtk 
else
    dpkg -L software-properties-gtk | perl -ne 's,/([^/]*)\.py$,/__pycache__/\1.*, or next; unlink $_ or die $! foreach glob($_)'
    find /usr/lib/python3/dist-packages/ -type d -name __pycache__ -empty -print0 | xargs --null --no-run-if-empty rmdir
fi

# End automatically added section

So py3clean is part of python3-minimal and this script calls on python3 in the shebang on the first line of the file as seen from the following commands:

which py3clean

and this should show /usr/bin/py3clean so:

head /usr/bin/py3clean 

output:

#! /usr/bin/python3
# vim: et ts=4 sw=4

# Copyright © 2010-2012 Piotr Ożarowski <[email protected]>
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is

This is important because an actual file for /usr/bin/python3 does not exist as you can see by the following command:

file /usr/bin/python3

On 20.04, this tells me that /usr/bin/python3 is a symbolic link to python3.8 and which python3.8 shows this file is /usr/bin/python3.8. Furthermore, file /usr/bin/python3.8 confirms that this is the actual Python executable used for python3.

So the software-properties-gtk package is expecting that, if python3 is installed, that the version of python3 installed is python3.8.

The py3clean script may or may not work with a different python version, but there's often enough changes to python that there's probably at least one change somewhere in that script that would cause some type of error.

Even if not for this particular situation, you get the idea.

I guess to summarize, this isn't actually a problem with apt but more of a python problem because python scripts often use a generic shebang that expects a particular version of python when the shebang specifies /usr/bin/python or /usr/bin/python3 or even /usr/bin/env python or /usr/bin/env python3 as none of these point to an actual file, they all point to a symbolic link that points to your default version of python installed on the system.

Since most packages are released for a particular version of Ubuntu, they all expect the default version of Python to be the same. Otherwise, each package would depend on a arbitrarily different Python version and we may end up with 3 or more versions of Python installed on the same system just to satisfy dependencies.

Typically, if you want or need a different version of Python installed, it's because you have a particular reason. And there's nothing stopping you from using a different version of Python for that reason, so long as you do not mess with the default version of python that other software use when they call on python or python3.

So if you do install an alternate version you will not have the convenience of calling that alternate version by python or python3 as your default version — you will need to specify the version specifically by calling python3.9, for example, when you use it.

I guess I should add as a disclaimer for others reading this answer. If you want to install a different version of python on your system, you should ask around to find the proper way to do it. It is possible to have multiple versions of Python on one system but to avoid problems, you must not uninstall, change, or alter in any way the default version of Python. But I don't think going into more detail would be within the scope of this question.

Esther avatar
es flag
one thing I'm unclear about is why a *newer* python version (which, as I understand, should have all the language features of an older one) still causes breaking changes. Although I've seen random other Python packages not work on newer versions, so I guess this wouldn't be any different, and it's more of a Python question
Steve Barnes avatar
cn flag
@Esther - the breaking issues generally fall into one of two categories 1) things that the author could get away with on the older version _but shouldn't have been doing_ and 2) things that have been deprecated for ages _that have been ignored for as long_. They are, of course, sometimes in a dependency of a given package rather than that package. Having had to maintain compatibility between python versions for several years with a little care it can be done. Of course this takes some effort from the maintainers but more from the user who need to **report** problems complete with test cases.
mchid avatar
bo flag
@Esther I think you're wrong to assume that the newer version has all the language features of the old one. From my experience, this is not the case. Python often changes and deprecates language features.
mchid avatar
bo flag
@Esther For example, [here's some stuff that was removed in 3.9 that would work in 3.8](https://docs.python.org/3/whatsnew/3.9.html#removed) and also some other [language changes that function differently in 3.9 compared to 3.8](https://docs.python.org/3/whatsnew/3.9.html#other-language-changes).
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.