When was the last time you downgraded your Linux system’s Bash?

Hopefully, the answer is a resounding “Never!”. Too many moving pieces rely on standard Unix environment infrastructure, such as Bash. It’d be silly to make incompatible changes to critical infrastructure.

Chances are, your favorite distro comes bundled with one (or two) Python installs of some version. And with that omnipotence comes reliance: Python has become a fundamental piece of a modern (Linux) system. Little scripts here and there (especially in Ubuntu) are powered by its assumed existence.

That begs an interesting question: what does an innocent Python update mean for your system?

It’s ok, I know what I’m doing

While the Python team does their best to soften the boat rocking, minor versions can make breaking changes. Overwriting a system Python can break assumptions or workarounds.

What about Python packages? Here’s a fun command for anyone on a Debian variant: count the number of non python3-* packages this returns.

apt-cache rdepends --no-recommends --no-suggests --no-enhances python3-requests

A manual, unlucky requests upgrade down the line could trample the package manager’s hard work, and replace an essential dependency.

In short: if you aren’t careful, customizing a system Python install can mean disaster.

Environment setups: which is best?

Best practices change quickly: my year old post about its tooling is already out of date! To keep misinformation to a minimum, let’s look at the current state of Python tooling & environments.

What options are available for Python + a few packages? Let’s start with the simplest.

The classic “sudo pip install” (spoilers: don’t do it)

Let’s add a package to my Ubuntu system via pip. requests is already installed: we can find its location and version.

$ python3 -c "import requests; print(requests.get('http://jibby.org'))"
<Response [200]>

$ python3 -c "import requests; print(requests.__file__)"

$ ls /usr/lib/python3/dist-packages | grep requests


2.18.4 is a bit old. Let’s upgrade to 2.23.0.

$ sudo pip3 install requests==2.23.0
Collecting requests==2.23.0
  Downloading https://files.pythonhosted.org/packages/1a/70/1935c770cb3be6e3a8b78ced23d7e0f3b187f5cbfab4749523ed65d7c9b1/requests-2.23.0-py2.py3-none-any.whl (58kB)
    100% |████████████████████████████████| 61kB 1.1MB/s
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests==2.23.0) (2018.8.24)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/lib/python3/dist-packages (from requests==2.23.0) (3.0.4)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/lib/python3/dist-packages (from requests==2.23.0) (1.24.1)
Requirement already satisfied: idna<3,>=2.5 in /usr/lib/python3/dist-packages (from requests==2.23.0) (2.6)
Installing collected packages: requests
  Found existing installation: requests 2.21.0
    Not uninstalling requests at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'requests'. No files were found to uninstall.

$ python3 -c "import requests; print(requests.__file__)"

$ ls /usr/local/lib/python3.6/dist-packages | grep requests

That “Not uninstalling requests” line is important: pip used a separate folder, instead of overwriting the system’s requests. This security is what keeps a sudo pip uninstall requests from touching something it shouldn’t.

Smart, right? Yes, but not without consequences. What does that mean for the rest of the system?

Let’s look for a system utility something that uses requests:

$ find /usr -regex ".*.py" -exec grep "import requests" {} \+
/usr/lib/python3/dist-packages/apport/ui.py:    import requests_unixsocket
/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceScaleway.py:import requests
/usr/lib/python3/dist-packages/cloudinit/url_helper.py:import requests
/usr/lib/python3/dist-packages/jsonschema/validators.py:    import requests
/usr/lib/python3/dist-packages/pip/download.py:from pip._vendor import requests, six
/usr/lib/python3/dist-packages/pip/req/req_set.py:from pip._vendor import requests
/usr/lib/python3/dist-packages/requests/__init__.py:   >>> import requests
/usr/lib/python3/dist-packages/requests/adapters.py:      >>> import requests
/usr/lib/python3/dist-packages/requests/api.py:      >>> import requests
/usr/lib/python3/dist-packages/requests/models.py:      >>> import requests
/usr/lib/python3/dist-packages/requests/models.py:      >>> import requests
/usr/lib/python3/dist-packages/requests/sessions.py:      >>> import requests
/usr/lib/python3/dist-packages/requests_unixsocket/__init__.py:import requests
/usr/lib/python3/dist-packages/ssh_import_id/__init__.py:import requests
/usr/local/lib/python3.6/dist-packages/requests/__init__.py:   >>> import requests
/usr/local/lib/python3.6/dist-packages/requests/adapters.py:      >>> import requests
/usr/local/lib/python3.6/dist-packages/requests/api.py:      >>> import requests
/usr/local/lib/python3.6/dist-packages/requests/models.py:      >>> import requests
/usr/local/lib/python3.6/dist-packages/requests/models.py:      >>> import requests
/usr/local/lib/python3.6/dist-packages/requests/sessions.py:      >>> import requests
/usr/local/lib/python3.6/dist-packages/requests/status_codes.py:    >>> import requests

ssh_import_id is a little Ubuntu tool for adding keyfiles. Which requests does it use? Let’s edit the beginning of /usr/bin/ssh-import-id to read:

import argparse
import sys
from ssh_import_id import *


And we’ll find out:

$ ssh-import-id --help
usage: ssh-import-id [-h] [-o FILE] [-r] [-u USERAGENT] USERID [USERID ...]

Authorize SSH public keys from trusted online identities.

positional arguments:
  USERID                User IDs to import

optional arguments:
  -h, --help            show this help message and exit
  -o FILE, --output FILE
                        Write output to file (default ~/.ssh/authorized_keys)
  -r, --remove          Remove a key from authorized keys file
  -u USERAGENT, --useragent USERAGENT
                        Append to the http user agent string

That’s the requests in /usr/local/lib, our new version. The result is the same if we’re root. Concerning.

Just to be safe, I modified a Python-using systemd service (cloud-init) to verify the same thing: at service startup, the new requests sneaks in.

Even though package manager files are not overwritten, system applications are using incorrect package versions.

The cause here is a bit complicated: sys.path shows where Python looks for packages first, and the site module determines how it gets there. Python has no way to specify versions of an import, so whatever folder is checked first takes priority.

Icing on the cake: security

In the nitty-gritty of a Python package, a little setup.py file describes what to install. Adding that package means executing a potentially untrustworthy setup.py: don’t do that with root access.


The virtualenv

As a rule of thumb: Don’t add or overwrite things on your system’s Python install. But can you use it without modifying it?

In the Python world, a virtualenv (shortened as venv) is a separated environment overlaying a Python install. Generally, venvs are created on a per-project basis, and specific packages/versions for a project are installed into the venv.

Say you have two applications running on your machine, each using a different version of requests. If both those used the host Python install, there’s no way to install 2 versions of requests in the same environment. However, if each project uses a venv with its respective version of requests, they can share the same base Python installation without any issues.

TODO creating a venv

TODO pip freeze, and never apply this to your Python install

TODO how pip freeze won’t include apt installed packages

TODO no dependency pinning for pip freeze