When was the last time you downgraded your Linux system’s Bash?

Never (I hope, oh do I hope). Too many moving pieces rely on “standard” Linux infrastructure like Bash. It’d be silly to make incompatible changes to critical infrastructure.

But, uh. Python is everywhere. Chances are, your favorite distro comes bundled with one (or two) Python of some version. And with that omnipotence comes reliance: Python has become a fundamental piece of a modern Linux system: little scripts are powered by its existence.

That begs an interesting question: what does an innocent Python update mean for your system?

It’s ok, I know what I’m doing

While the Python team does their best to soften the boat rocking, minor versions can make breaking changes. Overwriting a system Python can break assumptions or workarounds.

What about Python packages? Here’s a fun command for anyone on a Debian variant: count the number of non python3-* packages this returns.

apt-cache rdepends --no-recommends --no-suggests --no-enhances python3-requests

An unlucky requests upgrade down the line could trample the package manager’s hard work, and replace an essential dependency.

In short: if you aren’t careful, customizing a system Python install can mean disaster.

Environment setups: which is best?

Best practices change quickly: my year old post about its tooling is already out of date! To keep misinformation to a minimum, let’s look at the current state of Python tooling & environments.

What options are available for Python + a few packages? Let’s start with the simplest.

The classic “sudo pip install” (spoilers: don’t do it)

Let’s add a package to my Ubuntu system via pip. requests is already installed: we can find its location and version.

$ python3 -c "import requests; print(requests.get('http://jibby.org'))"
<Response [200]>

$ python3 -c "import requests; print(requests.__file__)"
/usr/lib/python3/dist-packages/requests/__init__.py

$ ls /usr/lib/python3/dist-packages | grep requests
requests
requests-2.18.4.egg-info
requests_unixsocket
requests_unixsocket-0.1.5.egg-info

$

2.18.4 is a bit old. Let’s upgrade to 2.23.0.

$ sudo pip3 install requests==2.23.0
Collecting requests==2.23.0
  Downloading https://files.pythonhosted.org/packages/1a/70/1935c770cb3be6e3a8b78ced23d7e0f3b187f5cbfab4749523ed65d7c9b1/requests-2.23.0-py2.py3-none-any.whl (58kB)
    100% |████████████████████████████████| 61kB 1.1MB/s
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests==2.23.0) (2018.8.24)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/lib/python3/dist-packages (from requests==2.23.0) (3.0.4)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/lib/python3/dist-packages (from requests==2.23.0) (1.24.1)
Requirement already satisfied: idna<3,>=2.5 in /usr/lib/python3/dist-packages (from requests==2.23.0) (2.6)
Installing collected packages: requests
  Found existing installation: requests 2.21.0
    Not uninstalling requests at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'requests'. No files were found to uninstall.

$ python3 -c "import requests; print(requests.__file__)"
/usr/local/lib/python3.6/dist-packages/requests/__init__.py

$ ls /usr/local/lib/python3.6/dist-packages | grep requests
 requests
 requests-2.23.0.dist-info

That “Not uninstalling requests” line is important: pip used a separate folder, instead of overwriting the system’s requests. This security is what keeps a sudo pip uninstall requests from touching something it shouldn’t.

Smart, right? Yes, but not without consequences. What does that mean for the rest of the system?

Let’s look for a system utility something that uses requests:

$ find /usr -regex ".*.py" -exec grep "import requests" {} \+
/usr/lib/python3/dist-packages/apport/ui.py:    import requests_unixsocket
/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceScaleway.py:import requests
/usr/lib/python3/dist-packages/cloudinit/url_helper.py:import requests
/usr/lib/python3/dist-packages/jsonschema/validators.py:    import requests
/usr/lib/python3/dist-packages/pip/download.py:from pip._vendor import requests, six
/usr/lib/python3/dist-packages/pip/req/req_set.py:from pip._vendor import requests
/usr/lib/python3/dist-packages/requests/__init__.py:   >>> import requests
/usr/lib/python3/dist-packages/requests/adapters.py:      >>> import requests
/usr/lib/python3/dist-packages/requests/api.py:      >>> import requests
/usr/lib/python3/dist-packages/requests/models.py:      >>> import requests
/usr/lib/python3/dist-packages/requests/models.py:      >>> import requests
/usr/lib/python3/dist-packages/requests/sessions.py:      >>> import requests
/usr/lib/python3/dist-packages/requests_unixsocket/__init__.py:import requests
/usr/lib/python3/dist-packages/ssh_import_id/__init__.py:import requests
/usr/local/lib/python3.6/dist-packages/requests/__init__.py:   >>> import requests
/usr/local/lib/python3.6/dist-packages/requests/adapters.py:      >>> import requests
/usr/local/lib/python3.6/dist-packages/requests/api.py:      >>> import requests
/usr/local/lib/python3.6/dist-packages/requests/models.py:      >>> import requests
/usr/local/lib/python3.6/dist-packages/requests/models.py:      >>> import requests
/usr/local/lib/python3.6/dist-packages/requests/sessions.py:      >>> import requests
/usr/local/lib/python3.6/dist-packages/requests/status_codes.py:    >>> import requests

ssh_import_id is a little Ubuntu tool for adding keyfiles. Which requests does it use? Let’s edit the beginning of /usr/bin/ssh-import-id to read:

import argparse
import sys
from ssh_import_id import *

print(requests.__file__)

And we’ll find out:

$ ssh-import-id --help
/usr/local/lib/python3.6/dist-packages/requests/__init__.py
usage: ssh-import-id [-h] [-o FILE] [-r] [-u USERAGENT] USERID [USERID ...]

Authorize SSH public keys from trusted online identities.

positional arguments:
  USERID                User IDs to import

optional arguments:
  -h, --help            show this help message and exit
  -o FILE, --output FILE
                        Write output to file (default ~/.ssh/authorized_keys)
  -r, --remove          Remove a key from authorized keys file
  -u USERAGENT, --useragent USERAGENT
                        Append to the http user agent string

That’s the requests in /usr/local/lib, our new version. The result is the same if we’re root. Concerning.

Just to be safe, I modified a Python-using systemd service (cloud-init) to verify the same thing: at service startup, the new requests sneaks in.

Even though package manager files are not overwritten, system applications are using incorrect package versions.

The cause here is a bit complicated: sys.path shows where Python looks for packages first, and the site module determines how it gets there. Python has no way to specify versions of an import, so whatever folder is checked first takes priority.

Icing on the cake: security

In the nitty-gritty of a Python package, a little setup.py file describes what to install. Adding that package means executing a potentially untrustworthy setup.py: don’t do that with root access.

As a rule of thumb: Don’t touch your system’s Python install. But can you use it without modifying it?

https://leemendelowitz.github.io/blog/how-does-python-find-packages.html