Python packaging, dependency management and isolation

Sorry, GPU/TPU/etc users

tl;dr: only pip and conda support hardware specification in practice. Users of GPUs must ignore any other options, no matter how attractive all the other options might seem at first glance.

Many packages specify local versions for particular architectures as a part of their functionality. For example, pytorch comes in various flavours, which when using pip

# CPU flavour
pip install torch==1.10.0+cpu -f https://download.pytorch.org/whl/cpu/torch_stable.html
# GPU flavour
pip install torch==1.10.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html

The local version is given by the +cpu or +cu113 bit, and it changes what code will be executed when using these packages. Specifying a GPU version is essential for many machine learning projects (essential, that is, if I do not want my code to run orders of magnitude slower). The details of how this can be controlled with regard to the python packaging ecosystem are somewhat contentious and complicated and thus not supported by any of the new wave options like poetry or pipenv. Brian Wilson argues,

During my dive into the open-source abyss that is ML packages and +localVersions I discovered lots of people have strong opinions about what it should not be and like to tell other people they're wrong. Other people with opinions about what it could be are too afraid of voicing them lest there be some unintended consequence. PSF has asserted what they believe to be the intended state in PEP-440 (no local versions published) but the solution (PEP-459) is not an ML Model friendly solution because the installation providers (pip, pipenv, poetry) don’t have enough standardized hooks into the underlying hardware (cpu vs gpu vs cuda lib stack) to even understand which version to pull, let alone the Herculean effort it would take to get even just pytorch to update their package metadata.

There is no evidence that this logjam will resolve any time soon. It turns out that Packaging projects with GPU code is hard. Since I do neural network stuff and thus use GPU/CPU version of packages, this means that I can effectively ignore most of the python environment alternatives on this page. The two that work are conda and pip, which support a minimum viable local version package system de facto, and if they are less smooth or pleasant than the new systems, at least I am not alone.

Writing a package

• Least nerdview guide: Vicki Boykis, Alice in Python projectland.

• Simplest readable guide is python-packaging

• PyPI Quick and Dirty, includes some good tips such as using twine to make it automaticker.

• Open-sourcing a python project the right way.

• Official docs are no longer awful but are slightly stale, and especially perfunctory for compilation

• There is a community effort to document the issues of compiled packages in pypackaging-native (tldr it is hard)

• Kenneth Reitz shows rather than tells with a heavily documented setup.py

• Try Zed Shaw’s signature aggressively cynical and reasonably practical explanation of project structure, with bonus explication of how you should expect much time wasting yak shaving like this if you want to do software.

  • Or copy pyskel.
  • Or generate a project structure with cookiecutter.

• updated: What the heck is pyproject.toml?

pip

The default python package installer. It is best spelled as

python -m pip install package_name

To snapshot dependencies:

python -m pip freeze > requirements.txt

To restore dependencies:

python -m pip install -r requirements.txt

venv works. It is a good default choice, widely supported and adequate, if not awesome, workflow.

Versions look a little like this:

SomeProject
SomeProject == 1.3
SomeProject >= 1.2, < 2.0
SomeProject[foo, bar]
SomeProject ~= 1.4.2 
SomeProject == 5.4 ; python_version < '3.8'
SomeProject ; sys_platform == 'win32'
requests [security] >= 2.8.1, == 2.8.* ; python_version < "2.7"

The ~= is a handy lazy first-stop; it permits point releases, but not minor releases, so e.g. ~=1.3.0 will also satisfy itself with version 1.3.9.

conda

A parallel system to pip, designed to do all the work of installing especially python software with hefty compiled dependencies.

There are two parts here with two separate licenses

1. the anaconda python distribution
2. the conda python package manager.

I am slightly confused about how these two relate (Can I install a non-anaconda python distribution through the conda package manager?) There distinction is important, since licensing anaconda can be expensive. See, e.g.

• Anaconda is not free for commercial use (anymore) so what are the alternatives?
• Conda/Anaconda no longer free to use?
• See also mamba below, which aims to reduce licensing risk by reimplementing the more licensing-vulnerable parts of the anaconda ecosystem

Some things that are (or were?) painful to install by pip are painless via conda. Contrarywise, some things that are painful to install by conda are easy by pip.

I recommend working out which pain points are worse in this complicated decision by trial and error. Sometimes it would be worth the administrative burden of understanding conda’s current licensing and future licensing risks, but if it does not bring substantial value, choose pip.

This is an updated recommendation; previously I preferred conda — pip used to be much worse, and anaconda’s licensing used to be less restrictive. Now I think anaconda cannot be relied upon IP-wise.

bash Miniconda3-latest-Linux-x86_64.sh
# login/logout here
# or do something like `exec bash -` if you are fancy
# Less aggressive conda
conda config --set auto_activate_base false
# conda for fish users
conda init fish

curl -L -O https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-$(uname)-$(uname -m).sh
bash Mambaforge-$(uname)-$(uname -m).sh

conda install pytables=3.2
conda install pyqt=4

source (conda info --root)/etc/fish/conf.d/conda.fish

conda init fish

conda install nb_conda_kernels

conda env export > environment.yml

conda env create --file environment.yml

conda clean --all --yes

conda config --add pkgs_dirs /SOME/OTHER/PATH/.conda

chmod a-rwx ~/.conda

conda create -n pynomkl python nomkl

conda config --set env_prompt '({name})'
conda env create --prefix ./env/myenv --file environment_linux.yml
conda activate ./env/myenv

conda config --set env_prompt '\({name}\)'

env_prompt: ({name})

venv

venv is now a built-in python virtual environment system in python 3. It doesn’t support python 2 but fixes various problems, e.g. it supports framework python on macOS which is important for GUIs, and is covered by the python docs in the python virtual environment introduction.

# Create venv
python3 -m venv ./venv --prompt some_arbitrary_name
# or if we want to use system packages:
python3 -m venv ./venv --prompt some_arbitrary_name --system-site-packages
# Use venv from fish OR
source ./venv/bin/activate.fish
# Use venv from bash
source ./venv/bin/activate

pyenv

pyenv is the core tool of an ecosystem which eases and automates switching between python versions. Manages python and thus implicitly can be used as a manager for all the other managers. The new new hipness, at least for platforms other than windows, where it does not work.

BUT WHO MANAGES THE VIRTUALENV MANAGER MANAGER? Also, what is going on in this ecosystem of bits? Logan Jones explains:

• pyenv manages multiple versions of Python itself.
• virtualenv/venv manages virtual environments for a specific Python version.
• pyenv-virtualenv manages virtual environments for across varying versions of Python.

Anyway, pyenv compiles a custom version of python and as such is extremely isolated from everything else. An introduction with emphasis on my area: Intro to Pyenv for Machine Learning.

#initial pyenv install
pyenv init
# install a specific python version
pyenv install 3.8.13
# ensure we can find that version
pyenv rehash
# switch to that version
pyenv shell 3.8.13

Of course, because this is adjacent to the python packaging ecosystem, it immediately becomes complicated and confusing when you try to interact with the rest of the ecosystem, e.g.,

pyenv-virtualenvwrapper is different from pyenv-virtualenv, which provides extended commands like pyenv virtualenv 3.4.1 project_name to directly help out with managing virtualenvs. pyenv-virtualenvwrapper helps in interacting with virtualenvwrapper, but pyenv-virtualenv provides more convenient commands, where virtualenvs are first-class pyenv versions, that can be (de)activated. That’s to say, pyenv and virtualenvwrapper are still separated while pyenv-virtualenv is a nice combination.

Huh. I am already too bored to think. However, I did work out a command which installed a pyenv tensorflow with an isolated virtualenv:

brew install pyenv pyenv-virtualenv
pyenv install 3.8.6
pyenv virtualenv 3.8.6 tf2.4
pyenv activate tf2.4
pip install --upgrade pip wheel
pip install 'tensorflow-probability>=0.12' 'tensorflow<2.5' jupyter

For fish shell you need to add some special lines to config.fish:

set -x PYENV_ROOT $HOME/.pyenv
set -x PATH $PYENV_ROOT/bin $PATH
## fish <3.1
# status --is-interactive; and . (pyenv init -|psub)
# status --is-interactive; and . (pyenv virtualenv-init -|psub)
## fish >=3.1
status --is-interactive; and pyenv init - | source
status --is-interactive; and pyenv virtualenv-init - | source

Poetry

No! wait! The new new new hipness is poetry. All the other previous hipnesses were not the real eternal ultimate hipness that transcends time. I know we said this every previous time, but this time its real and our love will last forever ONO.

⛔️⛔️UPDATE⛔️⛔️: OK, turns out this love was not actually quite as eternal as it seemed. Lovely elegant design does not make up for the fact that the project is logjammed and broken in various ongoing ways; see Issue #4595: Governance—or, “what do we do with all these pull requests?”. It might be usable if your needs are modest or you are prepared to jump into the project discord, which seems to be where the poetry hobbyists organise, but since I want to use this project merely incidentally, as a tool to develop something else, hobbyist level of engagement is not something I can participate in. poetry is not ready for prime-time.

Note also that poetry is having difficulty staying current with local versions, as made famous by CUDA-supporting packages. There is an example of the kind of antics that make it work below.

Poetry is a tool for dependency management and packaging in Python. It allows you to declare the libraries your project depends on and it will manage (install/update) them for you.

From the introduction:

Packaging systems and dependency management in Python are rather convoluted and hard to understand for newcomers. Even for seasoned developers it might be cumbersome at times to create all files needed in a Python project: setup.py, requirements.txt, setup.cfg, MANIFEST.in and the newly added Pipfile.

So I wanted a tool that would limit everything to a single configuration file to do: dependency management, packaging and publishing.

It takes inspiration in tools that exist in other languages, like composer (PHP) or cargo (Rust).

And, finally, I started poetry to bring another exhaustive dependency resolver to the Python community apart from Conda’s.

What about Pipenv?

In short: I do not like the CLI it provides, or some of the decisions made, and I think we can make a better and more intuitive one.

Editorial side-note: Low-key dissing on similarly-dysfunctional, competing projects is an important part of python packaging.

Lazy install is via this terrifying command line (do not run if you do not know what this does):

curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.py | python -

Poetry could be regarded as a similar thing to pipenv, in that it (per default, but not necessarily) manages the dependencies in a local venv. It has a much more full-service approach than systems built on pip. For example, it has its own dependency resolver, which makes use of modern dependency metadata but also will work with previous dependency specifications by brute force if needed. It separates specified dependencies from the ones that it contingently resolves in practice, which means that the dependencies seem to transport much better than conda, which generally requires you to hand-maintain a special dependency file full of just the stuff you actually wanted. In practice the many small conveniences and thoughtful workflow are helpful. For example, it will set up the current package for developing per default so that imports work as similarly as possible across this local environment and when it is distributed to users.

Recommended config:

poetry config virtualenvs.create  true
poetry config virtualenvs.in-project true  # local venvs are easier for my brain.

The cache gets corrupted, giving errors about hashes:

rm -r ~/Library/Caches/pypoetry/cache
rm -r ~/Library/Caches/pypoetry/artifacts

However, poetry does not support installing build variants/profiles, which means I cannot install GPU software, so it is useless to me.

[tool.poetry.dependencies]
python = "^3.10"
numpy = "^1.23.2"
torch = { version = "1.12.1", source="torch"}
torchaudio = { version = "0.12.1", source="torch"}
torchvision = { version = "0.13.1", source="torch"}

[[tool.poetry.source]]
name = "torch"
url = "https://download.pytorch.org/whl/cu116"
secondary = true

pipenv

⛔️⛔️UPDATE⛔️⛔️: Note that the pipenv system does not support “local versions” and thus in practice cannot be used for machine learning applications. This project is dead to me. (Bear in mind that my opinions about will become increasingly outdated depending on when you read this.)

venv has a higher-level, er, …wrapper (?) interface (?) called pipenv.

Pipenv is a production-ready tool that aims to bring the best of all packaging worlds to the Python world. It harnesses Pipfile, pip, and virtualenv into one single command.

I switched to pipenv from poetry because it looked like it might be less chaos than poetry. I think it is, although the race is close.

HOWEVER, it is still pretty awful. TBH, I would just use plain pip and requirements.txt which, while it is primitive and broken, is at least broken and primitive in a well-understood way.

At time of writing the pipenv website was 3 weeks into an outage, because dependency management is a quagmire of sadness and comically broken management with terrible Bus factor. However, the backup docs site is semi-functional, albeit too curt to be useful and AFAICT outdated. The documentation site inside github is readable.

The dependency resolver is, as the poetry devs point out, broken in its own special ways. The procedure to install modern ML frameworks, for example, is gruelling.

An introduction showing pipenv and venv used together.

For my configuration, important configuration settings are

export WORKON_HOME=~/.venvs

To get the venv inside the project (required for sanity in my HPC) I need the following:

export PIPENV_VENV_IN_PROJECT=1

Pipenv will automatically load dotenv files which is a nice touch.

Generic dependency managers

Does python’s slapstick ongoing shambles of a failed consensus on dependency management system fill you with distrust? Do you have the vague feeling that perhaps you should use something else to manage python since python cannot manage itself? See generic dependency managers for an overview

Other packaging systems

Look at all these other package/build systems, which I only just now noticed! How interoperable are they?

Documenting my package

Here are two famous options.

• Welcome — Sphinx documentation
• Getting Started - MkDocs