Python packaging, environment and dependency management
“import antigravity” raises an exception if you want CUDA antigravity
2011-04-18 — 2026-10-23
In Which a Labyrinthine Struggle With 39 Disparate Python Environments Is Recounted, Leading to the Selection of Uv as a Singular Remedy Against the Security Vulnerabilities Once Incurred by an eMudhra Certificate Crisis.
On this page
How do I install the right versions of everything for some Python code I’m developing? How do I deploy it reliably? How do I share it with others? How do I minimise the tedious manual labour of doing the above?
As with many other vibrant, dynamic languages, maintaining consistent project dependencies in Python is a complicated mess. See also software package managers, software dependency managers and intractable standards battles…
Python is especially terrible for this—IMO, worse even than the notoriously bad node.js. While node.js has been criticized for settling on a bad package manager, Python settled on several bad package managers. It is confusing, chaotic, inconvenient and dangerous. In my experience, this fact is the single biggest barrier to entry for new Python users, both technically and socially, because all the systems are plagued by long-running arguments, some of which are highly personal. To use a language, you should not have to develop opinions about so many community disputes. Here is one angry overview of the situation.
Although Python is one of the world’s most popular programming languages, there are no easy answers to basic questions about how to install it, and sometimes even the hard answers are insane.
Harm reduction tips follow.
We no longer have the luxury of dabbling in a million different Python package managers. Now that we live in the world of constant security flaws, every unpatched packaging system is a potential attack vector, and every duplicate environment is another thing to patch when the CVEs rain down.
During the notorious eMudhra intermediate CA keys SNAFU, I lost many pointless hours to tracking down which of my Python environments were affected because I had so many bloody duplicates scattered across my system. The remediation list was long: (certifi in every environment, pip in every environment…). Which is to say, there were 39 separate Python environments I had to wrangle across seven different packaging systems. 39 that I found, that is to say. God knows what time bombs are still ticking away in the weird crevices of my filesystem. Conclusion: we are not in the world which allows such self-indulgent sloppiness.
Result: Now I go hard on uv and nothing else. It is not immune to security breaches, but having a single tool to manage everything reduces the attack surface and simplifies patching. One tool for Python versions, environments, command-line tools and lockfiles means my life is much easier. Bonus feature: uv patches other environments from outside, without running their own possibly-compromised pip:
1 Things we can safely ignore
In the before-times, many Python packaging standards existed. AFAICT, unless migrating extremely old code or performing digital archaeology, I should ignore everything about these.
The following are all deprecated or irrelevant: distutils, easy_install, virtualenv (superseded by venv)… Any HOWTOs that include them are probably not going to be useful.
Are you considering rye? Rye is kinda merging with uv so maybe you should consider uv instead.
Next decision point: Are you doing machine learning on the GPU? If so, go to the GPU quagmire section, because that is some stuff you cannot ignore.
If you are CUDA-free, some of the more horrible complexities can be ignored. If I were you, o blessed innocent, I would simply use uv if I want to be modern. I should be modern, indeed, because tradition has even less to recommend it than the awkward and confusing modernity.
That said, I would not mindlessly follow the latest fad because that strategy is dangerous too. If I followed the latest fad every time I started a new project, I would have migrated between conda, flit, poetry, uv, mamba, pipenv, etc., and would not have had any time to write code. If something is new, consider ignoring it for a year or two.
2 pip-like Systems
pip is the default Python package installer.
3 Conda-like systems
A parallel system to pip, which is a generalization and kinda-sorta compatible. It replaces many build difficulties with portability and intellectual-property difficulties. See conda-like python packaging.
4 Build systems
pyproject.toml now includes build system declarations. Python Packaging User Guide explains this:
Tools like pip and build do not actually convert your sources into a distribution package (like a wheel); that job is performed by a build backend. The build backend determines how your project will specify its configuration, including metadata (information about the project, for example, the name and tags that are displayed on PyPI) and input files. Build backends have different levels of functionality, such as whether they support building extension modules, and you should choose one that suits your needs and preferences.
You can choose from a number of backends; this tutorial uses Hatchling by default, but it will work identically with Setuptools, Flit, PDM, and others that support the
[project]table for metadata.
4.1 Setuptools.
Old. Popular for historical reasons.
4.2 Hatchling
The build backend from Hatch, seems to be the uv default right now:
4.3 Poetry’s build system
Poetry’s build system:**
4.4 PDM
PDM’s build system seems popular for folks with heavy C/C++ dependencies:
4.5 Flit core
Flit’s build system (Looks like it only handles pure Python?)
4.6 Maturin
Maturin seems to support Rust-backed packages, so it is very zeitgeisty.
5 Non-Python-specific dependency managers
Does Python’s slapstick ongoing shambles of a failed consensus on dependency management fill you with distrust? Do you have the vague feeling that perhaps we should use something else to manage Python since Python cannot manage itself? See generic dependency managers for an overview. Some of these are known to be an OK means of managing Python specifically, even though they are more general.
5.1 Spack
Supercomputing dependency manager spack has Python-specific support.
PyPI has hundreds of thousands of packages that are not yet in Spack, and
pipmay be a perfectly valid alternative to using Spack. The main advantage of Spack overpipis its ability to compile non-Python dependencies. It can also build cythonized versions of a package or link to an optimised BLAS/LAPACK library like MKL, resulting in calculations that run orders of magnitudes faster. Spack does not offer a significant advantage over other Python-management systems for installing and using tools like flake8 and sphinx. But if you need packages with non-Python dependencies like numpy and scipy, Spack will be very valuable to you.Anaconda is another great alternative to Spack and comes with its own
condapackage manager. Like Spack, Anaconda is capable of compiling non-Python dependencies. Anaconda contains many Python packages not yet in Spack, and Spack contains many Python packages not yet in Anaconda. The main advantage of Spack over Anaconda is its ability to choose a specific compiler and BLAS/LAPACK or MPI library. Spack also has better platform support for supercomputers and can build optimised binaries for your specific microarchitecture.
5.2 Meson
- meson-python uses The Meson Build system for Python. Is that… good?
6 Writing a package
Least nerdview guide: Vicki Boykis, Alice in Python projectland.
Simplest readable guide is python-packaging
PyPI Quick and Dirty, includes good tips for twine to make it more automatic.
Official docs are no longer awful but are slightly stale, and are especially perfunctory for compilation.
There is a community effort to document the issues of compiled packages in pypackaging-native (tldr it is hard)
Kenneth Reitz shows rather than tells with a heavily documented
setup.pyTry Zed Shaw’s signature aggressively cynical and reasonably practical explanation of project structure, with a bonus explication of why we should expect much time-wasting yak shaving if we want to build software.
- Or copy pyskel.
- Or generate a project structure with a templating/scaffolding system.
Updated: What the heck is
pyproject.toml?
6.1 Scaffolding and templating
Generating all those files is boring. The Python packaging ecosystem has several tools to automate this.
6.1.1 Copier
Slightly newer than cookiecutter, copier is a more modern and flexible tool for generating project templates.
6.2 Documenting my package
Here are two famous options. I’ve used Sphinx, and it is adequate and well-integrated, but has its own markup language which the world outside of Python does not use. MkDocs seems less blessed by the mainstream Python foundation but also uses markdown which is more widely used and has a rich ecosystem of tools.
7 Python versions
If we use conda or uv then the Python version is handled for us, along with generic dependency managers. With pip, we need to manage it ourselves. Poetry is in between — it knows about Python versions but cannot install Python for us, and as such is perpetually sad about its own inadequacy.
7.1 pyenv
I find pyenv baffling as it interacts with all the other tools in the Python packaging ecosystem in a way that is not immediately obvious to me.
I prefer to avoid it completely and let uv handle Python versions, which it does in a relatively seamless way, without me needing to consider local and global versions and remember which sub-sub-version of Python I compiled with what for which and whom, moreover why.
pyenv is the core tool of an ecosystem that eases and automates switching between Python versions. It manages Python and thus can be used as a manager for all the other managers.
BUT WHO MANAGES THE VIRTUALENV MANAGER MANAGER? Also, what is going on in this ecosystem of parts? Logan Jones explains:
- pyenv manages multiple versions of Python itself.
- virtualenv/venv manage virtual environments for a specific Python version.
- pyenv-virtualenv manages virtual environments across varying Python versions.
Anyway, pyenv compiles a custom version of Python and stays extremely isolated from everything else. Weights and Biases’ Intro to Pyenv for Machine Learning helps.
Of course, because this is adjacent to Python packaging, it’s infected with the same brainworms. Everything immediately becomes complicated and confusing when I try to interact with the rest of the ecosystem, e.g.,
pyenv-virtualenvwrapper is different from
pyenv-virtualenv, which provides extended commands likepyenv virtualenv 3.4.1 project_nameto directly help out with managing virtualenvs.pyenv-virtualenvwrapperhelps in interacting withvirtualenvwrapper, butpyenv-virtualenvprovides more convenient commands, where virtualenvs are first-class pyenv versions, that can be (de)activated. That’s to say,pyenvandvirtualenvwrapperare still separated whilepyenv-virtualenvis a nice combination.
Huh. I am already too bored to think. However, I worked out a command that installed a pyenv TensorFlow with an isolated virtualenv:
For fish shell I needed to add some special lines to config.fish:
set -x PYENV_ROOT $HOME/.pyenv
set -x PATH $PYENV_ROOT/bin $PATH
## fish <3.1
# status --is-interactive; and . (pyenv init -|psub)
# status --is-interactive; and . (pyenv virtualenv-init -|psub)
## fish >=3.1
status --is-interactive; and pyenv init - | source
status --is-interactive; and pyenv virtualenv-init - | sourceFor bash/zsh (resp. .bashrc/.zshrc) it is as follows:
8 Sorry, GPU/TPU/etc users
Users of GPUs must ignore any other options, no matter how attractive all the other options might seem at first glance. The stupid drudge work of venv is the price of hardware support for now. Only pip and conda support hardware specification in practice.
UPDATE: poetry now supports Pytorch with CUDA. uv has had a crack at it too.
Terminology you need to learn: Many packages specify local versions for particular architectures as part of their functionality. For example, pytorch comes in various flavours, which, when using pip, we can select in the following fashion:
The local version is given by the +cpu or +cu113 bit, which changes which code we execute when using these packages. Specifying a GPU version is essential for many machine learning projects (essential, that is, if I don’t want my code to run orders of magnitude slower). The details of how we control this within the Python packaging ecosystem are somewhat contentious and complicated; consequently, the new wave of options like poetry or pipenv do not support it. Brian Wilson argues,
During my dive into the open-source abyss that is ML packages and
+localVersionsI discovered lots of people have strong opinions about what it should not be and like to tell other people they’re wrong. Other people with opinions about what it could be are too afraid of voicing them lest there be some unintended consequence. PSF has asserted what they believe to be the intended state in PEP-440 (no local versions published) but the solution (PEP-459) is not an ML Model friendly solution because the installation providers (pip, pipenv, poetry) don’t have enough standardised hooks into the underlying hardware (cpu vs gpu vs cuda lib stack) to even understand which version to pull, let alone the Herculean effort it would take to get even just pytorch to update their package metadata.
There is no evidence that this logjam will resolve any time soon. However, it turns out that machine learning isn’t going away, and ML projects use GPUs. It turns out that packaging projects with GPU code is hard. Since I do neural network work and thus use GPU/CPU versions of packages, I can effectively ignore most of the Python environment alternatives on this page. The two that work are conda and pip. These all support a minimum viable local version package system de facto which does what we want. If you want something fancier, try containerization using a GPU-compatible system such as apptainer.
