How do you install the right versions of everything for some python code you are developing? How to deploy that sustainably? How to share it with others.
Not so hard, but confusing and chaotic due to many long-running disputes only lately resolving.
Least nerdview guide: Vicki Boykis, Alice in Python projectland.
Simplest readable guide is python-packaging
Kenneth Reitz shows rather than tells with a heavily documented setup.py
Try Zed Shaw’s signature aggressively cynical and reasonably practical explanation of project structure, with bonus explication of how you should expect much time wasting yak shaving like this if you want to do software.
The default python package installer. It is best spelled
python -m pip install package_name
Pro tip: pipx:
pip is a general-purpose package installer for both libraries and apps with no environment isolation. pipx is made specifically for application installation, as it adds isolation yet still makes the apps available in your shell: pipx creates an isolated environment for each application and its associated packages.
The distribution you use if you want to teach a course in numerical python without dicking around with a 5 hour install process.
bash Miniconda3-latest-Linux-x86_64.sh # login/logout here # # Less aggressive conda conda config --set auto_activate_base false # conda for fish users conda init fish
The upshot for the end user is that if you want to install something with tricky dependencies like ViTables, you do this:
conda install pytables=3.2 conda install pyqt=4
source (conda info --root)/etc/fish/conf.d/conda.fish
NB Conda will fill up your hard disk if not regularly disciplined. via conda clean.
conda clean -pt
One exports the current conda environment config, by convention, into
conda env export > environment.yml
conda env create -f environment.yml
conda create -n pynomkl python nomkl
between the various versions and install MKL alone was using about 10GB total on my mac when I last checked Reducing the harm of this kind of nonsense is one reason I only ever install the minimalist miniconda as my base anaconda distribution, cautiously adding things as I need them.
venv is now a built-in python virtual environment system in python 3. It doesn’t support python 2 but fixes various problems, e.g. it supports framework python on macOS which is important for GUIs, and is covered by the python docs in the python virtual environment introduction. It has a higher-level, er, …wrapper (?) called pipenv.
# Create venv python3 -m venv ~/.virtualenvs/learning_gamelan_keras_2 # Use venv from fish source ~/.virtualenvs/learning_gamelan_keras_2/bin/activate.fish # Use venv from bash source ~/.virtualenvs/learning_gamelan_keras_2/bin/activate
Python environment management management
One suggestion I’ve heard is to use pyenv. which eases and automates switching between all the other python environments created by virtualenv, python.org python, os python, anaconda python etc.
BUT WHO MANAGES THE VIRTUALENV MANAGER MANAGER? What is going on?
- pyenv manages multiple versions of Python itself.
- virtualenv/venv manages virtual environments for a specific Python version.
- pyenv-virtualenv manages virtual environments for across varying versions of Python.
Anyway, pyenv compiles a custom version of python and as such is extremely isolated from everything else. Here is an introduction with emphasis on my area: Intro to Pyenv for Machine Learning.
Of course, because this is a python packaging solution, it immediately becomes complicated and confusing when you try to interact with the rest of the ecosystem.
Attention: This plugin is different from
pyenv-virtualenv, which provides extended commands like
pyenv virtualenv 3.4.1 project_nameto directly help out with managing virtualenvs.
pyenv-virtualenvwrapperhelps in interacting with
pyenv-virtualenv providesmore convenient commands, where virtualenvs are first-class pyenv versions, that can be (de)activated. That’s to say,
virtualenvwrapperare still separated while
pyenv-virtualenvis a nice combination.
Huh. I am already too bored to think.
However, I did nut otu a command which installed a pyenv tensorflow with an isolate virtualenv:
brew install pyenv pyenv-virtualenv pyenv install 3.8.6 pyenv virtualenv 3.8.6 tf2.4 pyenv activate tf2.4 pip install --upgrade pip wheel pip install 'tensorflow-probability>=0.12' 'tensorflow<2.5' jupyter
set -x PYENV_ROOT $HOME/.pyenv set -x PATH $PYENV_ROOT/bin $PATH ## fish <3.1 # status --is-interactive; and . (pyenv init -|psub) # status --is-interactive; and . (pyenv virtualenv-init -|psub) ## fish >=3.1 status --is-interactive; and pyenv init - | source status --is-interactive; and pyenv virtualenv-init - | source