Python packaging and environments

General

• Least nerdview guide: Vicki Boykis, Alice in Python projectland.

• Simplest readable guide is python-packaging

• PyPI Quick and Dirty, includes some good tips such as using twine to make it automaticker.

• Open-sourcing a python project the right way.

• Official docs are no longer awful but are slightly stale, and especially perfunctory for compilation

• Kenneth Reitz shows rather than tells with a heavily documented setup.py

• Try Zed Shaw’s signature aggressively cynical and reasonably practical explanation of project structure, with bonus explication of how you should expect much time wasting yak shaving like this if you want to do software.

  • Or copy pyskel.
  • Or generate a project structure with cookiecutter.

Anaconda

The distribution you use if you want to teach a course in numerical python without dicking around with a 5 hour install process.

Download e.g. Linux x64 Miniconda, from the download page.

bash Miniconda3-latest-Linux-x86_64.sh
# login/logout here
#
# Less aggressive conda
conda config --set auto_activate_base false
# conda for fish users
conda init fish

Has a slightly different packaging workflow. See, e.g. Tim Hoppper’s workflow which explains this environment.yml malarkey, or the creators’ rationale and manual.

The upshot for the end user is that if you want to install something with tricky dependencies like ViTables, you do this:

conda install pytables=3.2
conda install pyqt=4

Aside: Do you use fish shell? You need to do some extra setup. Specifically, add the line

source (conda info --root)/etc/fish/conf.d/conda.fish

into ~/.config/fish/config.fish.

NB Conda will fill up your hard disk if not regularly disciplined. via conda clean.

conda clean -pt

One exports the current conda environment config, by convention, into environment.yml.

conda env export > environment.yml

conda env create -f environment.yml

You might also want to not have the gigantic MKL library installed, of which I am in any case not a fan. You can usually disable it per request:

conda create -n pynomkl python nomkl

between the various versions and install MKL alone was using about 10GB total on my mac when I last checked Reducing the harm of this kind of nonsense is one reason I only ever install the minimalist miniconda as my base anaconda distribution, cautiously adding things as I need them.

venv

venv is now a built-in python virtual environment system in python 3. It doesn’t support python 2 but fixes various problems, e.g. it supports framework python on macOS which is important for GUIs, and is covered by the python docs in the python virtual environment introduction. It has a higher-level, er, …wrapper (?) called pipenv.

# Create venv
python3 -m venv ~/.virtualenvs/learning_gamelan_keras_2
# Use venv from fish
source ~/.virtualenvs/learning_gamelan_keras_2/bin/activate.fish
# Use venv from bash
source ~/.virtualenvs/learning_gamelan_keras_2/bin/activate

Python environment management management

One suggestion I’ve has is to use pyenv. which eases and automates switching between all the other python environments created by virtualenv, python.org python, os python, anaconda python etc.

BUT WHO MANAGES THE VIRTUALENV MANAGER MANAGER?