Pretty display of objects

ipython display protocol

Check out the Rich display protocol which allows you to render objects as arbitrary graphics.

How to use this? The display api docs explain that you should basically implement methods such as _repr_svg_.

I made a thing called latex_fragment which leverages this to display arbitrary latex inline. This is how you use it:

def _figure_data(self, format):
   fig, ax = plt.subplots()
   ax.plot(self.data, 'o')
   ax.set_title(self._repr_latex_())
   data = print_figure(fig, format)
   # We MUST close the figure, otherwise IPython’s display machinery
   # will pick it up and send it as output, resulting in a double display
   plt.close(fig)
   return data

  # Here we define the special repr methods that provide the IPython display protocol
  # Note that for the two figures, we cache the figure data once computed.

  def _repr_png_(self):
   if self._png_data is None:
     self._png_data = self._figure_data('png')
   return self._png_data

For a non-graphical non-fancy terminal, you probably simply want nice formatting of dictionaries etc:

from pprint import pprint, pformat
pprint(obj)  # display it
print(pformat(obj))  # get a nicely formatted representation

Wait, you want to write your own pretty-printer, with correct indentation? Use tiles.

String formatting things that are unnecessarily hard to discern from the manual

What a nightmare is that manual for the string formatting. While all the information you need is in there, it is arranged in perverse inversion of some mixture of the frequency and the priority with which you use it. See Marcus Kazmierczak’s cookbook instead.

Highlights:

## float precision
>>> print("{:.2f}".format(3.1415926))
3.14
## left padding
>>> print("{:0>2d}".format(5))
05

## power tip which the manual does not make clear:
## variable formatting of variable formatting
>>> pi = 3.1415926
>>> precision = 4
>>> print( "{:.{}f}".format( pi, precision ) )
3.1415

f-strings

f-strings make things somewhat easier for python 3.6+ because they don’t need to mess around with naming things for the .format(foo=foo) call:

>>> name = "Fred"
>>> f"He said his name is {name!r}."
"He said his name is 'Fred'."

Timestamps

Why is a now timestamp in UTC not the first line in every academic research workbook/paper/data analysis? Because it’s tedious to look up the different bits.

Here you go:

from datetime import datetime
datetime.utcnow().isoformat(timespec='seconds')

Rendering HTML output

You have a quick and dirty chunk o’ HTML you need to output? You aren’t writing some damnable webapp with a nested hierarchy of template rendering and CSS integration into some design framework? You just want to pump out some markup?

I recommend yattag which is fast, simple, good and has a 1-page manual. It works real good.

Rendering markdown as HTML

from markdown import markdown
html_string = markdown("""
# Title

Paragraph. *emphasis*.
""")

Cython

Lowish-friction, well tested, well-document works everywhere that Cpython extensions can be compiled. Compiles most python code (apart from generators and inner functions). Optimises python code using type defs and extended syntax. Here, read Max Burstein’s intro.

Highlights: It works seamlessly with numpy. It makes calling C-code easy

Problems: No generic dispatch. Debugging is nasty, like debugging C with extra crap in your way.

numba

More specialised than cython, uses LLVM instead of the generic C compiler. Numba make optimising inner numeric loops easy.

Highlights: jit-compiles plain python, so it’s easy to use normal debuggers then switch on the compiler for performance improvements using the @jit Generic dispatch using the @generated_jit decorator. Compiles to multi-core vectorisations as well as CUDA. In principle this means you can do your calculations on the GPU.

Problems: LLVM is a shifty beast and sensitive version dependencies are annoying. Documentation is a bit crap, or at least unfriendly to outsiders. Practically, getting performance out of a GPU is trickier than working out you can optimise away one tedious matrix op, and doing it at this level is hard. There is too much messing with details of how many processors to allocate what to.

You might find it easier to use julia if a well-maintained and documented LLVM infrastructure is a real selling point for you.

General

• Least nerdview guide ever: Vicki Boykis, Alice in Python projectland.

• Simplest readable guide is python-packaging

• PyPI Quick and Dirty, includes some good tips such as using twine to make it automaticker.

• Open-sourcing a python project the right way.

• Official docs are no longer awful but are slightly stale, and especially perfunctory for compilation

• Kenneth Reitz shows rather than tells with a heavily documented setup.py

• Try Zed Shaw’s signature aggressively cynical and reasonably practical explanation of project structure, with bonus explication of how you should expect much time wasting yak shaving like this if you want to do software.

  • Or copy pyskel.
  • Or generate a project structure with cookiecutter.

Anaconda

The distribution you use if you want to teach a course in numerical python without dicking around with a 5 hour install process.

Download e.g. Linux x64 Miniconda, from the download page.

bash Miniconda3-latest-Linux-x86_64.sh
# login/logout here
conda config --set auto_activate_base false # don't be so aggressive conda
conda init fish  # fish users

Has a slightly different packaging workflow. See, e.g. Tim Hoppper’s workflow which explains this environment.yml malarkey, or the creators’ rationale.

The upshot for the end user is that if you want to install something with tricky dependencies like ViTables, you do this:

conda install pytables=3.2
conda install pyqt=4

Aside: Do you use fish shell? You need to do some extra setup. Specifically, add the line

source (conda info --root)/etc/fish/conf.d/conda.fish

into ~/.config/fish/config.fish.

NB Conda will fill up your hard disk if not regularly disciplined. via conda clean.

conda clean -pt

You might also want to not have the gigantic MKL library installed, of which I am in any case not a fan. You can usually disable it per request:

conda create -n pynomkl python nomkl

between the various versions and install MKL alone was using about 10GB total on my mac when I last checked Reducing the harm of this kind of nonsense is one reason I only ever install the minimalist miniconda as my base anaconda distribution, cautiously adding things as I need them.

Python environment/version management

venv is now a built-in python virtual environment system in python 3. It doesn’t support python 2 but fixes various problems, e.g. it supports framework python on macOS which is important for GUIs, and is covered by the python docs in the python virtual environment introduction. It has a higher-level, er, …wrapper (?) called pipenv.

# Create venv
python3 -m venv ~/.virtualenvs/learning_gamelan_keras_2
# Use venv from fish
source ~/.virtualenvs/learning_gamelan_keras_2/bin/activate.fish
# Use venv from bash
source ~/.virtualenvs/learning_gamelan_keras_2/bin/activate

Python environment management management

One suggestion I’ve has is to use pyenv. which eases and automates switching between all the other weird python environments created by virtualenv, python.org python, os python, anaconda python etc.

BUT WHO MANAGES THE VIRTUALENV MANAGER MANAGER?

Neat python 3 stuff

Alex Rogozhnikov, Python 3 with pleasure highlights some new tricks which landed recently.

Useful for us is a friendlier python struct-like thing, the data class Geir Arne Hjelle explains.

Override module accessors.

Asynchrony is less awful.

Python 2 and 3 compatibility

TL;DR: I am no employee of giant enterprise-type business with a gigantic legacy code base, and so I don’t use python 2. My code is not python 2 compatible. Python 3 is more productive, and no-one is paying me to be less productive right now. Python 2 code is usually easy to port to python 3. It is possible to write code which is compatible with python 2 and 3, but then I would miss out on some of the work that has gone into making python 3 easier and better, and waste time porting elegant easy python 3 things to hard boring python 2 things.

Writing python 2 and 3 compatible code.

🏗 six versus future.