Python

Debugging, profiling and testing

See Python debugging, profiling and testing.

ipython, the interactive python upgrade

The python-specific part of jupyter, which can also run without jupyter. Long story.

The main thing I forget here is…

def _figure_data(self, format):
    fig, ax = plt.subplots()
    ax.plot(self.data, 'o')
    ax.set_title(self._repr_latex_())
    data = print_figure(fig, format)
    # We MUST close the figure, otherwise
    # IPython’s display machinery
    # will pick it up and send it as output,
    # resulting in double display
    plt.close(fig)
    return data

# Here we define the special repr methods
# that provide the IPython display protocol
# Note that for the two figures, we cache
# the figure data once computed.

def _repr_png_(self):
    if self._png_data is None:
      self._png_data = self._figure_data('png')
    return self._png_data

from pprint import pprint, pformat
# display it
pprint(obj)
# get a formatted representation
pretty = pformat(obj)

Pro tip: dotenv

dotenv allows easy configuration through OS environment variables or text files in the parent directory. You should probably use this. PRO-TIP: there are lots of packages with similar names. Make sure you install using this one

pip install python-dotenv
%% or
conda install -c conda-forge python-dotenv

String formatting/interpolation

## float precision
>>> print("{:.2f}".format(3.1415926))
3.14
## left padding
>>> print("{:0>2d}".format(5))
05

## power tip which the manual does not make clear:
## variable formatting of variable formatting
>>> pi = 3.1415926
>>> precision = 4
>>> print( "{:.{}f}".format( pi, precision ) )
3.1415

>>> name = "Fred"
>>> f"He said his name is {name!r}."
"He said his name is 'Fred'."

from datetime import datetime
datetime.utcnow().isoformat(timespec='seconds')

from markdown import markdown
html_string = markdown("""
# Title

Paragraph. *emphasis*.
""")

Compiling and FFI

see Accelerating python.

Packaging and environments

See python packaging and environments.

Asynchrony in python

See asynchronous python.

functional-style programming

The expressive world of functional programming is often conceptually useful, and it can sometimes even be made performance in python.

toolz and the accelerated version cytoolz provide many useful functional abstractions which look fairly natural

Watching files for changes

Does this inotify solution work for non-Linux? Because macOS uses FSEvents and Windows uses I-don’t-even-know.

watchdog asserts that it is cross-platform. (source)

Python 2 vs 3

Are you old? New to python 3?

Sebastian Raschka, The key differences between Python 2.7.x and Python 3.x with examples

IDEs

I meant to list some, but in fact I only ever use VS Codium so that is the only current advice I would be able to give.

Typing

Python 3.6+ includes type hinting, and projects such as mypy support static analysis using type hints.¹ There were not many tutorials on the details of this at time of writing. Here’s one.

Short version: you go from this:

def fib(n):
  a, b = 0, 1
  while a < n:
    yield a
    a, b = b, a+b

to this:

def fib(n: int) -> Iterator[int]:
  a, b = 0, 1
  while a < n:
    yield a
    a, b = b, a+b

A version with actual corporate backing is pyright, which uses type checking to provide usablity improvements in VS Codium. Might be useful.

Command line parsers

Another bike-shedding danger zone is command-line parsing, leading to the need to spend too much time parsing command line parsers rather than parsing command lines.

• argparse is built-in to python stdlib and is OK, so why not just use that and avoid other dependencies? Answer: a dependency you might already have is likely to have introduced another CLI parsing library.

• “Hydra is a framework for elegantly configuring complex applications”

  Builds CLIs with autocomplete and other fun stuff.

• Google framework abseil has a python cli adapter whose selling points that

  1. Works across google ML apps
  2. integrates C++ arguments somehow?
  3. allows distributed definition of arguments rather than centralized.

• Click

  is a Python package for creating beautiful command line interfaces in a composable way with as little code as necessary. It’s the “Command Line Interface Creation Kit”. It’s highly configurable but comes with sensible defaults out of the box. […]

  • arbitrary nesting of commands
  • automatic help page generation
  • supports lazy loading of subcommands at runtime

  Aims to offer an alternative to the built-in argparse, which they regard as excessively magical. Its special feature is setuptools integration enabling installation of command-line tools from your current ipython virtualenv.

• Invoke

  provides a clean, high level API for running shell commands and defining/organizing task functions from a tasks.py file […] it offers advanced features as well — namespacing, task aliasing, before/after hooks, parallel execution and more.

• argh was/is a popular extension to argparse

  Argh is fully compatible with argparse. You can mix Argh-agnostic and Argh-aware code. Just keep in mind that the dispatcher does some extra work that a custom dispatcher may not do.

• clip.py comes with a passive-aggressive app name, (+1) is all about wrapping generic python commands in command-line applications easily, much like click.

Miscellaneous stuff I always need to look up

• Python exception hierarchy
• Python data structures

Progress meters

tqdm seems to be a de facto standard.

Also works in jupyter. Install it:

> pip install ipywidgets
> jupyter nbextension enable --py widgetsnbextension
> jupyter labextension install @jupyter-widgets/jupyterlab-manager

Use it:

%%capture
from tqdm import tqdm_notebook as tqdm
tqdm().pandas()

# Only this last bit is needed outside jupyter
with tqdm(total=len(my_list)) as pbar:
    for x in my_list:
        pbar.update(1)

Misc recommendations

• Pext is a python extension to script handy things in a handy GUI.
• Let’s get weird: Hy is a lisp that compiles to python syntax trees