Python

Debugging, profiling and testing

See Python debugging, profiling and testing.

Idiom

Python people set great value on mastering python idiom. For example, python idiom for accessing a nullable member is a and a.x Python idiom for ‘python person’ is pythonista. Python idiom for ‘idiomatic’ is pythonic. Much has been written on how to be pythonic. But I think learning requires also negative examples. Here, may I proffer to you John Burkardt’s Hypershere integrals? That is an example of the least idiomatic python code I have seen that was not in an obfuscated code contest.

ipython

The interactive python upgrade The python-specific part of jupyter, which can also run without jupyter. Long story. But think of it as a REPL, a CLI-style execution environment, that is a little friendlier and has colourisation and autocomplete and such.

(ipython locate)/profile_default/startup/

conda install jedi==0.17.2

def _figure_data(self, format):
    fig, ax = plt.subplots()
    ax.plot(self.data, 'o')
    ax.set_title(self._repr_latex_())
    data = print_figure(fig, format)
    # We MUST close the figure, otherwise
    # IPython’s display machinery
    # will pick it up and send it as output,
    # resulting in double display
    plt.close(fig)
    return data

# Here we define the special repr methods
# that provide the IPython display protocol
# Note that for the two figures, we cache
# the figure data once computed.

def _repr_png_(self):
    if self._png_data is None:
      self._png_data = self._figure_data('png')
    return self._png_data

from pprint import pprint, pformat
# display it
pprint(obj)
# get a formatted representation
pretty = pformat(obj)

Pro tip: store condig in .env

There are many, many python tools to load environment variables from local files. Confusingly this is usually called “Twelve-Factor App” config. I am doing my best to never google that.

One I have used per default is dotenv. I will describe it for congcreteness. dotenv allows easy configuration through OS environment variables or text files in the parent directory. Everyone should probably use this. PRO-TIP: there are lots of packages with similar names. Make sure you install using this one.

pip install python-dotenv
%% or
conda install -c conda-forge python-dotenv

Also very similar, henriquebastos/python-decouple. sloria/environs: simplified environment variable parsing Dynaconf is more sophisticated, and comes closer to a full configuration system like hydra.

Let us imagine we are using basic dotenv for now. Then we can be indifferent to whether files came from an FS config or an environment variable.

from dotenv import load_dotenv
load_dotenv()  # take environment variables from .env.
# Code of your application, which uses environment variables (e.g. from `os.environ` or
# `os.getenv`) as if they came from the actual environment.

The library rich includes a colorized pretty printer.

String formatting/interpolation

## float precision
>>> print("{:.2f}".format(3.1415926))
3.14
## left padding
>>> print("{:0>2d}".format(5))
05

## power tip which the manual does not make clear:
## variable formatting of variable formatting
>>> pi = 3.1415926
>>> precision = 4
>>> print( "{:.{}f}".format( pi, precision ) )
3.1415

>>> name = "Fred"
>>> f"He said his name is {name!r}."
"He said his name is 'Fred'."

from datetime import datetime
datetime.utcnow().isoformat(timespec='seconds')

from markdown import markdown
html_string = markdown("""
# Title

Paragraph. *emphasis*.
""")

Compiling and FFI

see Accelerating python.

Packaging and environments

See python packaging and environments.

Asynchrony in python

See asynchronous python.

functional-style programming

The style functional programming is often conceptually useful, and it can sometimes even be made performant in python.

toolz and the accelerated version cytoolz provide many useful functional abstractions which look fairly natural

ExitStack

If you are acquiring several context managers, there is a modern handy idion called ExitStack. For an explanation see On the Beauty of Python’s ExitStack by Nikolaus Rath.

with ExitStack() as cm:
    res1 = acquire_resource_one()
    cm.callback(release_resource, res1)
    # do stuff with res1
    res2 = acquire_resource_two()
    cm.callback(release_resource, res2)
    # do stuff with res1 and res2

Watching files for changes

Does this inotify solution work for non-Linux? Because macOS uses FSEvents and Windows uses I-don’t-even-know.

watchdog asserts that it is cross-platform (source).

emcrisostomo/fswatchclaims to be a universal filewatcher. There is a python frontend.

Python 2 vs 3

Are you old? New to python 3?

Sebastian Raschka, The key differences between Python 2.7.x and Python 3.x with examples

IDEs

I meant to list some, but in fact I only ever use VS Codium so that is the only current advice I would be able to give.

Typing

Python 3.6+ includes type hinting, and projects such as mypy support static analysis using type hints.¹ There were not many tutorials on the details of this at time of writing. Here’s one.

Short version: you go from this:

def fib(n):
  a, b = 0, 1
  while a < n:
    yield a
    a, b = b, a+b

to this:

def fib(n: int) -> Iterator[int]:
  a, b = 0, 1
  while a < n:
    yield a
    a, b = b, a+b

A version with actual corporate backing is pyright, which uses type checking to provide usability improvements in VS Codium. Might be useful.

Command line parsers

I am furious at how many options there are to build python CLI scripts. This is the new contender for most bike-shedded function. AFAICT if you write some python software and don’t write a new command-line parsing library then you will be regarded as somehow feckless. If you are indeed so dissolute as to not make your own quirky artisinal command-line parser, you could try the listing at python CLIs.

Miscellaneous stuff I always need to look up

• Python exception hierarchy
• Python data structures

Progress meters

tqdm seems to be a de facto standard.

Also works in jupyter. Install it:

> pip install ipywidgets
> jupyter nbextension enable --py widgetsnbextension
> jupyter labextension install @jupyter-widgets/jupyterlab-manager

Use it:

%%capture
from tqdm import tqdm_notebook as tqdm
tqdm().pandas()

# Only this last bit is needed outside jupyter
with tqdm(total=len(my_list)) as pbar:
    for x in my_list:
        pbar.update(1)

Misc recommendations

• Pext is a python extension to script interactive CLI processes things in a handy GUI.
• Let’s get weird: Hy is a lisp that compiles to python syntax trees