Python

1 Debugging, profiling and testing

See Python debugging, profiling and testing.

2 Linting and formatting

There are many linters. Here are some I have used.

• ruff is my favourite. The defaults are good and it is fast. Downside: one needs to install an additional binary to use it. However, it pip installs easily.
• Pylint whose defaults I like, but it is slow.
• flake8 which is too alarmist per default (CRITICAL RED ALERT: SPLIT INFINITIVE) but is adequately fast.

I briefly experimented with writing custom rules for my linter, but decided that this is something that only makes sense at a corporate scale, and otherwise is yak shaving in the derogatory sense.

• How to Write a Checker - Pylint
• How to add custom Flake8 rules?
• Writing Plugins for Flake8

3 Pyodide

Python from the browser, including mathematical libraries.

• Documentation
• Pyodide blog
• Starboard, a shareable in-browser notebook that runs python (!)

HT Emma Krantz for showing me this while solving my web interactive maths problems for me.

4 Idiom

Python people set great value on mastering python idiom. For example, python idiom for accessing a nullable member is a and a.x. I never remember this and always write getattr(a, 'x', None). Python idiom for ‘python person’ is pythonista. Python idiom for ‘idiomatic’ is pythonic. Much has been written on how to be pythonic.

I think learning can benefit also from negative examples. Here, may I proffer to you John Burkardt’s Hypershere integrals? That the least idiomatic python code I have seen. It is a line-for-line translation of C code into slow, verbose python. If you are going to do that, write C code.

Python is a moving target. Here is an argument that the classic __init__ methods are obsolete. Here is a class in the new style:

from typing import Callable
from typing import NewType

FileDescriptor = NewType("FileDescriptor", int)

_open: Callable[[str], FileDescriptor] = fileio.open  # type:ignore[assignment]
_read: Callable[[FileDescriptor, int], bytes] = fileio.read
_close: Callable[[FileDescriptor], None] = fileio.close

from typing import Self

@dataclass
class FileReader:
    _fd: FileDescriptor
    @classmethod
    def open(cls, path: str) -> Self:
        return cls(_open(path))
    def read(self, length: int) -> bytes:
        return _read(self._fd, length)
    def close(self) -> None:
        _close(self._fd)

All of this makes me feel that OOP itself is a bad idea.

5 ipython

The interactive python upgrade The python-specific part of jupyter, which can also run without jupyter. Long story. But think of it as a REPL, a CLI-style execution environment, that is a little friendlier and has colourisation and autocomplete and such. If sounds useful, read on

6 Pro tip: store config in .env

There are many, many python tools to load environment variables from local files. See .env..

7 Pretty printing

For a non-graphical, non-fancy terminal, the most common useful rich display probably simply wants nice formatting of big data structures.

from pprint import pprint, pformat
# display it
pprint(obj)
# get a formatted representation
pretty = pformat(obj)

• If I want to write my own pretty-printer, with correct indentation, I might use tiles. This has its own extension protocol.
• The library rich includes a colorized pretty printer.

8 String formatting/interpolation

## float precision
>>> print("{:.2f}".format(3.1415926))
3.14
## left padding
>>> print("{:0>2d}".format(5))
05

## power tip which the manual does not make clear:
## variable formatting of variable formatting
>>> pi = 3.1415926
>>> precision = 4
>>> print( "{:.{}f}".format( pi, precision ) )
3.1415

>>> name = "Fred"
>>> f"He said his name is {name!r}."
"He said his name is 'Fred'."

from datetime import datetime
datetime.utcnow().isoformat(timespec='seconds')

datetime.datetime.now().strftime("%Y%m%d-%H%M%S"))

from markdown import markdown
html_string = markdown("""
# Title

Paragraph. *emphasis*.
""")

9 Compiling and FFI

See Accelerating python.

10 Packaging and environments

See python packaging and environments.

11 Asynchrony in python

See asynchronous python.

12 functional-style programming

The style functional programming is often conceptually useful, and it can sometimes even be made performant in python. It rapidly gets clunky because the type system is bad.

toolz and the accelerated version cytoolz provide many useful functional abstractions which look fairly natural

13 IDEs

I meant to list some, but in fact I only ever use VS Codium so that is the only current advice I would be able to give.

PyCharm also looks very good but I am hopelessly out of date on it

14 Search path stunts

add something to the path (e.g. to add some binary to the path which I forgot to include on the HPC)

import os

# Add a directory to the PATH
new_path = "/path/to/your/programs"
os.environ["PATH"] += os.pathsep + new_path

# Verify the updated PATH
print(os.environ["PATH"])

Capture a path update from a shell script

import subprocess
import os

# Run the bash script and capture the new PATH
result = subprocess.run("source /path/to/modify_path_script.sh && echo $PATH",
                        shell=True, executable='/bin/bash', text=True, capture_output=True)

# Update the Python environment with the modified PATH
new_path = result.stdout.strip()
os.environ["PATH"] = new_path

# Verify the updated PATH
print(os.environ["PATH"])

15 Typing

Python 3.6+ includes type hinting, and projects such as mypy support static analysis using type hints.¹ There were not many tutorials on the details of this at time of writing. Here’s one.

Short version: you go from this:

def fib(n):
  a, b = 0, 1
  while a < n:
    yield a
    a, b = b, a+b

to this:

def fib(n: int) -> Iterator[int]:
  a, b = 0, 1
  while a < n:
    yield a
    a, b = b, a+b

A version with actual corporate backing is pyright, which uses type checking to provide usability improvements in VS Codium. Might be useful.

16 Command line parsers

I am furious at how many options there are to build python CLI scripts. This is the new contender for most bike-shedded function now that the python web server world has calmed down a bit. AFAICT if you write some Python software and don’t write a new command-line parsing library, you will be regarded as somehow feckless. If you don’t make your own quirky artisanal command-line parser, try the listing at python CLIs.

17 Exceptions

TIL about raise from. See Python’s raise: Effectively Raising Exceptions in Your Code – Real Python

• Python exception hierarchy

18 Miscellaneous stuff I always need to look up

• Python data structures

19 Progress meters

tqdm seems to be a de facto standard.

Basic version with a console progress bar:

pip install tqdm

Use it:

for x in tqdm(my_list):
    pbar.update(1)

Fancy integrated HTML display is available in Jupyter. Install it:

pip install ipywidgets
jupyter nbextension enable --py widgetsnbextension
jupyter labextension install @jupyter-widgets/jupyterlab-manager

Use it:

%%capture
from tqdm import tqdm_notebook as tqdm
tqdm().pandas()

with tqdm(total=len(my_list)) as pbar:
    for x in my_list:
        pbar.update(1)

20 Images

Loading and handling images is balkanised, janky and tedious for complicated historical reasons.

These days I use Willow image library, which is a meta-library that mediates between many squabbling backend libraries. It will load many challenging formats such as SVG, HEIF and AVIF, as well as animated GIFs etc.

pip install pillow-heif Willow
# or
pip install Willow[heif]

Previously I have used

• Wand (an ImageMagick wrapper)
• Pillow

Also handy if using Pillow:

pip install pillow-avif-plugin

We also need to import it:

from PIL import Image
import pillow_avif
...

Now that I know about libvips, I would probably use that instead next time. pyvips seems to be pretty good.

21 ExitStack

If you are acquiring several context managers, there is a modern handy idiom called ExitStack. For an explanation see On the Beauty of Python’s ExitStack by Nikolaus Rath.

with ExitStack() as cm:
    res1 = acquire_resource_one()
    cm.callback(release_resource, res1)
    # do stuff with res1
    res2 = acquire_resource_two()
    cm.callback(release_resource, res2)
    # do stuff with res1 and res2

22 Everything is files

• fsspec: Filesystem interfaces for Python

  Filesystem Spec (fsspec) is a project to provide a unified pythonic interface to local, remote and embedded file systems and bytes storage.

  There are many places to store bytes, from in memory, to the local disk, cluster distributed storage, to the cloud. Many files also contain internal mappings of names to bytes, maybe in a hierarchical directory-oriented tree. Working with all these different storage media, and their associated libraries, is a pain. fsspec exists to provide a familiar API that will work the same whatever the storage backend. As much as possible, we iron out the quirks specific to each implementation, so you need do no more than provide credentials for each service you access (if needed) and thereafter not have to worry about the implementation again.

  fsspec provides two main concepts: a set of filesystem classes with uniform APIs (i.e., functions such as cp, rm, cat, mkdir, …) supplying operations on a range of storage systems; and top-level convenience functions like fsspec.open(), to allow you to quickly get from a URL to a file-like object that you can use with a third-party library or your own code.

• andrewcb/cope: Continuous Processing Engine, a small Python framework for turning input files into output files on an ongoing basis

23 Misc recommendations

• Let’s get weird: Hy is a lisp that compiles to Python syntax trees.

More Itertools: A bunch of essential (IMO) extra iterators for Python, essential for streaming work.

24 Piping functions

Everyone wants shell-style function piping to work in Python, and so there are many implementations, e.g. here are two recent ones.

• Robert Yi, Flexible piping in Python with Pipey
• Thuwarakesh Murallie, Use Pipe Operations in Python: Write Clean Codes Faster

How is it supposed to interact with generators? Many questions. Will ignore for now.

25 Python 2 vs 3

New to Python 3? These days that makes you very old. Here are some notes for people porting legacy code, or who have been frozen in a glacier for the last decade.

Sebastian Raschka, The key differences between Python 2.7.x and Python 3.x with examples.

26 Incoming

• All About ‘Bank Python,’ a Finance-Specific Language Fork