Python

Syntactic saccharine for compiled code



Assumed audience:

Active but not deep users of python

Guido van Rossum and Python, by Sir Frederick Leighton

A Swiss army knife of coding tools. Good matrix library, general scientific tools, statistics tools, web servers, art tools, and interoperation with many other tools in C, C++, Fortan etc. Python is most useful as a lingua franca It is a true lingua franca, in that

  1. It allows intercommunication of many other languages
  2. it is a Frankish creation

It includes HTTP clients, parsers, API libraries, and all the other fruits of a thriving diverse community with a common trade language. Fast enough, easy to debug, garbage-collected. If some bit is too slow, you call into a compiled language, otherwise, you relax. A good default if you’d rather get stuff done than write code. Not because it is easy to do stuff in Python as such — meh, it’s fine I guess, state-of-the-art for 90s open source — but because the chances are good that someone else has already done most of the stuff you need and released it as a library.

I typically do my stats and graphs in R. My user interface is javascript. My linear algebra library is fortran, and I use julia for my other scientific calculations. Python is the thread that stitches this Frankensteinisch monster together.

Of course, it could be better. clojure is more elegant, scala is easier to parallelise, julia prioritises science more highly, MATLAB is what your academic advisor is obsessed with…

In terms of using a well-supported satisfactory and unexciting language that goes on my computer right now, and requires me to reinvent few wheels, and which is transferable across number crunching, web development, UIs, text processing, graphics and sundry other domains, and does not require heavy licensing costs… this is an adequate one. See Hillel Wayne, The hard part of learning a language that lists all the main friction points that various languages have, and which I read as a low-key endorsement of the acceptable mediocrity of this particular language. Maybe that mediocrity is why it the tooling so good. If your language is too powerful, maybe your community is not incentivised work together on getting it to go.

What a pitch! Wow! Python for the marginal, begrudging win! Now, let’s look closer and see all the horrid things that are wrong with it.

Debugging, profiling and testing

See Python debugging, profiling and testing.

Piping functions

Everyone wants function piping to work in python, and so there are many implementations, e.g. here are two recent ones.

No one has converged upon a standard. Also, how it is supposed to interact with generators? Many questions. Will ignore for now.

Pyodide

Python from the browser, including mathematical libraries.

HT Emma Krantz for showing me this while solving my web interactive maths problems for me.

Idiom

Python people set great value on mastering python idiom. For example, python idiom for accessing a nullable member is a and a.x. I never remember this and always write getattr(a, 'x', None). Python idiom for ‘python person’ is pythonista. Python idiom for ‘idiomatic’ is pythonic. Much has been written on how to be pythonic.

I think learning can benefit also from negative examples. Here, may I proffer to you John Burkardt’s Hypershere integrals? That the least idiomatic python code I have seen. It is a line-for-line translation of C code into slow, verbose python. If you are going to do that, write C code.

ipython

The interactive python upgrade The python-specific part of jupyter, which can also run without jupyter. Long story. But think of it as a REPL, a CLI-style execution environment, that is a little friendlier and has colourisation and autocomplete and such. If sounds useful, read on

Pro tip: store config in .env

There are many, many python tools to load environment variables from local files. See .env..

String formatting/interpolation

Non obvious hacks

What a nightmare is that manual for the string formatting. While all the information I need is in there, it is arranged in perverse inversion of a comprehensible order as measured by the frequency and the priority with which I use it. See instead Marcus Kazmierczak’s cookbook.

Highlights:

## float precision
>>> print("{:.2f}".format(3.1415926))
3.14
## left padding
>>> print("{:0>2d}".format(5))
05

## power tip which the manual does not make clear:
## variable formatting of variable formatting
>>> pi = 3.1415926
>>> precision = 4
>>> print( "{:.{}f}".format( pi, precision ) )
3.1415

f-strings

f-strings make things somewhat easier for python 3.6+ because they don’t need to mess around with naming things for the .format(foo=foo) call:

>>> name = "Fred"
>>> f"He said his name is {name!r}."
"He said his name is 'Fred'."

Timestamps

Why is a now timestamp in UTC not the first line in every academic research workbook/paper/data analysis? Because it’s tedious to look up the different bits.

Here you go:

from datetime import datetime
datetime.utcnow().isoformat(timespec='seconds')

Or if we want less punctuation in there:

datetime.datetime.now().strftime("%Y%m%d-%H%M%S"))

Rendering HTML output

You have a quick and dirty chunk o’ HTML you need to output? You aren’t writing some damnable webapp with a nested hierarchy of template rendering and CSS integration into some design framework? You just want to pump out some markup?

I recommend yattag which is fast, simple, good and has a 1-page manual. It works real good.

Rendering markdown as HTML

from markdown import markdown
html_string = markdown("""
# Title

Paragraph. *emphasis*.
""")

Compiling and FFI

See Accelerating python.

Packaging and environments

See python packaging and environments.

Asynchrony in python

See asynchronous python.

functional-style programming

The style functional programming is often conceptually useful, and it can sometimes even be made performant in python.

toolz and the accelerated version cytoolz provide many useful functional abstractions which look fairly natural

ExitStack

If you are acquiring several context managers, there is a modern handy idion called ExitStack. For an explanation see On the Beauty of Python’s ExitStack by Nikolaus Rath.

with ExitStack() as cm:
    res1 = acquire_resource_one()
    cm.callback(release_resource, res1)
    # do stuff with res1
    res2 = acquire_resource_two()
    cm.callback(release_resource, res2)
    # do stuff with res1 and res2

Watching files for changes

Does this inotify solution work for non-Linux? Because macOS uses FSEvents and Windows uses I-don’t-even-know.

watchdog asserts that it is cross-platform (source).

emcrisostomo/fswatchclaims to be a universal filewatcher. There is a python frontend.

IDEs

I meant to list some, but in fact I only ever use VS Codium so that is the only current advice I would be able to give.

PyCharm also looks very good but I am hopelessly out of date on it

Typing

Python 3.6+ includes type hinting, and projects such as mypy support static analysis using type hints.1 There were not many tutorials on the details of this at time of writing. Here’s one.

Short version: you go from this:

def fib(n):
  a, b = 0, 1
  while a < n:
    yield a
    a, b = b, a+b

to this:

def fib(n: int) -> Iterator[int]:
  a, b = 0, 1
  while a < n:
    yield a
    a, b = b, a+b

A version with actual corporate backing is pyright, which uses type checking to provide usability improvements in VS Codium. Might be useful.

Command line parsers

I am furious at how many options there are to build python CLI scripts. This is the new contender for most bike-shedded function now that the python web server world has calmed down a bit. AFAICT if you write some python software and don’t write a new command-line parsing library then you will be regarded as somehow feckless. If you are indeed so dissolute as to not make your own quirky artisinal command-line parser, you could try the listing at python CLIs.

Miscellaneous stuff I always need to look up

Progress meters

tqdm seems to be a de facto standard.

Basic version with a console progress bar:

pip install tqdm

Use it:

for x in tqdm(my_list):
    pbar.update(1)

Fancy integrated HTML display is available in jupyter. Install it:

pip install ipywidgets
jupyter nbextension enable --py widgetsnbextension
jupyter labextension install @jupyter-widgets/jupyterlab-manager

Use it:

%%capture
from tqdm import tqdm_notebook as tqdm
tqdm().pandas()

with tqdm(total=len(my_list)) as pbar:
    for x in my_list:
        pbar.update(1)

Misc recommendations

  • Pext is a python extension to script interactive CLI processes things in a handy GUI.
  • Let’s get weird: Hy is a lisp that compiles to python syntax trees

More Itertools A bunch of essential (IMO) extra iterators for python, essential for streaming work.

Python 2 vs 3

Are you very old? New to python 3? These days that makes you very old.

Sebastian Raschka, The key differences between Python 2.7.x and Python 3.x with examples

Neat python 3 stuff

Alex Rogozhnikov, Python 3 with pleasure highlights some tricks which landed in the python 3 lineage. There are a lot of those.

Useful for me is a friendlier python struct-like thing, the data class Geir Arne Hjelle explains.

Also nice, we can override module accessors.

Asynchrony is less awful (although still kind of awful).

Python 2 and 3 compatibility

TL;DR: I am no employee of giant enterprise-type business with a gigantic legacy code base, and so I don’t use python 2. My code is not python 2 compatible. Python 3 is more productive, and no-one is paying me to be less productive right now. Python 2 code is usually easy to port to python 3. It is possible to write code which is compatible with python 2 and 3, but then I would miss out on some of the work that has gone into making python 3 easier and better, and waste time porting elegant easy python 3 things to hard boring python 2 things.

🏗 six versus future.


  1. If I am going to that much trouble, I would possibly prefer to be using julia, which takes type-hinting further, by actually using it to JIT-compile optimised code.↩︎


No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.