The Jupyter Cinematic Universe

1 Why jupyter?

What does jupyter buy us? Is it worth the set-up time spent configuring this contraption?

It took me a while to realise that part of the answer to the first question is that jupyter is a de facto standard for running remote computation jobs interactively. The browser-based, network-friendly jupyter notebook is a natural, easy way to execute tedious computations on some other computer somewhere else, with some kind of a paper trail. In particular, it is much better over unreliable networks than are remote terminals or remote desktops, because the client/server architecture doesn’t need to do so many round-trips to get the code output back to the user. A good feature of jupyter (maybe its best) is a kind of re-designed network terminal. Certainly, if what we need to do could be executed over remote desktop or jupyter, jupyter is going to be less awful over laggy network connections, when every mouse click and keystroke involves waiting and twiddling your fingers.

What else? People make UX arguments, e.g. that jupyter is friendly and supports interactive plots and so on. I am personally ambivalent about those arguments. Jupyter can do some things better than the console. That artificially restricted comparison does not reassure me; we are not limited to the console. Indeed, most things that jupyter does, it does worse than a proper IDE or decent code editor. Sometimes those other tools are not available on, say, the local HPC cluster or cloud compute environment, and then this becomes a relevant advantage. Usually though, we can install VS Code Remote, unless we have angered the sysadmins. If we really want a quick minimalist UI, Marimo is probably better.

These factors (UI, remote-network-friendliness) were both game-changing in 2014, and I think people are still excited about this element of jupyter, even though there are greatly superior alternatives around, some of them also free and open source, which build upon the same infrastructure but throw out the annoying python front-end.

But for now the main takeaway, I think, is that if, like me, you are confused by jupyter enthusiasts claiming it is “easy” or “fun”, it may make more sense if you mentally append the proviso “…in comparison to some other horrible thing which I was forced to use by ignorance or circumstance several years ago but which you probably never used.” It’s like your grandad espousing a fax machine because it’s so much better than a telex machine.

There are other comparisons to make — some like jupyter as a documentation format/literate coding environment. Once again, sure, it is better than text files. But then, Quarto is more portable, VS Code notebook mode versions better etc.

We can generically answer “Is it worth it?” with “That depends on the alternatives. Jupyter is adequate, and commonly available.”

2 Alternatives

My ambivalence about jupyter leads me to consider other options for interactive code execution in python.

First, be aware there are many variant front-ends to jupyter, which ameliorate some of the pain points of the jupyter notebook interface, e.g. quarto uses jupyter python kernels, but disregards the jupyter notebook interface in favour of a more traditional document-based interface.

But also! There are other interactive python environments which entirely re-imagine the python notebook concept.

3 Python-specific bits

Jupyter supports multiple languages, but is itself built in python (and javascript) and there are some python-specific bits integrations via IPython, especially the “magics”, % and %% commands which are used to control the notebook environment.

4 Engineering pain points

What technical difficulties can we expect when using jupyter?

Jupyter is often touted as a wonderful solution for data science which makes stuff generally easier. This is probably true at certain points on the learning curve, and for individuals at very specific points on their python learning journey. It is not true in general. Rather than simply easing away difficulties, jupyter introduces all manner of difficulties of its own, in which the pain points are often surprising and novel, which does not make them better.

I’m an equivocal advocate of the jupyter notebook interface, which some days seems to counteract every plus with a minus, but at least is a pretty easy starting point compared to something scary-looking like the command-line. My qualms are partly due to the particulars of jupyter’s design decisions, and partly because of the problems of notebook interfaces generally (Chattopadhyay et al. 2020). As with so many computer interfaces, my lukewarm endorsement is, in relative terms, fannish enthusiasm because there are so many horrible alternatives.

Jupyter:

• …is friendly to use

  • but hard to install

• …makes it easy to explore my data with graphs and such

  • but hard to keep that exploration in version control

• …lets me type code in easily

  • but makes it hard to work out why it is not working because jupyter clashes with the fancy debugger that would make it easy to explore my code bugs.

• is open source, and written in an easy scripting language, python, so it seems it should be easy to tweak to taste

  • but in practice it is a nightmare to extend. In practice it’s an ill-explained spaghetti of python, javascript, compiled libraries and browsers that relate to one another in obscure ways that few people with a day job have time to understand or contribute to.

People are not blind to these problems. There have been so many reboots, rewrites, re-architectures, and reorganisations that it’s hard to know what is going on. Each line of development takes place in a separate timeline in an extended cinematic multiverse, wherein the writer’s rooms are occasionally merged, but the timelines are never reconciled. Things regularly break either at the server or client side and I might need to upgrade either or both to fix it. I might have many different installs of each and need to upgrade a half-dozen different installs to keep them all working, because jupyter is deeply intertwined with the paint points python packaging hell and in many ways makes them worse by multiplying the number of python environments I need to manage. It claims to be extensible but if I use any extensions, it is a constant struggle to keep jupyter finding the many intricate dependencies that are needed to keep the entire contraption running. The sum total is IMO no more easy to run than most of the other UI development messes that we tolerate in academic software. Case study: look a dependency of a dependency of the autocomplete function broke something and thus spawned a multi-month confusion of cascading problems and cost me several hours to fix across the few dozen different python environments I manage across several computers. This kind of tedious intermittent breakage is much the cost of doing business with jupyter, and has been so for as long as I have been using the project, which is as long as it has existed.

These pain points are perhaps not so intrusive for projects of small-to-intermediate complexity and/or longevity. Indeed, jupyter seems good at making quick data science projects look smooth, shiny, and inviting. That is, at the crucial moment when I need to make my data science project look sophisticated-yet-friendly, it lures colleagues into my web(-based IDE). Then it is too late mwhahahahah you have fallen into my trap now you are committed you had better find an engineering budget to maintain this mess. This entrapment might be a feature not a bug, as far as the realities of team dynamics and their relation to software development and organisational support. We want to lure people in until our problems become their problems, because a problem shared is a problem divided. Also shared trauma is a bonding experience.

Some argue that the weird / irritating constraints of jupyter is a strength, and can even incentivise good architecture. See Guillaume Chevallier and Jeremy Howard.

I think of the famous adage “The fastest code is code that doesn’t need to run, and the best code is code you don’t need to write”. The uncharitable corollary might be “Thus, let’s make writing code horrible so that you write less of it”. That is not even necessarily a crazy position, and if that is what Guillaume and Jeremy are saying, I guess I’ll bear it in mind? This some analogue of the traffic calming measures that make a neighbourhood more liveable. If we make it horrible to drive here, people will bring in fewer cars.

Here is some verbiage by Will Crichton which explores some of these themes, The Future of Notebooks: Lessons from JupyterCon.

5 Terminology

Pain point: The lexicon of jupyter is confusing. Terminology tarpit alert.

A notebook is on one hand a style of interface, to which jupyter conforms to one interpretation of. Other applications with a notebook style of interface are Mathematica and MATLAB.

Jupyter interfaces communicate with a computational backend, which is called a kernel¹

These are software packages in which a unit of development is a type of notebook file on your disk, containing both code and output of that code. (In the case of jupyter this file format is marked by file extension .ipynb, which is short for “ipython notebook”, for fraught historical reasons.) One implementation of a notebook frontend interface over a notebook protocol for jupyter is called the jupyter notebook, launched by the jupyter notebook command which will open up a javascript-backed notebook interface in a web browser. Another, more recent notebook-style interface implementation is called jupyter lab, which additionally uses much of the same jupyter notebook infrastructure but is distinct and only sometimes interoperable in ways which I do not pretend to know in depth. But there are multiple ‘frontends’ besides which interact over the jupyter notebook protocol to talk to a kernel.

Which sense of notebook is intended you have to work out from context, e.g. the following sentence is contentful:

You dawg, I heard you like notebooks, so I started up your jupyter notebook kernel in jupyter notebook

6 Jupyter as UI

See jupyter UI.

7 Front ends

See jupyter front ends.

8 Jupyter kernels

Jupyter kernels now come in (at least) 2 flavours. I do not know to what extent they are interchangeable. Classic flavour is ipykernel which is shambolic but works. xeus is a new entrant.

A persistent process (e.g. Python if you are running Python) hovers in the background waiting for code from you and sends back the results to be displayed in the notebook.

pip install ipykernel
python -m ipykernel install --user --name=my-virtualenv-name

conda install nb_conda_kernels

{
    "display_name": "dan-gpu",
    "language": "python",
    "argv": [
        "/usr/bin/optirun",
        "--no-xorg",
        "/home/me/.virtualenvs/dan/bin/python",
        "-m",
        "ipykernel_launcher",
        "-f",
        "{connection_file}"
    ]
}

CUDA_VISIBLE_DEVICES= $*

9 Jupyterlite

• JupyterLite

JupyterLite is a JupyterLab distribution that runs entirely in the browser built from the ground-up using JupyterLab components and extensions.

10 Hosting static Jupyter notebooks on the web

Various options. For one, GitHub will attempt to render Jupyter notebooks in GitHub repos.; I have had various glitches and inconsistencies with images and equations rendering in such notebooks. Perhaps it is better in…

The fastest way to share your notebooks - announcing NotebookSharing.space - Yuvi Panda

You can upload your notebook easily via the web interface at notebooksharing.space: Once uploaded, the web interface will just redirect you to the beautifully rendered notebook, and you can copy the link to the page and share it!

Or you can directly use the nbss-upload command line tool: …

When uploading, you can opt-in to have collaborative annotations enabled on your notebook via the open source, web standards-based hypothes.is service. You can thus directly annotate the notebook, instead of having to email back and forth about ‘that cell where you are importing matplotlib’ or ‘that graph with the blue border’. This is one of the coolest features of notebooksharing.space.

11 Hosting live Jupyter notebooks on the web

Jupyter can host online notebooks, even multi-user notebook servers — if you are brave enough to let people execute weird code on your machine. I’m not going to go into the security implications. tl;dr encrypt and password-protect that connection. Here, see this jupyterhub tutorial.

12 Pro tips and gotchas

%%writefile basic_imports.py
%load basic_imports.py

python -m IPython.external.MathJax /path/to/source/MathJax.zip

jupyter notebook --generate-config
atom ~/.jupyter/jupyter_notebook_config.py

13 Securing

Modern Jupyter is suspicious of connections per default and will ask you either for a magic token or a password and thereafter, I think, encrypts the connection (?), which is sometimes what I want. Not always.

When I am in HPC hell, accessing Jupyter notebooks through a double SSH tunnel, the last thing I need is to put a hat on a hat by triply securing the connection. This is now adding more points of failure without any additional benefit. Also, sometimes the tokens do not work over SSH tunnels for me and I cannot work out why. I think it is something about some particular Jupyter version mangling tokens, or possibly failing to report that it has not claimed a port used by someone else (although it happens more often than is plausible for the latter case). CodingMatters notes that the following invocation will disable all Jupyter-side security measures:

$ jupyter notebook --port 5000 --no-browser --ip='*' --ServerApp.token='' --ServerApp.password=''

Obviously never do this unless you believe that everyone sharing a network with that machine has your best interests at heart. The best way to ensure that is to be accessing a machine through a firewall to a locked-down port.

There are various other useful settings which one could use to reduce security. In config file format for ~/.jupyter/jupyter_notebook_config.py:

c.ServerApp.disable_check_xsrf = True #irritates ssh tunnel for me that one time
c.ServerApp.open_browser = False # consumes a 1 time token and is pointless from a headless HPC
c.ServerApp.use_redirect_file = False # forces display of token rather than writing it to some file that gets lost in the containerisation and is useless in headless HPC
c.ServerApp.allow_password_change = True # Allow password setup somewhere sensible.
c.ServerApp.token = '' # no auth needed
c.ServerApp.password = password # actually needs to be hashed - see below

Eric Hodgins recommends this hack for a simple password without messing about trying to be clever with their browser infrastructure (which TBH does seem to break pretty often for me):

c = get_config()
c.ServerApp.ip = '*'
c.ServerApp.open_browser = False
c.ServerApp.port = 5000

# setting up the password
from IPython.lib import passwd
password = passwd("your_secret_password")
c.ServerApp.password = password

14 As a proxy

jupyter-server-proxy:

Jupyter Server Proxy lets you run arbitrary external processes (such as RStudio, Shiny Server, syncthing, PostgreSQL, etc) alongside your notebook, and provide authenticated web access to them.

Note

This project used to be called nbserverproxy. if you have an older version of nbserverproxy installed, remember to uninstall it before installing jupyter-server-proxy - otherwise they may conflict

The primary use cases are:

1. Use with JupyterHub / Binder to allow launching users into web interfaces that have nothing to do with Jupyter - such as RStudio, Shiny, or OpenRefine.
2. Allow access from frontend JavaScript (in classic notebook or JupyterLab extensions) to access web APIs of other processes running locally in a safe manner. This is used by the JupyterLab extension for dask.

15 References