Jupyter front end systems

UX design by underfunded committee is how I like my data science experience

February 9, 2017 — March 27, 2024

faster pussycat
premature optimization
Figure 1

Jupyter, the python code dev environment, is not a monolithic thing, but a whole ecology. Different language back end kernels talk to various front end interfaces using various protocols for interaction and rendering . I can execute python code interactively using several different front-ends, each offering different user experiences. Because the, er, food web of this ecosystem is complicated it is not always easy to know which chunk of code is responsible for which part of the user experience.

The default frontend (either “classic” or “lab”) is not my favourite user experience. Notebook classic was a great first draft, although there were some things I did not enjoy. Subsequent developments have doubled down on the things I do not like. the worse user experience.

But there are choices for all tastes and some of them are not as bad. Let us examine some:

1 Command line

Command line:

jupyter run notebook.ipynb

What is the difference between this and just running normal python scripts from the command line?, you might ask. For one thing, jupyter wastes more hard disk space and doesn’t version nicely.

If we want to mix the two we can, for example, execute a notebook from the command line and capture the output in a new notebook, which might provide some advantage.

jupyter nbconvert --to notebook --execute my_notebook.ipynb

See the nbconvert executing notebooks manual for more options.

2 Papermill

papermill takes it further (too far for me in fact), turning a jupyter notebook into a python script with a python command line interface.

papermill is a tool for parameterizing, executing, and analyzing Jupyter Notebooks.

Papermill lets you:

  • parameterize notebooks
  • execute notebooks

This opens up new opportunities for how notebooks can be used. For example:

  • Perhaps you have a financial report that you wish to run with different values on the first or last day of a month or at the beginning or end of the year, using parameters makes this task easier.
  • Do you want to run a notebook and depending on its results, choose a particular notebook to run next? You can now programmatically execute a workflow without having to copy and paste from notebook to notebook manually.

These are all fine features, but if I were going to build them, would I not rather write a script without jupyter? Nonetheless, this probably makes it easier for some people and certainly beats excel.

3 Mercury

Mercury allows you to add interactive widgets in Python notebooks, so you can share notebooks as web applications. Forget about rewriting notebooks to web frameworks just to share your results. Mercury offers a set of widgets with simple re-execution of cells.

You can build with Mercury:

  • Turn your notebook into beautiful Web Apps,
  • Create interactive Presentations with widgets, you can recompute slides during the show,
  • Share notebooks as static Websites,
  • Build data-rich Dashboards with widgets,
  • Create Reports with PDF exports, automatic scheduling, and email notifications (coming soon),
  • Serve Python notebooks as REST API endpoints (coming soon).

4 VS Code Juptyer

VS code’s jupyter integration is good-ish and its support for plain python is excellent.

A jupyter frontend is already an app that we need to install to interact with the code; so why not install a good one instead of a bad one? Further, there are a huge number of benefits to using a proper editor to edit code, instead of jupyter. More benefits than I care to enumerate, but I’ll start.

I do not need to learn new keyboard shortcuts. I do not need to arse around with the various defective reimplementations of text editors from inside the browser. Further, heaps of other things that jupyter cannot even dream of just magically work! Session sharing? No problem. Remote editing? Easy! Type inference! Autocomplete! Debugger injection! Search and replace across a whole project! Style checking! refactor assistance! Comprehensible documentation.

See VS Code for python for more details.

Still, if that is not your jam, read on for some onerous alternatives that I hate.

5 Notebook classic

The one that comes recommended until, I dunno, 2015 or so I guess? I cannot recall. Worth knowing about, to set expectations for what all the other frontends are attempting to emulate.

5.1 Configuring

The location of themeing infrastructure, widgets, CSS etc has moved of late; check your version number. The current location is ~/.jupyter/custom/custom.css, not the former location ~/.ipython/profile_default/static/custom/custom.css

Julius Schulz’s ultimate setup guide is also the ultimate pro tip compilation.

5.2 Auto-closing parentheses

If I must use jupyter notebook then I kill parenthesis molestation (referred to in the docs as bracket autoclose) with fire. I do not like having to fight with the notebook’s misplaced faith in its ability to read my mind. The setting is tricky to find, because it is not called “put syntax errors in my code without me asking Y/N”, but instead cm_config.autoCloseBrackets and is not in the preference menus. According to a support ticket, the following should work.

# Run this in Python once, it should take effect permanently

from notebook.services.config import ConfigManager
c = ConfigManager()
c.update('notebook', {"CodeCell": {
  "cm_config": {"autoCloseBrackets": False}}})

or add the following to custom.js:

], function(Jupyter) {
    Jupyter.CodeCell.options_default.cm_config.autoCloseBrackets = false;

or maybe create ~/.jupyter/nbconfig/notebook.json with the content

  "CodeCell": {
    "cm_config": {
      "autoCloseBrackets": false

That doesn’t work with jupyterlab, which is even more righteously sure that it knows better than you about what you truly wish to do with parentheses. Perhaps the following does work? Go to Settings --> Advanced Settings Editor and add the following to the User Overrides section:

  "codeCellConfig": {
    "autoClosingBrackets": false

or in the file ~/.jupyter/lab/user-settings/@jupyterlab/notebook-extension/tracker.jupyterlab-settings.

Ahhhhhhhh. Update: it is easier now.

Figure 2

5.3 Notebook extensions

Jupyter classic is more usable if you install the notebook extensions, which includes, e.g. drag-and-drop image support.

$ pip install --upgrade jupyter_contrib_nbextensions
$ jupyter contrib nbextension install --user

For example, if you run nbconvert to generate a HTML file, this image will remain outside of the html file. You can embed all images by using the calling nbconvert with the EmbedPostProcessor.

$ jupyter nbconvert --post=embed.EmbedPostProcessor

Update — broken in Jupyter 5.0

Wait, that was still pretty confusing; I need the notebook configurator whatsit.

$ pip install --upgrade jupyter_nbextensions_configurator
$ jupyter nbextensions_configurator enable --user

6 Jupyter lab

jupyter lab (sometimes styled jupyterlab) is the current cutting edge according to jupyter mainline, and reputedly is much nicer to develop plugins for than the notebook interface. From the user perspective it’s more or less the same thing, but the annoyances are different. It does not strictly dominate notebook in terms of user experience, although I understand it may do in terms of experience for plugin developers.

A pitch targeted at us users explains some practical implications of jupyterlab and how it is the one true way and the righteous future etc. Since we are not in the future, though, we must deal with certain friction points in the present, actually-existing jupyterlab.

6.1 Jupterlab UI

The UI, though… The mysterious curse of javascript development is that once you have tasted it, you are unable to resist an uncontrollable urge to reimplement something that already works, but as a jankier javascript version. The jupyter lab have tasted that forbidden fruit, have the spidermonkey on their backs, and they hanger to reinvent things using javascript. So far they ahve reimplemented copy, paste, search/replace, browser tabs and the command line.

Yo dawg I heard you like notebook tabs so I put notebook tabs in your browser notebook tab.

The replacement jupyter tab system clashes with the browser tab system, keyboard shortcuts and generally confuses the eye. Why do we get some a search function which AFAICT non-deterministically sometimes does regexp matching but then doesn’t search the whole page? Possibly the intention is that it should not be run through a normal browser, but in a custom single-tab embedded browser? Or maybe for true believers, once you load up a jupyter notebook you like it so hard that you never need to browse to a website on the internet again and you close all the other tabs forever. Whatever the rationale, the learning curve for these weird UI choices is bumpy, and the lessons are not transferable.

Is the learning process worth it? I am not using jupyter for its artisanal, quirky alternate take on tabs, cut-and-paste etc, but because I want a quick interface to run some shareable code with embedded code and graphics. Does it get me that?

Maybe. If I get in and out fast, do most of my development in a real code editor and leave the jupyter nonsense for results sharing, I neither need the weird UI, nor am I bothered by it. The other features are like that button on the microwave labelled “fish”, which no one has ever need or intentionally used, but which does not stop the microwave from defrosting things if you use the normal controls.

At the same time, some jupyterlab enthusiasts want to re-implement text editors, which is an indicator that there might be a contagion of NIH fever going around int the community, and it makes me nervouse.

Whether I like the overwrought jupyter lab UX or not, we should allow it a moderate nonsense-baggage allowance, if the developer API is truly cleaner and easier to work with. That would be a solid win in terms of delivering the features I would actually regard as improvements including, maybe, ultimately, a better UI.

6.2 Collaborative options

Simultaneous users are supported natively in jupyterlab with an OK UI for identifying running notebooks. IMO this could be a killer feature of jupyter lab. As currently implemented it is useable but sometime calculation output gets lost in the system somewhere.

There is an under-documented project to introduce real time collaboration to jupyterlab coordinating on notebook content code, output and backend state, which apparently works, but is barely mentioned in the docs. Maybe the JupyterLab RTC Design Documentation would help there.

If you have a snakepit of different jupyter sessions running on some machine you have just logged in to and which to open up the browser to get a UI for them, then you want to work out which are running on a given machine so that you can attach them. The command is (for either jupyter notebook notebooks or jupyter lab sessions):

jupyter notebook list

6.3 Lab extensions

Related to, inspired but and maybe conflicting or intersecting with the nbextensions are the labextensions, which add bits of extra functionality to the lab interface rather than the notebook interface (where the lab interface is built upon the notebook interface and runs notebooks just like it but with different moving parts under the hood.)

I try to keep the use of these to a minimum as I have a possibly irrational foreboding that some complicated death spiral of version clashes is beginning between all the different juptyer kernel and lab and notebook installations I have cluttering up my hard disk and it can’t improve things to put various versions of lab extensions in the mix can it? And I really don’t want to have to understand how it works to work out whether that is true or not, so please don’t explain it to me. I moreover do not wish to obsessively update lab extensions everywhere.

Anyway there are some useful ones, so I live with it by running install and update commands obsessively in every combination of kernel/lab/whatever environment in the hope that something sticks.

Life is easier with jupyerlab-toc which allows you to navigate your lab notebook by markdown section headings.

jupyter labextension install @jupyterlab/toc

The upgrade command is

jupyter labextension update @jupyterlab/toc

Integrated diagram editor? Someone integrated drawio as jupyterlab-drawio to prove a point about the developer API thing.

jupyter labextension install jupyterlab-drawio

LaTeX editor? As flagged, I think this is a terrible idea. Even worse than the diagram editor. There are better editors than jupyter, better means of scientific communication than latex, and better specific latex tooling, but I will concede there is some kind of situation where this sweet spot of mediocrity might be useful, e.g. as a plot point in a contrived techno-thriller script written by cloistered nerds. If you find yourself in such dramaturgical straits:

jupyter labextension install @jupyterlab/latex

One nerdy extension is jupyter-matplotlib, a.k.a., confusingly, ipympl, which integrates interactive plotting into the notebook better.

pip install ipympl
# If using JupyterLab

# Install nodejs: https://nodejs.org/en/download/
jupyter labextension install @jupyter-widgets/jupyterlab-manager
jupyter labextension install jupyter-matplotlib

jupyterlab/jupyterlab-hdf5 claims to provide a UI for HDF5 files.

7 Quarto

Quarto® is an open-source scientific and technical publishing system built on Pandoc

See quarto.

8 Jupyterbook

Special fancy thing: Jupyter Book is a publishing engine for jupyter.

9 qtconsole

A classic, i.e. non-web-browser based client for jupyter. No longer fashionable? Seems to work fine but is sometimes difficult to compile and doesn’t support all the fancy client-side extensions.

9.1 Multiple clients connecting to a single kernel

jupyter qtconsole can connect two frontends to the same kernel. This will be loopy since they update the same variables (presumably) but AFAICT not the same notebook content, so some care would be required to make sure you are doing what you intend.


10 Google colab

A proprietary google fork/extension, Colaboratory is a jupyter thingo integrated with some fancy hosting and storage infrastructure, and you get free GPUs. Looks like a neat way of sharing things, and all that they demand is your soul. Don’t be fooled, though, claims you see on the internet that this is a real-time collaborative environment are false, Goggle killed realtime interaction.

11 nteract

nteract is an Electron desktop app which runs jupyter notebooks. Could possibly host native-type interactions

nteract is an ecosystem of React components, JavaScript packages, and applications built on top of the Jupyter specification. Together they enhance interactive computing and data science workflows.

What can I do with nteract?](https://docs.nteract.io/#what-can-i-do-with-nteract)

  • Parameterize and execute notebooks with papermill.
  • Build your own interactive notebook application with our React components.
  • Host notebooks with commuter.

12 hydrogen

hydrogen, a plugin for the atom text editor, provides a more unified coding experience with a normal code editor. See the intro blog post. Discontinued.

13 Pweave

pweave is like knitr for python. It also executes jupyter kernels. The emphasis is not interactive but rather reproducible documents.