Jupyter front end systems

UX design by underfunded committee is how I like my data science experience



Jupyter, the python code dev environment, is a whole ecology of different language back end kernels talking to various front end interfaces. That is, I can execute python code interactively using several different front-ends, each offering different user experiences. Because the plumbing of this whole thing is complicated, it is easy to miss this fact in the docs. It was worth finding out for me because of the many irritations I have with jupyter, the IMO horrible default frontend (“classic” or “lab”) provide the worse user experience. But there are choices for all tastes and some of them are not as bad. Let us examine some:

  • Classic jupyter notebook, the archetypal browser-based coding environment. The command jupyter notebook starts this frontend.
  • jupyterlab, the newer iteration of classic notebook, extends and redesigns the classic notebook into a pseudo-IDE with text editors, notebooks, REPL terminals etc. The command jupyter lab starts this front-end.
  • base ipython shell can execute notebooks. (Is that the same as jupyter console?)
  • vscodeJupyter is an editor for notebooks VS Code. VS code has jupyter integration with corporate backing It is fast and IMO much less nasty than the horrible jupyter notebook UX which I constantly whinge about. This is what I currently use. IMO, IDE interfaces like this represent a generally better way of doing python execution, because for most of us, executing python code requires writing python code. These interfaces are great at running code, but not so great at editing code. Code editors exist and have had much effort poured into them, to the point where they are pretty good at editing code. Jupyter should not need to reinvent text editors; Although my opinions will not dissuade some Quixote from continuing to try. (C&C the phenomenon of Rstudio insistently reinventing code editors for R).
  • hydrogen is an equivalent for Atom that I have not used much but might be fine.
  • nteract is a system for turning juptyer notebooks into apps somehow?
  • pweave, also executes jupyter kernels as part of a reproducible document.
  • qtconsole is a traditional client which de-emphasises browser-based stuff in favour of desktop-integrated windows.
  • Probably others too.

VS Code Juptyer

If you use VS code at all, its jupyter integration is worthwhile. In fact it is so much better that I would argue it is a worthwile reason to install VS code even if you don't normally use it. You may as well; a jupyter frontend is already an app that you need to install; so why not install a good one instead of a bad one. All in all it is much better than the jupyter notebook browser front end or the new jupyter lab. I imagine that some jupyter extensions don't work, but since all the extensions I was installing were simply to try and make jupyter behave a little bit more like vs code, they are, AFAICT, redundant.

There are a huge number of benefits. More than I care to enumerate, but I'll start. I do not need to learn new keyboard shortcuts. I do not need to arse around with the various defective reimplementations of text editors from inside the browser.

Further, heaps of other things that jupyter cannot even dream of just magically work! Session sharing? No problem. Remote editing? Easy! Type inference! Autocomplete! Debugger injection! Search and replace across a whole project! Style checking! refactor assistance! Comprehensible documentation.

Still, if that is not your jam, read on for some onerous alternatives that I hate.

Notebook classic

Configuring

The location of themeing infrastructure, widgets, CSS etc has moved of late; check your version number. The current location is ~/.jupyter/custom/custom.css, not the former location ~/.ipython/profile_default/static/custom/custom.css

Julius Schulz’s ultimate setup guide is also the ultimate pro tip compilation.

Auto-closing parentheses

Kill parenthesis molestation (a.k.a. bracket autoclose) with fire. Unless you like having to fight with your IDE’s misplaced faith in its ability to read your mind. The setting is tricky to find, because it is not called “put syntax errors in my code without me asking Y/N”, but instead cm_config.autoCloseBrackets and is not in the preference menus. According to a support ticket this should work.

# Run this in Python once, it should take effect permanently

from notebook.services.config import ConfigManager
c = ConfigManager()
c.update('notebook', {"CodeCell": {
  "cm_config": {"autoCloseBrackets": False}}})

or add the following to your custom.js:

define([
    'base/js/namespace',
], function(Jupyter) {
    Jupyter.CodeCell.options_default.cm_config.autoCloseBrackets = false;
})

or maybe create ~/.jupyter/nbconfig/notebook.json with the content

{
  "CodeCell": {
    "cm_config": {
      "autoCloseBrackets": false
    }
  }
}

That doesn’t work with jupyterlab, which is even more righteously sure that it knows better than you about what you truly wish to do with parentheses. Perhaps the following does work? Go to Settings --> Advanced Settings Editor and add the following to the User Overrides section:

{
  "codeCellConfig": {
    "autoClosingBrackets": false
  }
}

or in the file ~/.jupyter/lab/user-settings/@jupyterlab/notebook-extension/tracker.jupyterlab-settings.

Ahhhhhhhh.

Notebook extensions

Jupyter classic is more usable if you install the notebook extensions, which includes, e.g. drag-and-drop image support.

$ pip install --upgrade jupyter_contrib_nbextensions
$ jupyter contrib nbextension install --user

For example, if you run nbconvert to generate a HTML file, this image will remain outside of the html file. You can embed all images by using the calling nbconvert with the EmbedPostProcessor.

$ jupyter nbconvert --post=embed.EmbedPostProcessor

Update — broken in Jupyter 5.0

Wait, that was still pretty confusing; I need the notebook configurator whatsit.

$ pip install --upgrade jupyter_nbextensions_configurator
$ jupyter nbextensions_configurator enable --user

Jupyter lab

jupyter lab (sometimes styled jupyterlab) is the current cutting edge, and reputedly is much nicer to develop plugins for than the notebook interface. From the user perspective it’s more or less the same thing, but the annoyances are different. It does not strictly dominate notebook in terms of user experience, although I understand it does in terms of experience for plugin developers.

A pitch targeted at us users explains some practical implications of this and how it is the one true way and the righteous future etc. Since we are not in the future, though, we must deal with certain friction points in the present, actually-existing jupyterlab.

Jupterlab UI

The UI, though… The mysterious curse of javascript development is that once you have tasted it, you are unable to resist an uncontrollable urge to reimplement something that already worked, but as a jankier javascript version. The jupyter lab creators have succumbed as far as reimplementing copy, paste, search/replace, browser tabs and the command line. In the tradition of jupyter this I think of this as

Yo dawg I heard you like notebook tabs so I put notebook tabs in your notebook browser tab.

The replacement jupyter tab system clashes with the browser tab system, keyboard shortcuts and generally confuses the eye. Why do we get some a search function which AFAICT non-deterministically sometimes does regexp matching but then doesn’t search the whole page? Possibly the intention here is that it should not really be run through the browser, but in a custom app? Or maybe for true believers, once you load up a jupyter notebook you like it so hard that you never need to browse to a website on the internet again and you close all the other tabs forever. Whatever the rationale, the learning curve for this particular bag of weird UI choices is bumpy, and the lessons are not transferable. Is the learning process worth it though? Am I merely grumpy because I do not like some things? I am not using jupyter for its artisinal, quirky alternate take on tabs, cut-and-paste etc, but because I want a quick interface to run some shareable code with embedded code and graphics. Does it get me that?

Maybe. If I get in and out fast, do most of my development in a real code editor and leave the jupyter nonsense for results sharing, I neither need the weird UI, nor am I bothered by it. It is like the weird button on the microwave labelled “fish” which no one has ever pressed, but which does not stop the microwave from defrosting things. At the same time, some jupyterlab enthusiasts want to re-implement text editors, which really feels like there might be a contagion of NIH fever going about the community. But until that imagined problem becomes a real problem, I will try to ignore it.

Whether I like the overwrought jupyter lab UX or not, we should allow it a general nonsense baggage allowance, if the developer API is truly cleaner and easier to work with. That would be a solid win in terms of delivering the features I would actually regard as improvements including, maybe, ultimately, a better UI.

Collaborative options

Simultaneous users are supported natively in jupyterlab with a good UI for identifying running notebooks. IMO this is the killer feature of jupyter lab. Which is to say, this is useable but sometime calculation output gets lost in the system somewhere.

There is an underdocumented project to introduce real time collaboration to jupyterlab coordinating on notebook content code, output and backend state, but that does not work yet. update: maybe it does now??

If you have a snakepit of different jupyter sessions running on some machine you have just logged in to and which to open up the browser to get a UI for them, then you want to work out which are running on a given machine so that you can attach them. The command is (for either jupyter notebook notebooks or jupyter lab sessions):

jupyter notebook list

Lab extensions

Related to, inspired but and maybe conflicting or intersecting with the nbextensions are the labextensions, which add bits of extra functionality to the lab interface rather than the notebook interface (where the lab interface is built upon the notebook interface and runs notebooks just like it but has some different moving parts under the hood.)

I try to keep the use of these to a minimum as I have a possibly irrational foreboding that some complicated death spiral of version clashes is beginning between all the different juptyer kernel and lab and notebook installations I have cluttering up my hard disk and it can’t improve things to put various versions of lab extensions in the mix can it? And I really don’t want to have to understand how it works to work out whether that is true or not, so please don’t explain it to me. I moreover do not wish to obsessively update lab extensions everywhere.

Anyway there are some useful ones, so I live with it by running install and update commands obsessively in every combination of kernel/lab/whatever environment in the hope that something sticks.

Life is easier with jupyerlab-toc which allows you to navigate your lab notebook by markdown section headings.

jupyter labextension install @jupyterlab/toc

The upgrade command is

jupyter labextension update @jupyterlab/toc

Integrated diagram editor? Someone integrated drawio as jupyterlab-drawio to prove a point about the developer API thing.

jupyter labextension install jupyterlab-drawio

LaTeX editor? As flagged, I think this is a terrible idea. Even worse than the diagram editor. There are better editors than jupyter, better means of scientific communication than latex, and better specific latex tooling, but I will concede there is some kind of situation where this sweet spot of mediocrity might be useful, e.g. as a plot point in a contrived techno-thriller script written by cloistered nerds. If you find yourself in such dramaturgical straits:

jupyter labextension install @jupyterlab/latex

One nerdy extension is jupyter-matplotlib, a.k.a., confusingly, ipympl, which integrates interactive plotting into the notebook better.

pip install ipympl
# If using JupyterLab

# Install nodejs: https://nodejs.org/en/download/
jupyter labextension install @jupyter-widgets/jupyterlab-manager
jupyter labextension install jupyter-matplotlib

jupyterlab/jupyterlab-hdf5 claims to provide a UI for HDF5 files.

qtconsole

A classic, i.e. non-web-browser based client for jupyter. No longer fashionable? Seems to work fine but is sometimes difficult to compile and doesn’t support all the fancy client-side extensions.

Multiple clients connecting to a single kernel

jupyter qtconsole can connect two frontends to the same kernel. This will be loopy since they update the same variables (presumably) but AFAICT not the same notebook content, so some care would be required to make sure you are doing what you intend.

%qtconsole

Google colab

A proprietary google fork/extension, Colaboratory is a jupyter thingo integrated with some fancy hosting and storage infrastructure, and you get free GPUs. Looks like a neat way of sharing things, and all that they demand is your soul. Don’t be fooled, though, claims you see on the internet that this is a real-time collaborative environment are false, Goggle killed realtime interaction.

hydrogen

hydrogen, a plugin for the atom text editor, provides a more unified coding experience with a normal code editor. See the intro blog post.

Pweave

pweave is like knitr for python. It also executes jupyter kernels. The emphasis is not interactive but rather reproducible documents.

nteract

nteract is a system for running jupyter notebooks as desktop apps, integrating with OS indexing services and looking pretty etc. Not totally sold on this idea because it looks so bloaty, but I would like to be persuaded.


No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.