Browser machine learning

WebNN

A unifying API?

The WebNN API is a specification for constructing and executing computational graphs of neural networks. It provides web applications with the ability to create, compile, and run machine learning networks on the web browsers. The WebNN API may be implemented in web browsers using the available native operating system machine learning APIs for the best performance and reliability of results.

• webnn explainer
• WebNN

TVM

• apache/tvm: Open deep learning compiler stack for cpu, gpu and specialized accelerators
• Apache TVM
• Apache TVM Unity: a vision for the ML software & hardware ecosystem in 2022

Apache TVM is a compiler stack for deep learning systems. It is designed to close the gap between the productivity-focused deep learning frameworks, and the performance- and efficiency-focused hardware backends. TVM works with deep learning frameworks to provide end to end compilation to different backends.

Impressive example: mlc-ai/web-stable-diffusion: Bringing stable diffusion models to web browsers. Everything runs inside the browser with no server support.

ONNX

microsoft/onnxruntime allows us to run neural nets in the browser:

ONNX Runtime Web is a Javascript library for running ONNX models on browsers and on Node.js.

ONNX Runtime Web has adopted WebAssembly and WebGL technologies for providing an optimized ONNX model inference runtime for both CPUs and GPUs.[…]

With ONNX Runtime Web, web developers can score models directly on browsers with various benefits including reducing server-client communication and protecting user privacy, as well as offering install-free and cross-platform in-browser ML experience.

• Inference machine learning models in the browser with JavaScript and ONNX Runtime Web
• Deploy on web
• Build a web app with ONNX Runtime
• microsoft/onnxruntime-nextjs-template

Tensorflow.js

A less generic runtime, in that it supports primarily tensorflow. NB it seems to support the superior jax.

Approximate scope is demonstrated in

• Platform and environment

TBD

Incoming

Have not updated the subsequent ones for quite some time and I suspect many of them are defunct.

• Teachable machine is a web-based tool that makes creating machine learning models fast, easy, and accessible to everyone.

  The models you make with Teachable Machine are real Tensorflow.js models that work anywhere javascript runs, so they play nice with tools like Glitch, P5.js, Node.js & more.

  Plus, export to different formats to use your models elsewhere, like Coral, Arduino & more.

• deeplearn.js, which can load tensorflow checkpoints Update: It’s now called tensorflow.js: “A WebGL accelerated, browser based JavaScript library for training and deploying ML models.” More precisely, “TensorFlow.js, an ecosystem of JavaScript tools for machine learning, is the successor to deeplearn.js which is now called TensorFlow.js Core.” Includes lots of sample code for integration with the browser etc.

  • subgraphs.js is a visual IDE for developing computational graphs, particularly designed for deep neural networks. Subgraphs is built with tensorflow.js, node, and react, and serves on Google Cloud.

• keras-js runs pre-trained keras deep learning models in the browser

  Run Keras models (trained using Tensorflow backend) in your browser, with GPU support. Models are created directly from the Keras JSON-format configuration file, using weights serialized directly from the corresponding HDF5 file.

  Inspiration is drawn from a number of deep learning / neural network libraries for JavaScript and the browser, including Tensorflow Playground, ConvNetJS, synaptic, brain, CaffeJS, MXNetJS. However, the focus of this library is on inference only.

  Tensor operations are extended on top of the ndarray library. GPU support is powered by WebGL through weblas.

• hardmaru presents an introduction to running sophisticated neural networks in the browser, targeted at artists

• mlweb

  This looks genuinely amazing in terms of functionality and even includes native support for worker threads and concurrency. However… it is lacking modern web wrappings such as npm packaging etc, so is not convenient to use from e.g. webpack.

  Twin to lalolib, a linear algebra library.

• Javascript inference and training: convnetjs

  • plus bonus interview
  • less active, sister project for recurrent networks: recurrentjs

• synapticjs is a full-featured javascript training, inference and visualisation system for neural network, with good documentation. Great learning resource, with plausible examples. Actively maintained.

  This library includes a few built-in architectures like multilayer perceptrons, multilayer long-short term memory networks (LSTM), liquid state machines or Hopfield networks, and a trainer capable of training any given network, which includes built-in training tasks/tests like solving an XOR, completing a Distracted Sequence Recall task or an Embedded Reber Grammar test, so you can easily test and compare the performance of different architectures.

• javascript inference only, neocortex.js in the browser. Civilised, but untouched since February 2016.

• brainjs is unmaintained now but looked like a simple javascript neural network library.

• mindjs is a simple one where you can see the moving parts.

• WebPPL is a successor to Church designed as a teaching language for probabilistic reasoning in the browser. Has hip features.

  • Runs on the command line with node.js or in the browser.
  • Supports modular and re-usable code using packages built on top of the npm package system, and interoperates with existing Javascript packages in the npm ecosystem.
  • Includes a large and expanding library of primitive distributions.
  • Implements a variety of inference algorithms, including exact inference via enumeration, rejection sampling, Sequential Monte Carlo, Markov Chain Monte Carlo, Hamiltonian Monte Carlo, and inference-as-optimization (e.g. variational inference).
  • Provides inference as a first-class operator in the language, allowing for nested inference (‘inference about inference’).
  • Supports optimizable models with neural network components using adnn.