Browser machine learning

1.1 TVM

• apache/tvm: Open deep learning compiler stack for CPU, GPU and specialized accelerators
• Apache TVM
• Apache TVM Unity: a vision for the ML software & hardware ecosystem in 2022

Apache TVM is a compiler stack for deep learning systems. It is designed to close the gap between productivity-focused deep learning frameworks and the performance- and efficiency-focused hardware backends. TVM works with deep learning frameworks to provide end-to-end compilation to different backends.

Impressive example: mlc-ai/web-stable-diffusion: Bringing stable diffusion models to web browsers. Everything runs inside the browser with no server support.

1.2 ONNX

microsoft/onnxruntime allows us to run neural nets in the browser:

ONNX Runtime Web is a Javascript library for running ONNX models on browsers and on Node.js.

ONNX Runtime Web has adopted WebAssembly and WebGL technologies for providing an optimized ONNX model inference runtime for both CPUs and GPUs.[…]

With ONNX Runtime Web, web developers can score models directly on browsers with various benefits including reducing server-client communication and protecting user privacy, as well as offering an install-free and cross-platform in-browser ML experience.

• Inference machine learning models in the browser with JavaScript and ONNX Runtime Web
• Deploy on web
• Build a web app with ONNX Runtime
• microsoft/onnxruntime-nextjs-template

1.3 Tensorflow.js

tensorflow.js is a less generic runtime, in that it supports primarily TensorFlow. NB it seems to support the superior jax.

I found Platform and environment explanatory.

tensorflow.js can load tensorflow checkpoints. “A WebGL accelerated, browser-based JavaScript library for training and deploying ML models.” More precisely, “TensorFlow.js, an ecosystem of JavaScript tools for machine learning, is the successor to deeplearn.js which is now called TensorFlow.js Core.” Includes lots of sample code for integration with the browser etc.

• subgraphs.js is a visual IDE for developing computational graphs, particularly designed for deep neural networks. Subgraphs is built with tensorflow.js, node, and react, and serves on Google Cloud.

• webcam transfer learning is an interesting essay in building a good UI for these things. It learns to use your webcam live.

• Teachable machine

  is a web-based tool that makes creating machine learning models fast, easy, and accessible to everyone.

  The models you make with Teachable Machine are real Tensorflow.js models that work anywhere javascript runs, so they play nice with tools like Glitch, P5.js, Node.js & more.

  Plus, export to different formats to use your models elsewhere, like Coral, Arduino & more.

1.4 Paddle.js

See 飞桨PaddlePaddle-源于产业实践的开源深度学习平台

1.5 WebNN

A W3C unifying API?

The WebNN API is a specification for constructing and executing computational graphs of neural networks. It provides web applications with the ability to create, compile, and run machine learning networks on the web browsers. The WebNN API may be implemented in web browsers using the available native operating system machine learning APIs for the best performance and reliability of results.

• webnn explainer
• WebNN

I am not sure that it is actually real, as opposed to a purely aspirational standard.

1.6 Unfiled tools

Have not updated the subsequent ones for quite some time and I suspect many of them are defunct.

• keras-js discontinued

• hardmaru presents an introduction to running sophisticated neural networks in the browser, targeted at artists

• mlweb

  This looks genuinely amazing in terms of functionality and even includes native support for worker threads and concurrency. However… it is lacking modern web wrappings such as npm packaging etc, so is not convenient to use from e.g. webpack.

  Twin to lalolib, a linear algebra library.

• Javascript inference and training: convnetjs

  • plus bonus interview
  • less active, sister project for recurrent networks: recurrentjs

• synapticjs is a full-featured javascript training, inference and visualization system for neural network, with good documentation. Great learning resource, with plausible examples. Actively maintained.

  This library includes a few built-in architectures like multilayer perceptrons, multilayer long-short term memory networks (LSTM), liquid state machines or Hopfield networks, and a trainer capable of training any given network, which includes built-in training tasks/tests like solving an XOR, completing a Distracted Sequence Recall task or an Embedded Reber Grammar test, so you can easily test and compare the performance of different architectures.

• neocortex.js discontinued

• mindjs is a simple one where you can see the moving parts.

• WebPPL is a successor to Church designed as a teaching language for probabilistic reasoning in the browser. Has hip features.

  • Runs on the command line with node.js or in the browser.
  • Supports modular and re-usable code using packages built on top of the npm package system, and interoperates with existing Javascript packages in the npm ecosystem.
  • Includes a large and expanding library of primitive distributions.
  • Implements a variety of inference algorithms, including exact inference via enumeration, rejection sampling, Sequential Monte Carlo, Markov Chain Monte Carlo, Hamiltonian Monte Carlo, and inference-as-optimization (e.g. variational inference).
  • Provides inference as a first-class operator in the language, allowing for nested inference (‘inference about inference’).
  • Supports optimizable models with neural network components using adnn.