AV controller interfaces

The input-data side for gesture recognition or controller mapping.

In non-musical circles they call this “physical computing”, or “natural user interfaces”, or “tangible computing”, depending upon whom they are pitching to for funding this month.


  • Less a software package than a protocol, OSCQuery attempts to automatically construct GUIs for remote control of AV apps. Flagship useful application is the automatic webapp interface, oscqueryhtml. (This is an example of the reinvention of TCP that I mention when discussing OSC).

  • libmapper bundles together UI signals and provides a discovery protocol; libraries in C right now; also there is python, and some puredata/maxMSP implementation. There is also some query support, much like OSCQuery.

    But if you were doing that, why not use some IoT tools and benefit from greater brainshare?

  • libossia is

    libossia is a modern C++, cross-environment distributed object model for creative coding.

    It allows to declare the architecture of your creative coding application’s functions as a tree of OSC nodes and parameters. These nodes/parameters can have attributes, which allow to declare many of their properties, such as their values, types, units, ranges, etc…

    This OSC tree-based architecture (coined “device” in the OSSIA terminology) can then be exposed over the network under several protocols, some of which allow this architecture, and the properties and values of its nodes, to be fully explored and queried. For now, protocols available in the implemenations are: plain OSC, OSCquery, Midi - more are part of libossia and will be made available in the future.

    libossia offers bindings and implementations for several environments: PureData, Max/MSP, Python, Unity3D, QML, Faust, SuperCollider.

    It is paired with an app that seems to use this for cross-media control, ossia. See below for a length rant on the theme of why I dislike OSC.

  • The Autobahn project:

    provides open-source implementations of the The WebSocket Protocol and The Web Application Messaging Protocol (WAMP) network protocols.

    WebSocket allows bidirectional real-time messaging on the Web and WAMP adds asynchronous Remote Procedure Calls and Publish & Subscribe on top of WebSocket.

    WAMP is ideal for distributed, multi-client and server applications, such as multi-user database-driven business applications, sensor networks (IoT), instant messaging or MMOGs (massively multi-player online games).

    includes javascript, python and a routing infrastructure called crossbar.

  • Luis Lloret’s OSMID aims to provide a lightweight, portable, easy to use tool to convert MIDI to OSC and OSC to MIDI.

  • Mark Francombe’s browser MIDI/OSC converter, MIDI MESSAGE GENERATOR.

  • OSC and python

    • The original, sorta-working thing: pyOSC
    • C-based and easier to use: pyliblo
  • tangible.js is a resource for real-world interfaces plugging into to javascript. They intermittently publish useful reviews.

  • There are various Javascript GUI frameworks designed for musical control.

  • Javascript OpenSoundControl bridges

    • osc.js is an Open Sound Control (OSC) library for JavaScript that works in both the browser and Node.js (And is still being maintained unlike many)

    • supercollider.js does this and much more.

    • OSC-JS exists, bridging websockets to OSC, but doesn’t look as maintained as osc.js. Are there others?

    • Yes. Legato.

      legato is a small node.js library written in coffeescript, but that doesn’t really matter. legato is designed to let you create beautifully simple connections between devices and software, regardless of environment.

  • Löve2d. Reasonably comprehensive support for MIDI (Lua)

  • HIDtoOSC

    a human-interface-device (HID) to open-sound-control (OSC) converter for macOS written in Python3. the program can send OSC to any IP and port but by default it will send to (localhost) on port 57120 (supercollider).

Interesting hardware

Soundcard input

Creating a controller from sound.

Tablet computers

iPad, Android, Windows tablet…

For Windows tablet, xotopad.

For iOS/Android, Touchosc, Lemur…

For anything +Ableton, yeco.

Sensel morph

A specialised tablet computer, https://morph.sensel.com/

Roli kit

Roli is a fancy audio company. How is their controller. Lightpad m.

Keith McMillen

Keith McMillen Instruments makes robust midi controllers out of some weird rubbery membrane.

I am a fan of the Quneo which is nearly indestructible. The bitwig control script is good.



3d interaction

The classic depth camera is the Kinect. More-open depth-camera: Orbbec3d

Calibration is tricky; Rulr attempts to solve this in an open-source, general way. (Rulr docs). openkinect does Kinect.



myo (discontinued) is a wristband sensor that measures your muscles directly using EMG. Similar: the XTH using MMG - “which captures motion, direction and orientation sensors (integrated in a 9-DoF IMU) and muscle sound (also known as mechanomyogram or MMG)”


Infrared hand tracker. In my experience, not really stable enough for on-stage use, (needs better Kalman filtering) but gee it’s small and portable and thus convenient.



[Turns] everyday objects into touchpads and combine them with the internet. It’s a simple Invention Kit for Beginners and Experts doing art, engineering, and everything in between

Hmm. I’m not sold on this, as it’s a rather expensive way of getting 1-dimensional controllers out of $2 contact mics, and you could do a lot more with this if you were clever. Nice if you are short of time and quirk, but not short of money,

Misc ghetto options


Another tiny and portable controller. Wiimote is almost a normal HID device, but has nasty sharp edges which means that it doesn’t really work out of the box as a music controller. So you avoid them using alternate libraries:

  • wiiuse is a library written in C that connects with several Nintendo Wii remotes. Supports motion sensing, IR tracking, nunchuk, classic controller, Balance Board, and the Guitar Hero 3 controller. Single threaded and nonblocking makes a light weight and clean API.

  • OS X mapper Darwiinremote.

  • macOS driver wjoy

  • osculator is a commercial product which does this; it’s pretty good.

  • libcwiid seems to be Linux-happy? But it’s a naked C library and apparently threading-tricky. Has a python interface.


See synestizer.


Human Instruments does good accessibile interface work.



More-or-less working since the 1980s; still the best idea, if you can live with 7-bit scalars as your lingua franca. See MIDI.


tl;dr No.

A short-lived project from the 1990s to produce a more flexible protocol than MIDI. Insinuated its way into many projects before death, and still haunts them. Because it is more flexible than MIDI, it is sometimes discussed as if it were the apotheosis of protocols, as opposed to half-baked system which incrementally improves on MIDI in some ways while introducing debilities of its own. Here are the debilities I have encountered:

  1. Stateless protocol designed to support UDP — and therefore it’s a one way. No query-and-response here, no clients and servers. One side needs to transmit the other. Then the other one needs to transmit back. These are inherently unrelated. Much paperwork must be done manually to ensure that on each side you are each responding to each other, have not missed anything and are responding to things in the correct order. All of this is stuff that TCP does for free. Therefore you always end up re-inventing TCP if you want to do 2 way communication. Which presumably you do, or you’d be using MIDI.
  2. because of UDP assumption, OSC is bad for transferring large data, e.g. audio, which seems unfortunate for an audio application domain.
  3. supports strings but only ASCII strings, so you can’t even work around (2) by transmitting filenames, which are not reliably ASCII on modern computers without onerous workarounds.
  4. supports lots of data types, including nested data trees and time execution, but only a partial subset is implemented by most software apart from, AFAICT, supercollider.
  5. has a rather unpleasant addressing protocol, where messages can be addressed to paths including wildcards. But the sender may use wildcards, and the receiver may not, which is backwards for all practical use-cases that I personally encounter.
  6. Doesn’t guarantee delivery, due to UDP assumption. When you mention this, supercollider fans tell you that you CAN in fact use TCP instead. Which you can, but AFAICT only if you are using supercollider, which rather diminishes the force of the “universal ultimate controller protocol for everything” argument. And then, AFAICT it still behaves as if you are using UDP so this does not in any case help.

The upside of all these limitations, people tell you, is that it is light enough to implement OSC on microcontrollers. Which is sort of true, in a useless sense. In practice, OSC is complicated enough that it is burdensome in both developer time and processor cycles, so you implement a very limited subset, and no two devices I hav yet encountered implement the same subset. Rather than providing a universal communication system, then, it requires you to embark upon a complicated compatibility negotiation, all to gain marginal functionality compared to MIDI. At the same time, even the broadest and most general implementation of the specification does not include the most basic comforts of modern networked communication. And at the same time, cheap entry-level microcontrollers implement much more mainstream protocols than this (e.g. HTTP). In trying to fulfill the needs of microscopically tiny and/or archeologically ancient microcontrollers and bigger computers at once, it fails to do either.

If you find yourself thinking OSC is a solution, perhaps you will do better to have big devices using web APIs talk to dumb peripherals using the same protocol, or in a pinch MIDI, or some other IoT protocol, backed by a faceless IoT corporation, rather than a standard that has been abandonware for more than 2 decades and adds nothing apart from an illusion of compatibility.


How your keyboard talks to your computer. 🏗



Sundry bluetooth protocols


No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.