AV controller interfaces

2.1 Soundcard input

Creating a controller from sound.

• Trigger finger
• Yehezkel Raz, Trigg.me! Audio triggers MIDI (M4L)

2.2 Tablet computers

iPad, Android, Windows tablet…

For Windows tablet, xotopad.

For iOS/Android, Touchosc, Lemur…

For anything +Ableton, yeco.

2.3 Sensel morph

A specialised tablet computer, https://morph.sensel.com/

2.4 Roli kit

Roli is a fancy audio company. How is their controller. Lightpad m.

3.1 Kinect

TBD

4.1 myo

myo (discontinued) is a wristband sensor that measures your muscles directly using EMG. Similar: the XTH using MMG — “which captures motion, direction and orientation sensors (integrated in a 9-DoF IMU) and muscle sound (also known as mechanomyogram or MMG)”

4.2 Leapmotion

Infrared hand tracker. In my experience, not really stable enough for on-stage use, (needs better Kalman filtering) but gee it’s small and portable and thus convenient.

4.3 Makeymakey

makeymakey

[Turns] everyday objects into touchpads and combine them with the internet. It’s a simple Invention Kit for Beginners and Experts doing art, engineering, and everything in between

Hmm. I’m not sold on this, as it’s a rather expensive way of getting 1-dimensional controllers out of $2 contact mics, and we could do a lot more with such inputs with meagre effort. Nice if you are short of time and quirk, but not short of money.

4.4 DIY options

• rockband keytar…

• optoforce 3D sensor Discontinued?

  • via Fredrik Olofsson
  • Australian distributor.

4.5 Wiimote

Another tiny and portable controller. Wiimote is almost a normal HID device, but has nasty sharp edges which means that it doesn’t really work out of the box as a music controller. So you avoid them using alternate libraries:

• wiiuse is a library written in C that connects with several Nintendo Wii remotes. Supports motion sensing, IR tracking, nunchuk, classic controller, Balance Board, and the Guitar Hero 3 controller. Single threaded and nonblocking makes a light weight and clean API.

• OS X mapper Darwiinremote.

• macOS driver wjoy

• osculator is a commercial product which does this; it’s pretty good.

• I used to use JunXion, but the creators were defunded

• libcwiid seems to be Linux-happy? But it’s a naked C library and apparently threading-tricky. Has a python interface.

  • nodewii uses it though for node linux
  • and node-wii-controller

4.6 webcam

See synestizer.

4.7 Portable wireless routers

Portable router solution for network device control?

6.1 MIDI

More-or-less working since the 1980s; still the best idea, if you can live with 7-bit scalars as your lingua franca. See MIDI.

6.2 OpenSoundControl

tl;dr No.

A short-lived project from the 1990s to produce a more flexible protocol than MIDI. Insinuated its way into many projects before death, and still haunts them. Because it is more flexible than MIDI, it is sometimes discussed as if it were the apotheosis of protocols, as opposed to half-baked system which incrementally improves on MIDI in some ways while introducing debilities of its own. Here are the debilities I have encountered:

1. Stateless protocol designed to support UDP — and therefore it’ one way. No query-and-response here, no clients and servers. One side needs to transmit the other. Then the other one needs to transmit back. These are inherently unrelated. Much paperwork must be done manually to ensure that on each side you are each responding to each other, have not missed anything and are responding to things in the correct order. All of this is stuff that TCP does for free. Therefore I always end up re-inventing TCP if I want to do 2 way communication, much as in most UDP systems Which presumably I do, or U’d be using MIDI.
2. because of UDP assumption, OSC is bad for transferring large data, e.g. audio, because of UDP packet size limits. Seems unfortunate for an audio application domain.
3. supports strings but only ASCII strings, so we can’t even work around (2) by transmitting filenames, which are not reliably ASCII on modern computers without onerous workarounds.
4. supports lots of data types, including nested data trees and time execution, but only a partial subset is implemented by most software apart from, AFAICT, supercollider.
5. has a, for my purposes, rather unpleasant addressing protocol, where messages can be addressed to paths including wildcards. But the sender may use wildcards, and the receiver may not, which is backwards for the practical use-cases that I personally encounter.
6. Doesn’t guarantee delivery, due to UDP assumption. When I mention this, supercollider advocates tell me that I CAN in fact use TCP instead. Which I can, but AFAICT only if I am using supercollider, which rather diminishes the force of the “universal ultimate controller protocol for everything” argument.

The upside of all these limitations, people tell me, is that OSC is light enough to implement on microcontrollers. Which is sort of true, in a useless sense, in that the entire protocol is heavy, but one can often implement a limited subset of the protocol on a microcontroller. No two devices I have seen so far implement the same subset, of course, and there is no protocol to communicate which subset is implemented, natch.) If all the microprocessors are implementing a subset anyway, why then must OSC at its most general be crippled? Even the broadest and most general implementation of the specification does not include the most basic comforts of modern networked communication. This confusing theatre where giant modern digital workstations ape tiny embedded controllers in an unhelpful charade of solidarity is not a great use of developer time or processor time.

Rather than providing a universal communication system, then, it requires us to embark upon a complicated compatibility negotiation, all to gain marginal functionality compared to MIDI. And at the same time, cheap entry-level microcontrollers implement much more mainstream protocols than this (e.g. HTTP). In trying to fulfil the needs of microscopically tiny and/or archealogically ancient microcontrollers and bigger computers at once, it fails to do either.

If you find yourself thinking OSC is a solution, perhaps you will do better to have big devices using web APIs talk to dumb peripherals using the same protocol, or in a pinch MIDI, or some other IoT protocol, backed by a highly-resourced IoT corporation, rather than a community-less fossilized standard that has been abandonware for more than 2 decades without even the mindshare or support of MIDI.

6.3 HID

How your keyboard talks to your computer. 🏗

6.4 Websockets

🏗

6.5 Sundry bluetooth protocols

🏗