Analysis/resynthesis of audio

Generative stochastic models for audio. Analyze audio using machine listening methods to decompose it into features, maybe over a sparse basis, as in learning gamelan, and possibly of low dimension due to some sparsification, source separation, maybe including some stochastic dependence, e.g. a random field or regression model of some kind. Then simulate features from that stochastic model. Depending on your cost function, how good your model fit was, and how you smoothed your data, this might produce something acoustically indistinguishable from the source, or perform concatenative synthesis from a sparse basis dictionary, or produce a parametric synthesizer software package.

There is a lot of funny business with machine learning for polyphonic audio. For a start, a naive linear-algebra-style decomposition doesn’t perform great because human acoustic perception is messy. e.g. all white noise sounds the same to us, but deterministic models need a large basis to minutely approximate it in the \(L_2\) norm. Our phase sensitivity is frequency dependent. Adjacent frequencies mask each other. Many other things I don’t know about. One could use cost functions based on psychoacoustic cochlear models, but those are tricky to synthesize from, (although possible if perhaps unsatisfying with a neural network). There are also classic alternate psychoacoustic decompositions such as the Mel Frequency Cepstral Transform, but these are even harder to invert.

1 Mosaicing synthesis

A.k.a concatenative synthesis. I’m publishing in this area.

• Mosaic Style Transfer using Sparse Autocorrelograms.

More soon.