Auditory features

descriptors, maps, representations for audio

November 13, 2019 — January 14, 2021

computers are awful
machine learning
machine listening
making things
signal processing
Figure 1

In machine listening and related tasks likes audio analysis we often want compact representations of audio signals in some manner that is not “raw”; something a little more useful than the simple record of the vibrations of the microphone as given by the signal pressure level time series. We might want a representation, for example, that tells us something about psychoacoustics, i.e. the parts of the sound that are relevant to a human listener. We might want to use these features to construct a musical metrics which tells us something about perceptual similarity or something.

These representations are called features or descriptors. This is a huge industry - compact summary features, for example, make audio convenient and compact for transmission. (hello mobile telephony, MP3) But this trick is also useful for understanding speech, music etc. There are as many descriptors as there are IEEE conference slots.

See, say, (Alías, Socoró, and Sevillano 2016) for an intimidatingly comprehensive summary.

I’m especially interested in

1 Just ask someone

There are protocols that measure musical similarity by asking human listeners to rate similarity.

MUSHRA is a useful software package/protocol.

2 Deep neural network feature maps

See, e.g. Jordi Pons’ Spectrogram CNN discussion for some introductions to the kind of features a neural network might “discover” in audio recognition tasks.

There is some interesting work on that; for example, Dieleman and Schrauwen (Dieleman and Schrauwen 2014) show that convolutional neural networks trained on raw audio (i.e. not spectrograms) for music classification recover Mel-like frequency bands. Thickstun et al (Thickstun, Harchaoui, and Kakade 2017) do some similar work.

And Keunwoo Choi shows that you can listen to what they learn.

There is also csteinmetz1/auraloss: Collection of audio-focused loss functions in PyTorch

3 Sparse comb filters

Differentiable! Conditionally invertible! Handy for syncing.

Moorer (Moorer 1974) proposed these for harmonic purposes, but Robertson et al (Robertson, Stark, and Plumbley 2011) have shown it to be handy for rhythm.

4 Autocorrelation features

Measure the signal’s full or partial autocorrelation with itself. Very nearly the power spectrum, thanks to the Wiener-Khintchine theorem.

5 Power spectrum

Power spectral density.

6 Bispectrum

see wikipedia

7 Linear Predictive coefficents

How do these transform? If we did this as all-pole or all-zeros might be useful; but many maxima.

8 Scattering transform coefficients

See scattering transform.

9 Cepstra

Classic, but inconvenient to invert.


Mel-frequency Ceptral Coefficients, or Mel Cepstral transform. Take the perfectly respectable-if-fiddly cepstrum and make it really messy, with a vague psychoacoustic model in the hope that the distinctions in the resulting “MFCC” might correspond to distinctions correspond to human perceptual distinctions.

Folk wisdom holds that MFCC features are Eurocentric, in that they destroy, or at least obscure, tonal language features. Ubiquitous, but inconsistently implemented; MFCCs are generally not the same across implementations, probably because the Mel scale is itself not universally standardised.

Asides from being loosely psychoacoustically motivated features, what do the coefficients of an MFCC specifically tell me?

Hmm. If I have got this right, these are “generic features”; things we can use in machine learning because we hope they project the spectrum into a space which approximately preserves psychoacoustic dissimilarity, whilst having little redundancy.

This heuristic pro is weighted with the practical con that they are not practically differentiable, nor invertible except by heroic computational effort, nor are they humanly interpretable, and riven with poorly-supported somewhat arbitrary steps. (The cepstrum of the Mel-frequency-spectrogram is a weird thing that no longer picks out harmonics in the way that God and Tukey intended.)

11 Filterbanks

Inc bandpasses, Gammatones… Random filterbanks?

12 Dynamic dictionaries

See sparse basis dictionaries.

13 Cochlear activation models


14 Units

Erbs, Mels, Sones, Phones… There are various units design to approximate the human perceptual process; I tried to document them under psychoacoustics.

15 References

Abe, Kobayashi, and Imai. 1995. Harmonics Tracking and Pitch Extraction Based on Instantaneous Frequency.” In International Conference on Acoustics, Speech, and Signal Processing, 1995. ICASSP-95.
Alías, Socoró, and Sevillano. 2016. A Review of Physical and Perceptual Feature Extraction Techniques for Speech, Music and Environmental Sounds.” Applied Sciences.
Anglade, Benetos, Mauch, et al. 2010. Improving Music Genre Classification Using Automatically Induced Harmony Rules.” Journal of New Music Research.
Bogert, Healy, and Tukey. 1963. “The Quefrency Alanysis of Time Series for Echoes: Cepstrum, Pseudo-Autocovariance, Cross-Cepstrum and Saphe Cracking.” In.
Chen, and Wang. n.d. “High-Level Music Descriptor Extraction Algorithm Based on Combination of Multi-Channel Cnns and Lstm.” In Proceedings of the 18th International Society for Music Information Retrieval Conference (ISMIR’2017), Suzhou, China.
Childers, Skinner, and Kemerait. 1977. The Cepstrum: A Guide to Processing.” Proceedings of the IEEE.
Choi, Fazekas, Sandler, et al. 2017. Transfer Learning for Music Classification and Regression Tasks.” In Proceeding of The 18th International Society of Music Information Retrieval (ISMIR) Conference 2017.
Defferrard, Benzi, Vandergheynst, et al. 2017. FMA: A Dataset For Music Analysis.” In Proceedings of the 18th International Society for Music Information Retrieval Conference (ISMIR’2017), Suzhou, China.
Dieleman, and Schrauwen. 2014. End to End Learning for Music Audio.” In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
Grosche, Muller, and Kurth. 2010. Cyclic Tempogram - a Mid-Level Tempo Representation for Music Signals.” In 2010 IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP).
Lattner, Dorfler, and Arzt. 2019. Learning Complex Basis Functions for Invariant Representations of Audio.” In Proceedings of the 20th Conference of the International Society for Music Information Retrieval.
Luo, Chen, Hershey, et al. 2016. Deep Clustering and Conventional Networks for Music Separation: Stronger Together.” arXiv:1611.06265 [Cs, Stat].
MacKinlay, and Botev. 2019. Mosaic Style Transfer Using Sparse Autocorrelograms.” In Proceedings of the 20th Conference of the International Society for Music Information Retrieval.
Makhoul, Kubala, Schwartz, et al. 1999. “Performance Measures For Information Extraction.” In In Proceedings of DARPA Broadcast News Workshop.
McDermott, Schemitsch, and Simoncelli. 2013. Summary Statistics in Auditory Perception.” Nature Neuroscience.
Moorer. 1974. The Optimum Comb Method of Pitch Period Analysis of Continuous Digitized Speech.” IEEE Transactions on Acoustics, Speech and Signal Processing.
Noll. 1967. Cepstrum Pitch Determination.” The Journal of the Acoustical Society of America.
Oppenheim, and Schafer. 2004. From Frequency to Quefrency: A History of the Cepstrum.” IEEE Signal Processing Magazine.
Phan, Hertel, Maass, et al. 2016. Robust Audio Event Recognition with 1-Max Pooling Convolutional Neural Networks.” In Interspeech 2016.
Pons, and Serra. 2017. Designing Efficient Architectures for Modeling Temporal Features with Convolutional Neural Networks.” In.
Robertson, Stark, and Plumbley. 2011. Real-Time Visual Beat Tracking Using a Comb Filter Matrix.” In Proceedings of the International Computer Music Conference 2011.
Rochebois, and Charbonneau. 1997. Cross-Synthesis Using Interverted Principal Harmonic Sub-Spaces.” In Music, Gestalt, and Computing. Lecture Notes in Computer Science 1317.
Salamon, Serrà, and Gómez. 2013. Tonal Representations for Music Retrieval: From Version Identification to Query-by-Humming.” International Journal of Multimedia Information Retrieval.
Schlüter, and Böck. 2014. Improved Musical Onset Detection with Convolutional Neural Networks.” In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
Schmidt, and Kim. 2011. Learning Emotion-Based Acoustic Features with Deep Belief Networks.” In 2011 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA).
Scholler, and Purwins. 2011. Sparse Approximations for Drum Sound Classification.” IEEE Journal of Selected Topics in Signal Processing.
Smith, Evan, and Lewicki. 2005. Efficient Coding of Time-Relative Structure Using Spikes.” Neural Computation.
Smith, Evan C., and Lewicki. 2006. Efficient Auditory Coding.” Nature.
Southall, Wu, Lerch, et al. 2017. MDB Drums — An Annotated Subset of MedleyDB for Automatic Drum Transcription.” In Late Breaking Demo (Extended Abstract), Proceedings of the International Society for Music Information Retrieval Conference (ISMIR).
Thickstun, Harchaoui, and Kakade. 2017. Learning Features of Music from Scratch.” In Proceedings of International Conference on Learning Representations (ICLR) 2017.
Wu, and Lerch. 2017. Automatic Drum Transcription Using the Student-Teacher Learning Paradigm with Unlabeled Music Data.” In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR).