Musical metrics and manifolds

September 26, 2014 — March 27, 2017

machine learning
making things

Metrics, kernels and affinities and the spaces and topologies they induce and what they reveal (or prescribe) about composition. This has considerable overlap with machine listening, but there I start from audio signals, and here I usually think about more “symbolic” data such as musical scores, and with rhythm, but there I care only about the time axis. There is overlap also with psychoacoustic units and auditory features which are the pure elements from which these amalgams are made.

One thing which has become outdated in this notebook since I wrote it is that these days I would naturally consider learned metrics, rather an a priori ones.

This is a specialized area with many fine careers built on it, and many schools with long and proud histories of calling adherents of other schools wrong. A short literature search will find many worth distinctions drawn between differences, cultural versus the biological, general versus individual, the different models applicable at different time scales, of masking effects and such.

I will pass over these fine contrasts without note, in my quest for some pragmatically useful, minimally complex features for using in machine-learning algorithms, which do not require truth but functionality.

See also machine listening, Arpeggiate by Numbers, clustering, manifold learning.

1 General musically-motivated metric spaces

Chords etc.

Consider grouping chords played on a piano with some known (idealised) 16-harmonic spectrum. How can we understand the harmonic relations of chords played upon this keyboard?

We know a-priori that this projection can be naturally embedded in a Euclidean space with something less than \(16\times 12 = 192\) dimensions, since there are at most that many spectral bands. In fact, since there are only 12 notes, we can get this down to 12 dimensions. The question is, can we get to an even smaller number of dimensions? How small? By “small” here, I mean, can I group chords on a lower number of dimensions than this in some musically interesting way? For bonus points, can I group chords into several different such maps and switch between them?

Without explanation, here is one similarity-distance embedding of all the chords using an ad hoc metric based on thinking about this question. (Brightness: The more notes in the chord, the darker. Hue: I forget.)

Question: can we use word-bag models for note adjacency representation? Since I asked that question I discovered the Chord2vec model of Walder’s lab, whose answer is “yes”. See Madjiheurem, Qu, and Walder (2016).

Figure 2: 3 dimensional MDS 1
Figure 3: 3 dimensional MDS 2

2 To understand

  • Basic group theory: What are the symmetries of the group of all pitch class co-occurrences in the 12-tet? The most common pitch class co-occurrence is a 12-vector over the Boolean algebra where a “1” in the first entry means “the note C was played”, in pos 2, “C-sharp was played” etc. Symmetries we care about are, for example, that (under equal temperament) chord harmonic relationships should be invariant under transposition — i.e. rotation of the entries of the vector. (this last is not so in the case of Sethares/Terhardt style dissonance theory, which considers only unwrapped harmonics.)

  • Dmitri Tymoczko’s Geometrical Methods in Recent Music Theory

  • Henjan Honing’s Musical Cognition: any good?

3 Dissonance

a.k.a. “Spectral roughness”. A particular empirically-motivated metric, with good predictive performance despite its simplicity, and wilful lack of concern for the actual mechanisms of the ear and the brain, or modern nuances such as masking effects and the influence of duration on sound perception etc. Invented by Plomp and Levelt (Plomp and Levelt 1965), and developed by variously, Sethares, Terhardt and Parncutt and others.

Some sources seem to distinguish roughness in the sense of Sethares from the Plomp and Levelt sense, although they use qualitatively equations. I suspect therefore that the distinction is philosophical, or possibly pointed failure to cite one another because someone said something rude at the after-conference drinkies.

An overview by Vassilakis might help, or the app based on his work by Dr Kelly Fitz.

Plomp and Levelt’s dissonance curves ((Plomp and Levelt 1965; Sethares 1997)).

Figure 4: Plomp and Levelt’s dissonance curves

Juan Sebastian Lach Lau produced some actual open-source software (DissonanceLib) that attempts to action this stuff in a musical setting. There is a slightly different version below attributed to Parncutt.

A convenient summary of both is in the DissonanceLib code.

It’s most useful for things where you are given the harmonics a priori; I’m not especially convinced about the tenability of directly inferring this metric from an audio signal (“how dissonant is this signal?”). We should be cautious about the identifiability of this statistic from signals nonparametrically e.g. windowed DTFT power-spectrogram peaks, just because beat frequency stuff is complicated and runs into the uncertainty principle. (Terhardt, Stoll, and Seewann 1982; Sethares 1998b) give it a go, though. Inferring dissonance between two signals known to be not dissonant might work though, or perhaps on might need parametric approaches, as in linear system identification

Dissonance an interesting measure, despite these problems, though because it is very much like a Mercer kernel, in that it constructs a distance defined on an (explicit) high-dimensional space; Also, the “nearly circular” geometry it induces is interesting; For harmonic spectra, you recover the equal-tempered 12-tone scale and the 2:1 octave by minimising dissonance between twelve notes with harmonic spectra (i.e. plucked string spectra), which is suggestive that it might do other useful things.

Also, it’s almost-everywhere differentiable with respect to your signal parameters, which makes fitting it or optimising its value easy.

David E Verotta’s thoughtfully illustrated an essay on this.

Anyway, details.

3.1 Plomp and Levelt’s dissonance curves

Attributed to Plomp and Levelt’s (Plomp and Levelt 1965), here is Sethares’ version (Sethares 1998a), also summarised on Sethares’ web page.

Dissonance between two pure sinusoidal frequencies, \(f_1 \leq f_2\), with amplitudes respectively \(v_1, v_2\), is given by:

\[ d_\text{PL}(f_1,f_2, v_1,v_2) := v_1v_2\left[ \exp\left( -as(f_2-f_1) \right) - \exp\left( -bs(f_2-f_1) \right) \right] \]


\[ s=\frac{d^*}{s_1 f_1+s_2} \]

and \(a=3.5, b=5.75, d^*=.24, s_1=0.21, s_2= 19\), the constants being fit by least-squares from experimental data.

If your note has more than one frequency, one sums the pairwise dissonances of all contributing frequencies to find the total dissonance, which is not biologically plausible but seems to work ok. Other ways of working out differences between two composite sounds could be possible (Hausdorff metric etc).

This looks to me like the kind of situation where the actual details of the curve are not so important as getting the points of maximal and minimal dissonance right. Clearly we have a minimal value at \(f_1=f_2\). We solve for the maximally dissonant frequency \(f_2\) with respect to a fixed \(f_1, v_1, v_2\):

\[\begin{aligned} -as\exp( -as(f_2-f_1) ) &= -bs\exp( -bs(f_2-f_1) )\\ a\exp( -as(f_2-f_1) ) &= b\exp( -bs(f_2-f_1) )\\ \ln a - as(f_2-f_1) &= \ln b -bs(f_2-f_1)\\ \ln a - \ln b &= as(f_2-f_1) -bs(f_2-f_1)\\ \ln a - \ln b &= s(a-b)(f_2-f_1) \\ f_2 &= f_1+\frac{\ln b - \ln a}{s(b-a)}\\ f_2 &= f_1(s_1+C)+s_2C \end{aligned}\]


\[ C:=\frac{\ln b - \ln a}{d^*(b-a)} \]

That affine difference is reminiscent of resolvability criteria in functional bases.

3.2 Parncutt and Barlow dissonance

Differences of exponentials are computationally tedious because of numerical concerns with large frequency values; this is suggestive of approximation by something more convenient, maybe of this form:

\[ d_\text{simple}(f_1,f_2,v_1, v_2):=C_1(f_2-f_1)\exp -C_2(f_2-f_1) \]

The Parncutt approximation takes this approach and additionally transforms the units into heuristically preferable psychoacoustic ones.

Cribbed from Lach Lau’s source code and thesis, where he attributes it to Parncutt and Barlow, although I can’t find any actual articles by Parncutt and/or Barlow which use this. (Mashinter 2006) implies it might be unpublished. (Hansen 2014) gives a squared version of the same formula.

For this we take frequencies \(b_1\leq b_2\) and volumes \(s_1, s_2\) in, respectively, barks and sones. Then

\[\begin{aligned} d_\text{PB}(b_1, b_2, s_1, s_2) &:=\sqrt{(s_1 s_2)}(4 ( b_2- b_1) \exp(1 - 4 ( b_2- b_1)))\\ &= \sqrt{(s_1 s_2)}(4 ( b_2- b_1) e \exp( - 4 ( b_2- b_1))) \end{aligned}\]

Since this scale is relative, I’m not quite sure why we have constants everywhere.

\[ d_\text{PB}'(b_1, b_2, s_1, s_2) := \sqrt{(s_1 s_2)}\frac{ b_2- b_1}{ \exp(b_2-b_1)}? \]

Possibly in order to more closely approximate Sethares?

4 Induced topologies

🏗 Nestke (2004) and Mazzola (2012). Tymozcko.

5 References

Barlow, and Lohner. 1987. Two Essays on Theory.” Computer Music Journal.
Bartlett, and Medhi. 1955. On the Efficiency of Procedures for Smoothing Periodograms from Time Series with Continuous Spectra.” Biometrika.
Bigand, and Parncutt. 1999. Perceiving Musical Tension in Long Chord Sequences.” Psychological Research.
Bigand, Parncutt, and Lerdahl. 1996. Perception of Musical Tension in Short Chord Sequences: The Influence of Harmonic Function, Sensory Dissonance, Horizontal Motion, and Musical Training.” Perception & Psychophysics.
Bigo, Giavitto, and Spicher. 2011. Building Topological Spaces for Musical Objects.” In Proceedings of the Third International Conference on Mathematics and Computation in Music. MCM’11.
Bingham, Godfrey, and Tukey. 1967. Modern Techniques of Power Spectrum Estimation.” Audio and Electroacoustics, IEEE Transactions on.
Bod. 2002. Memory-Based Models of Melodic Analysis: Challenging the Gestalt Principles.” Journal of New Music Research.
Boggs, and Rogers. 1990. Orthogonal Distance Regression.” Contemporary Mathematics.
Budney, and Sethares. 2014. Topology of Musical Data.” Journal of Mathematics and Music.
Callender, Quinn, and Tymoczko. 2008. Generalized voice-leading spaces.” Science (New York, N.Y.).
Cancho, and Solé. 2003. Least Effort and the Origins of Scaling in Human Language.” Proceedings of the National Academy of Sciences.
Carlos. 1987. Tuning: At the Crossroads.” Computer Music Journal.
Casey, M., Rhodes, and Slaney. 2008. Analysis of Minimum Distances in High-Dimensional Musical Spaces.” IEEE Transactions on Audio, Speech, and Language Processing.
Casey, M.A., Veltkamp, Goto, et al. 2008. Content-Based Music Information Retrieval: Current Directions and Future Challenges.” Proceedings of the IEEE.
Cook, Fujisawa, and Konaka. 2007. Why Not Study Polytonal Psychophysics? Empirical Musicology Review.
Cooper, and Fazio. 1984. A New Look at Dissonance.” Advances in Experimental Social Psychology.
Corral, León, and Liern. 2009. Compatibility of the Different Tuning Systems in an Orchestra.” In Mathematics and Computation in Music. Communications in Computer and Information Science 38.
Cousineau, McDermott, and Peretz. 2012. The Basis of Musical Consonance as Revealed by Congenital Amusia.” Proceedings of the National Academy of Sciences.
de Cheveigné. 2005. Pitch Perception Models.” In Pitch. Springer Handbook of Auditory Research 24.
Demaine, Gomez-Martin, Meijer, et al. 2005. The Distance Geometry of Deep Rhythms and Scales. In CCCG.
———, et al. 2009. The Distance Geometry of Music.” In Computational Geometry.
Demopoulos, and Katchabaw. 2007. Music Information Retrieval: A Survey of Issues and Approaches.”
Duffin. 1948. Function Classes Invariant Under the Fourier Transform.” Duke Mathematical Journal.
Du, Kibbe, and Lin. 2006. Improved Peak Detection in Mass Spectrum by Incorporating Continuous Wavelet Transform-Based Pattern Matching.” Bioinformatics.
Ferguson, and Parncutt. 2004. Composing In the Flesh: Perceptually-Informed Harmonic Syntax.” In Proceedings of Sound and Music Computing.
Févotte, Bertin, and Durrieu. 2008. Nonnegative Matrix Factorization with the Itakura-Saito Divergence: With Application to Music Analysis.” Neural Computation.
Flamary, Févotte, Courty, et al. 2016. Optimal Spectral Transportation with Application to Music Transcription.” In arXiv:1609.09799 [Cs, Stat].
Fokker. 1969. Unison Vectors and Periodicity Blocks in 3-Dimensional (3-5-7-) Harmonic Lattice of Notes.” Proceedings of the Koninklijke Nederlandse Akademie van Wetenschappen Series b-Physical Sciences.
Gashler, and Martinez. 2011. Tangent Space Guided Intelligent Neighbor Finding.” In.
———. 2012. Robust Manifold Learning with CycleCut.” Connection Science.
Hall. 2008. Geometrical Music Theory.” Science.
Hansen. 2014. Spatial Utilization of Sensory Dissonance and the Creation of Sonic Sculpture.” In.
Haussler. 1999. Convolution Kernels on Discrete Structures.”
Hawe, Kleinsteuber, and Diepold. 2013. Analysis Operator Learning and Its Application to Image Reconstruction.” IEEE Transactions on Image Processing.
Hemmen, and Vollmayr. 2013. Resonating Vector Strength: What Happens When We Vary the ‘Probing’ Frequency While Keeping the Spike Times Fixed.” Biological Cybernetics.
Hermes. 1988. Measurement of Pitch by Subharmonic Summation.” The Journal of the Acoustical Society of America.
Honingh, and Bod. 2011. In Search of Universal Properties of Musical Scales.” Journal of New Music Research.
Huron. 1994. Interval-Class Content in Equally Tempered Pitch-Class Sets: Common Scales Exhibit Optimum Tonal Consonance.” Music Perception: An Interdisciplinary Journal.
Huron, and Parncutt. 1993. An Improved Model of Tonality Perception Incorporating Pitch Salience and Echoic Memory. Psychomusicology: A Journal of Research in Music Cognition.
Jewell, Rhodes, and d’Inverno. 2010. Querying improvised music: Do you sound like yourself? In ISMIR.
Kameoka, and Kuriyagawa. 1969a. Consonance Theory Part I: Consonance of Dyads.” The Journal of the Acoustical Society of America.
———. 1969b. Consonance Theory Part II: Consonance of Complex Tones and Its Calculation Method.” The Journal of the Acoustical Society of America.
Krumhansl. 2000. Rhythm and Pitch in Music Cognition. Psychological Bulletin.
Kuswanto, Heru. 2011. Comparison Study of Saron Ricik Instruments’ Sound Color (Timbre) on the Gamelans of Nagawilaga and Gunturmadu from.” International Journal of Basic & Applied Sciences.
Kuswanto, H. 2012. Saron Demung’s Timbre and Sonogram of Gamelans Gunturmadu from Keraton Ngayogyakarta.” Jurnal Pendidikan Fisika Indonesia.
Leman. 1997. Music, Gestalt, and Computing: Studies in Cognitive and Systematic Musicology.
Lerdahl. 1988. Tonal Pitch Space.” Music Perception: An Interdisciplinary Journal.
———. 1996. Calculating Tonal Tension.” Music Perception: An Interdisciplinary Journal.
Levitin, Chordia, and Menon. 2012. Musical rhythm spectra from Bach to Joplin obey a 1/f power law.” Proceedings of the National Academy of Sciences of the United States of America.
Li. 2006. The Effect of Inharmonic and Harmonic Spectra in Javanese Gamelan Tuning (1): A Theory of the Sléndro.” In Proceedings of the 7th WSEAS International Conference on Acoustics & Music: Theory & Applications.
Macke, Berens, Ecker, et al. 2009. Generating spike trains with specified correlation coefficients.” Neural Computation.
Madjiheurem, Qu, and Walder. 2016. Chord2Vec: Learning Musical Chord Embeddings.”
Mashinter. 2006. Calculating Sensory Dissonance: Some Discrepancies Arising from the Models of Kameoka & Kuriyagawa, and Hutchinson & Knopoff.”
Mazzola. 2012. Singular Homology on Hypergestures.” Journal of Mathematics and Music.
McFee, and Ellis. 2011. Analyzing Song Structure with Spectral Clustering.” In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Moustafa, Schuurmans, and Ferrie. 2013. Learning a Metric Space for Neighbourhood Topology Estimation: Application to Manifold Learning.” In Journal of Machine Learning Research.
Nicolls, and de Jager. 2001. Uniformly Most Powerful Cyclic Permutation Invariant Detection for Discrete-Time Signals.” In.
Nordmark, and Fahlen. 1988. Beat Theories of Musical Consonance.” Speech Transmission Laboratory, Quarterly Progress and Status Report.
Park. 2004. Construction of Random Vectors of Heterogeneous Component Variables Under Specified Correlation Structures.” Computational Statistics & Data Analysis.
Parncutt. 1989. Harmony: a psychoacoustical approach. Springer series in information sciences 19.
———. 1994. A Perceptual Model of Pulse Salience and Metrical Accent in Musical Rhythms.” Music Perception: An Interdisciplinary Journal.
———. 1997. A Model of the Perceptual Root(s) of a Chord Accounting for Voicing and Prevailing Tonality.” In Music, Gestalt, and Computing. Lecture Notes in Computer Science 1317.
———. 2005. Psychoacoustics and Music Perception.” Musikpsychologie–Das Neue Handbuch.
———. 2006. Commentary on Keith Mashinter’s ‘Calculating Sensory Dissonance: Some Discrepancies Arising from the Models of Kameoka & Kuriyagawa, and Hutchinson & Knopoff’.”
———. 2013. Psychoacoustics and Cognition for Musicians.” Sound Musicianship: Understanding the Crafts of Music.
Parncutt, and Hair. 2011. Consonance and Dissonance in Music Theory and Psychology: Disentangling Dissonant Dichotomies.” J. Interdiscipl. Music Stud.
Parncutt, and Strasburger. 1994. Applying Psychoacoustics in Composition: ‘Harmonic’ Progressions of ‘Nonharmonic’ Sonorities.” Perspectives of New Music.
Perchy, and Sarria. 2009. Dissonances: Brief Description and Its Computational Representation in the RTCC Calculus.” In Proc. Of SMC2009, Porto, Portugal.
Plomp, and Levelt. 1965. Tonal Consonance and Critical Bandwidth.” The Journal of the Acoustical Society of America.
Rasch, and Plomp. 1999. The Perception of Musical Tones.” The Psychology of Music.
Reese, Yampolskiy, and Elmaghraby. 2012. A Framework for Interactive Generation of Music for Games.” In 2012 17th International Conference on Computer Games (CGAMES). CGAMES ’12.
Reitboeck, and Brody. 1969. A Transformation with Invariance Under Cyclic Permutation for Applications in Pattern Recognition.” Information and Control.
Reuter. 1997. Karl Erich Schumann’s Principles of Timbre as a Helpful Tool in Stream Segregation Research.” In Music, Gestalt, and Computing. Lecture Notes in Computer Science 1317.
Rohrmeier, Zuidema, Wiggins, et al. 2015. Principles of structure building in music, language and animal song.” Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences.
Serrà, Corral, Boguñá, et al. 2012. Measuring the Evolution of Contemporary Western Popular Music.” Scientific Reports.
Sethares. 1997. Specifying Spectra for Musical Scales.” The Journal of the Acoustical Society of America.
———. 1998a. Tuning, Timbre, Spectrum, Scale.
———. 1998b. Consonance-Based Spectral Mappings.” Computer Music Journal.
Sethares, Milne, Tiedje, et al. 2009. Spectral Tools for Dynamic Tonality and Audio Morphing.” Computer Music Journal.
Shiv. 2011. Improved Frequency Estimation in Sinusoidal Models Through Iterative Linear Programming Schemes.” In Sound and Music Computing Conference, Proceedings.
Stevens, and Volkmann. 1940. The Relation of Pitch to Frequency: A Revised Scale.” The American Journal of Psychology.
Stolzenburg. 2013. Harmony Perception by Periodicity Detection.” arXiv:1306.6458 [Cs].
Terhardt. 1974. Pitch, Consonance, and Harmony.” The Journal of the Acoustical Society of America.
Terhardt, Stoll, and Seewann. 1982. Algorithm for Extraction of Pitch and Pitch Salience from Complex Tonal Signals.” The Journal of the Acoustical Society of America.
Thompson, and Parncutt. 1997. Perceptual Judgments of Triads and Dyads: Assessment of a Psychoacoustic Model.” Music Perception.
Tillmann, Bharucha, and Bigand. 2000. Implicit Learning of Tonality: A Self-Organizing Approach. Psychological Review.
Toiviainen. 1997. Optimizing Self-Organizing Timbre Maps: Two Approaches.” In Music, Gestalt, and Computing. Lecture Notes in Computer Science 1317.
Toussaint, Godfried T. 2004. A Comparison of Rhythmic Similarity Measures. In ISMIR.
Toussaint, Godfried. 2005. Mathematical Features for Recognizing Preference in Sub-Saharan African Traditional Rhythm Timelines.” In Pattern Recognition and Data Mining.
Tymoczko. 2006. The Geometry of Musical Chords.” Science.
———. 2009a. Three Conceptions of Musical Distance.” In Mathematics and Computation in Music. Communications in Computer and Information Science 38.
———. 2009b. Generalizing Musical Intervals.” Journal of Music Theory.
———. 2011a. Mazzola’s Counterpoint Theory.” Mathematics and Computation in Music, Springer.
———. 2011b. A Geometry of Music: Harmony and Counterpoint in the Extended Common Practice.
———. 2012. The Generalized Tonnetz.” Journal of Music Theory.
———. 2013. Geometry and the Quest for Theoretical Generality.” Journal of Mathematics and Music.
Vassilakis, and Kendall. 2010. Psychoacoustic and Cognitive Aspects of Auditory Roughness: Definitions, Models, and Applications.” In IS&T/SPIE Electronic Imaging.
Wagh. 1976. Cyclic Autocorrelation as a Translation Invariant Transform.” India, IEE-IERE Proceedings.
Xin, and Qi. 2006. A Many to One Discrete Auditory Transform.” arXiv:math/0603174.
Zanette. 2006. Zipf’s Law and the Creation of Musical Context.” Musicae Scientiae.
Zhao, and Singer. 2013. Fourier–Bessel Rotational Invariant Eigenimages.” Journal of the Optical Society of America A.