Distances between Gaussian distributions

1 Wasserstein

A useful analytic result about Wasserstein-2 distance, i.e. $W_2(\mu ;\nu ):=inf\mathbb{E}(\Vert X-Y{\Vert }_2^2{)}^{1/2}$ for $X\sim \nu$, $Y\sim \mu$. Two Gaussians may be related thusly (Givens and Shortt 1984):

$$\begin{array}{rl}d& :=W_2(\mathcal{N}({\mu }_1,{\mathrm{\Sigma }}_1);\mathcal{N}({\mu }_2,{\mathrm{\Sigma }}_2))\\ \Rightarrow d^2& =\Vert {\mu }_1-{\mu }_2{\Vert }_2^2+\mathrm{tr}({\mathrm{\Sigma }}_1+{\mathrm{\Sigma }}_2-2({\mathrm{\Sigma }}_1^{1/2}{\mathrm{\Sigma }}_2{\mathrm{\Sigma }}_1^{1/2}{)}^{1/2}).\end{array}$$

In the centred case this is even simpler:

$$\begin{array}{rl}d& :=W_2(\mathcal{N}(0,{\mathrm{\Sigma }}_1);\mathcal{N}(0,{\mathrm{\Sigma }}_2))\\ \Rightarrow d^2& =\mathrm{tr}({\mathrm{\Sigma }}_1+{\mathrm{\Sigma }}_2-2({\mathrm{\Sigma }}_1^{1/2}{\mathrm{\Sigma }}_2{\mathrm{\Sigma }}_1^{1/2}{)}^{1/2}).\end{array}$$

2 Kullback-Leibler

Pulled from wikipedia:

$$D_{\text{KL}}(\mathcal{N}({\mu }_1,{\mathrm{\Sigma }}_1)\parallel \mathcal{N}({\mu }_2,{\mathrm{\Sigma }}_2))=\frac{1}{2}(\mathrm{tr}\left({\mathrm{\Sigma }}_2^{-1}{\mathrm{\Sigma }}_1\right)+({\mu }_2-{\mu }_1{)}^{\mathsf{T}}{\mathrm{\Sigma }}_2^{-1}({\mu }_2-{\mu }_1)-k+\ln \left(\frac{det{\mathrm{\Sigma }}_2}{det{\mathrm{\Sigma }}_1}\right)).$$

In the centred case this reduces to

$$D_{\text{KL}}(\mathcal{N}(0,{\mathrm{\Sigma }}_1)\parallel \mathcal{N}(0,{\mathrm{\Sigma }}_2))=\frac{1}{2}(\mathrm{tr}\left({\mathrm{\Sigma }}_2^{-1}{\mathrm{\Sigma }}_1\right)-k+\ln \left(\frac{det{\mathrm{\Sigma }}_2}{det{\mathrm{\Sigma }}_1}\right)).$$

3 Hellinger

Djalil defines Hellinger distance

$$\mathrm{H}(\mu ,\nu )={\Vert \sqrt{f}-\sqrt{g}\Vert }_{{\mathrm{L}}^2(\lambda )}=(\int (\sqrt{f}-\sqrt{g}{)}^2\mathrm{d}\lambda {)}^{1/2}.$$

via Hellinger affinity

$$\mathrm{A}(\mu ,\nu )=\int \sqrt{fg}\mathrm{d}\lambda ,\mathrm{H}(\mu ,\nu {)}^2=2-2A(\mu ,\nu ).$$

For Gaussians, it apparently turns out that

$$\mathrm{A}(\mathcal{N}(m_1,{\sigma }_1^2),\mathcal{N}(m_2,{\sigma }_2^2))=\sqrt{2\frac{{\sigma }_1{\sigma }_2}{{\sigma }_1^2+{\sigma }_2^2}}\exp (-\frac{(m_1-m_2{)}^2}{4({\sigma }_1^2+{\sigma }_2^2)}),$$

In multiple dimensions:

$$\mathrm{A}(\mathcal{N}(m_1,{\mathrm{\Sigma }}_1),\mathcal{N}(m_2,{\mathrm{\Sigma }}_2))=\frac{det({\mathrm{\Sigma }}_1{\mathrm{\Sigma }}_2{)}^{1/4}}{det(\frac{{\mathrm{\Sigma }}_1+{\mathrm{\Sigma }}_2}{2}{)}^{1/2}}\exp (-\frac{⟨\mathrm{\Delta }m,({\mathrm{\Sigma }}_1+{\mathrm{\Sigma }}_2{)}^{-1}\mathrm{\Delta }m⟩}{4}).$$

4 “Natural” / geodesic

High-speed introduction to geometry of distances: Moakher and Batchelor (2006), which discusses the problems of comparing covariance matrices on a natural manifold.

The geodesic distance between $\mathbf{P}$ and $\mathbf{Q}$ in $\mathcal{P}(n)$ is given by Lang (1999) $${\mathrm{d}}_R(\mathbf{P},\mathbf{Q})={\Vert \log \left({\mathbf{P}}^{-1}\mathbf{Q}\right)\Vert }_F={[\sum _{i=1}^n{\log }^2{\lambda }_i\left({\mathbf{P}}^{-1}\mathbf{Q}\right)]}^{1/2},$$ where ${\lambda }_i\left({\mathbf{P}}^{-1}\mathbf{Q}\right),1\le i\le n$ are the eigenvalues of the matrix ${\mathbf{P}}^{-1}\mathbf{Q}$. Because ${\mathbf{P}}^{-1}\mathbf{Q}$ is similar to ${\mathbf{P}}^{-1/2}\mathbf{Q}{\mathbf{P}}^{-1/2}$, the eigenvalues ${\lambda }_i\left({\mathbf{P}}^{-1}\mathbf{Q}\right)$ are all positive and hence [this] is well defined for all $\mathbf{P}$ and $\mathbf{Q}$ of $\mathcal{P}(n)$.

5 Maximum mean discrepancy

With the Gaussian kernel, there is a closed-form expression for the distance of an RV from the standard Gaussian (Rustamov 2021).

The computation of the MMD requires specifying a positive-definite kernel; in this paper we always assume it to be the Gaussian RBF kernel of width $\gamma$, namely, $k(x,y)=e^{-\parallel x-y{\parallel }^2/\left(2{\gamma }^2\right)}$. Here, $x,y\in {\mathbb{R}}^d$, where $d$ is the dimension of the code/latent space, and we use $\parallel \cdot \parallel$ to denote the ${\ell }_2$ norm. The population MMD can be most straightforwardly computed via the formula (Gretton et al. 2012): $${\mathrm{MMD}}^2(P,Q)={\mathbb{E}}_{x,x^{\prime }\sim P}\left[k(x,x^{\prime })\right]-2{\mathbb{E}}_{x\sim P,y\sim Q}[k(x,y)]+{\mathbb{E}}_{y,y^{\prime }\sim Q}\left[k(y,y^{\prime })\right]$$

[U]sing the sample $Q_n={\left\{z_i\right\}}_{i=1}^n$ of size $n$, we replace the last two terms by the sample average and the U-statistic respectively to obtain the unbiased estimator: $${\mathrm{MMD}}_u^2({\mathcal{N}}_d,Q_n)={\mathbb{E}}_{x,x^{\prime }\sim {\mathcal{N}}_d}\left[k(x,x^{\prime })\right]-\frac{2}{n}\sum _{i=1}^n{\mathbb{E}}_{x\sim {\mathcal{N}}_d}\left[k(x,z_i)\right]+\frac{1}{n(n-1)}\sum _{i=1}^n\sum _{j\ne i}^{n}k(z_i,z_j)$$ Our main result is the following proposition whose proof can be found in Appendix C: Proposition. The expectations in the expression above can be computed analytically to yield the formula: $${\mathrm{MMD}}_u^2({\mathcal{N}}_d,Q_n)={\left(\frac{{\gamma }^2}{2+{\gamma }^2}\right)}^{d/2}-\frac{2}{n}{\left(\frac{{\gamma }^2}{1+{\gamma }^2}\right)}^{d/2}\sum _{i=1}^n{e}^{-\frac{{\Vert z_i\Vert }^2}{2(1+{\gamma }^2)}}+\frac{1}{n(n-1)}\sum _{i=1}^n\sum _{j\ne i}^n{e}^{-\frac{{\Vert z_i-z_j\Vert }^2}{2{\gamma }^2}}$$

Extending this to arbitrary Gaussians, or analytic Gaussians, is left as an exercise.

Intuitively, we might prefer other kernels than the RBF if we are comparing Gaussians specifically; in particular, the RBF is “wasteful” in that it controls moments of all orders, whereas we might only care about the first two moments (mean and covariance), for Gaussians.

6 Infinite-dimensional Gaussian measures

Gets much weirder, but we care about this in Function-valued GPs. See (Quang 2021a, 2021b, 2022, 2023).

7 References