GP inducing variables

A classic trick in factoring GP likelihoods. Sparse Gaussian processes approximate the target posterior by summarising the data with a short list of inducing points that have nearly the same posterior density, by some metric; i.e. this is how data summarization works in the context of Gaussian processes. As with many general concepts specialised upon Gaussian processes, data summarisation ends up being especially feasible in this context and pairs nicely with variational inference.

These have been invented in various forms by various people, but most of the differences we can ignore. Based on citations, the Ground Zero for the Right Way is the version of Titsias (2009), which is the foundation stone of all subsequent methods. A summary of Titsias (2009) from Hensman, Fusi, and Lawrence (2013):

We consider an output vector $\mathbf{y},$ where each entry $y_i$ is a noisy observation of the function $f,$ i.e. $y_i=f({\mathbf{x}}_i)+{\epsilon }_i$ for all the points $\mathbf{X}={\left\{{\mathbf{x}}_i\right\}}_{i=1}^n.$ We assume the noises ${\epsilon }_i\sim \mathcal{N}(0,1/\beta )$ to be independent. We introduce a Gaussian process prior over $f(\cdot ).$ Now let the vector $\mathbf{f}=\{f(x_i){\}}_{i=1}^n$ contain values of the function evaluated at the inputs. We introduce a set of inducing variables, defined similarly: let the vector $\mathbf{u}=\{f({\mathbf{z}}_i){\}}_{i=1}^m$. Standard properties of Gaussian processes allow us to write $$\begin{array}{rl}p(\mathbf{y}\mid \mathbf{f})& =\mathcal{N}(\mathbf{y}\mid \mathbf{f},{\beta }^{-1}\mathbf{I})\\ p(\mathbf{f}\mid \mathbf{u})& =\mathcal{N}(\mathbf{f}\mid {\mathbf{K}}_{nm}{\mathbf{K}}_{mm}^{-1}\mathbf{u},\tilde{\mathbf{K}})\\ p(\mathbf{u})& =\mathcal{N}(\mathbf{u}\mid \mathbf{0},{\mathbf{K}}_{mm})\end{array}$$ where ${\mathbf{K}}_{mm}$ is the covariance function evaluated between all the inducing points and ${\mathbf{K}}_{nm}$ is the covariance function between all inducing points and training points and we have defined $\tilde{\mathbf{K}}={\mathbf{K}}_{nn}-{\mathbf{K}}_{nm}{\mathbf{K}}_{mm}^{-1}{\mathbf{K}}_{mn}$

Quoting them:

We first apply Jensen’s inequality on the conditional probability $p(\mathbf{y}\mid \mathbf{u})$ $$\begin{array}{rl}\log p(\mathbf{y}\mid \mathbf{u})& =\log ⟨p(\mathbf{y}\mid \mathbf{f}){⟩}_{p(\mathbf{f}\mid \mathbf{u})}\\ & \ge ⟨\log p(\mathbf{y}\mid \mathbf{f}){⟩}_{p(\mathbf{f}\mid \mathbf{u})}\triangleq {\mathcal{L}}_1\end{array}$$ where $⟨\cdot {⟩}_{p(x)}$ denotes an expectation under $p(x).$ For Gaussian noise, taking the expectation inside the log is tractable, but it results in an expression containing ${\mathbf{K}}_{nn}^{-1},$ which has a computational complexity of $\mathcal{O}\left(n^3\right).$ Bringing the expectation outside the log gives a lower bound, ${\mathcal{L}}_1,$ which can be computed with complexity $\mathcal{O}\left(m^3\right).$ Further, when $p(\mathbf{y}\mid \mathbf{f})$ factorises across the data, $$p(\mathbf{y}\mid \mathbf{f})=\prod _{i=1}^{n}p(y_i\mid f_i)$$ then this lower bound can be shown to be separable across y giving $$\exp \left({\mathcal{L}}_1\right)=\prod _{i=1}^n\mathcal{N}(y_i\mid {\mu }_i,{\beta }^{-1})\exp (-\frac{1}{2}\beta {\tilde{k}}_{i,i})$$ where $\mathit{\mu }={\mathbf{K}}_{nm}{\mathbf{K}}_{mm}^{-1}\mathbf{u}$ and ${\tilde{k}}_{i,i}$ is the $i$th diagonal element of $\tilde{\mathbf{K}}$. Note that the difference between our bound and the original log likelihood is given by the Kullback-Leibler (KL) divergence between the posterior over the mapping function given the data and the inducing variables and the posterior of the mapping function given the inducing variables only, $$\mathrm{KL}(p(\mathbf{f}\mid \mathbf{u})\parallel p(\mathbf{f}\mid \mathbf{u},\mathbf{y}))$$ This KL divergence is minimized when there are $m=n$ inducing variables and they are placed at the training data locations. This means that $\mathbf{u}=\mathbf{f}$, ${\mathbf{K}}_{mm}={\mathbf{K}}_{nm}={\mathbf{K}}_{nn}$ meaning that $\tilde{\mathbf{K}}=\mathbf{0}.$ In this case we recover $\exp \left({\mathcal{L}}_1\right)=p(\mathbf{y}\mid \mathbf{f})$ and the bound becomes equality because $p(\mathbf{f}\mid \mathbf{u})$ is degenerate. However, since $m=n$ there would be no computational or storage advantage from the representation. When $m<n$ the bound can be maximised with respect to $\mathbf{Z}$ (which are variational parameters). This minimises the KL divergence and ensures that $\mathbf{Z}$ are distributed amongst the training data $\mathbf{X}$ such that all ${\tilde{k}}_{i,i}$ are small. In practice this means that the expectations in (1) are only taken across a narrow domain (${\tilde{k}}_{i,i}$ is the marginal variance of $p(f_i\mid \mathbf{u})$ ), keeping Jensen’s bound tight.

…we recover the bound of Titsias (2009) by marginalising the inducing variables, $$\begin{array}{rl}\log p(\mathbf{y}\mid \mathbf{X})& =\log \int p(\mathbf{y}\mid \mathbf{u})p(\mathbf{u})\mathrm{d}\mathbf{u}\\ & \ge \log \int \exp \left\{{\mathcal{L}}_1\right\}p(\mathbf{u})\mathrm{d}\mathbf{u}\triangleq {\mathcal{L}}_2\end{array}$$ which with some linear algebraic manipulation leads to $${\mathcal{L}}_2=\log \mathcal{N}(\mathbf{y}\mid \mathbf{0},{\mathbf{K}}_{nm}{\mathbf{K}}_{mm}^{-1}{\mathbf{K}}_{mn}+{\beta }^{-1}\mathbf{I})-\frac{1}{2}\beta \mathrm{tr}(\tilde{\mathbf{K}})$$ matching the result of Titsias, with the implicit approximating distribution $q(\mathbf{u})$ having precision $$\mathbf{\Lambda }=\beta {\mathbf{K}}_{mm}^{-1}{\mathbf{K}}_{mn}{\mathbf{K}}_{nm}{\mathbf{K}}_{mm}^{-1}+{\mathbf{K}}_{mm}^{-1}$$ and mean $$\hat{\mathbf{u}}=\beta {\mathbf{\Lambda }}^{-1}{\mathbf{K}}_{mm}^{-1}{\mathbf{K}}_{mn}\mathbf{y}$$