Bayes linear regression and basis-functions in Gaussian process regression

Another way of cunningly chopping up the work of fitting a Gaussian process is to represent the process as a random function comprising basis functions $\varphi =({\varphi }_1,\dots ,{\varphi }_{\ell })$ with the Gaussian random weight vector $w$ so that $$f^{(w)}(\cdot )=\sum _{i=1}^{\ell }{w}_i{\varphi }_i(\cdot )\mathit{w}\sim \mathcal{N}(\mathbf{0},{\mathbf{\Sigma }}_{\mathit{w}}).$$ $f^{(w)}$ is a random function satisfying ${\mathit{f}}^{(\mathit{w})}\sim \mathcal{N}(\mathbf{0},{\mathbf{\Phi }}_n{\mathbf{\Sigma }}_{\mathit{w}}{\mathbf{\Phi }}^{\mathrm{\top }})$, where ${\mathbf{\Phi }}_n=\mathit{\varphi }(\mathbf{X})$ is a $|\mathbf{X}|\times \ell$ matrix of features. This is referred to as a weight space approach in ML.

TODO: I just assumed centred weights here, but that is crazy. Update to relax that assumption.

We might imagine this representation would be exact if we had countably many basis functions, and under sane conditions it is. We would like to know, further, that we can find a basis such that we need not too many basis functions to represent the process. Looking at the Karhunen-Loève theorem we might imagine that this can sometimes work out fine, and indeed it does, sometimes.

This is a classic; see Chapter 3 of Bishop (2006) is classic and nicely clear. Cressie and Wikle (2011) targets for the spatiotemporal context.

Hijinks ensue when selecting the basis functions. If we were to treat the natural Hilbert space here seriously we could consider identifying the bases as eigenfunctions of the kernel. This is not generally easy. We tend to use either global bases such as Fourier bases or more generally Karhunen-Loéve bases, or construct local bases of limited overlap (usually piecewise polynomials AFAICT).

The kernel trick writes a kernel $k$ as an inner product in a corresponding reproducing kernel Hilbert space (RKHS) ${\mathcal{H}}_k$ with a feature map $\phi :\mathcal{X}\to {\mathcal{H}}_k.$ In sufficiently nice cases the kernel is well approximated $$k(\mathit{x},{\mathit{x}}^{\prime })={⟨\phi (\mathit{x}),\phi \left({\mathit{x}}^{\prime }\right)⟩}_{{\mathcal{H}}_k}\approx \mathit{\varphi }(\mathit{x}{)}^{\mathrm{\top }}\overline{\mathit{\varphi }\left({\mathit{x}}^{\prime }\right)}$$ where $\mathit{\varphi }:\mathcal{X}\to {\mathbb{C}}^{\ell }$ is a finite-dimensional feature map. TODO: What is the actual guarantee here?

3 K-L basis

We recall from the Karhunen-Loéve notebook that the mean-square-optimal $f^{(w)}$ for approximating a Gaussian process $f$ is found by truncating the Karhunen-Loéve expansion $$f(\cdot )=\sum _{i=1}^{\mathrm{\infty }}{w}_i{\varphi }_i(\cdot )w_i\sim \mathcal{N}(0,{\lambda }_i)$$ where ${\varphi }_i$ and ${\lambda }_i$ are, respectively, the $i$-th (orthogonal) eigenfunction and eigenvalue of the covariance operator $\psi \mapsto {\int }_{\mathcal{X}}\psi (\mathit{x})k(\mathit{x},\cdot )\mathrm{d}\mathit{x}$, written in decreasing order of ${\lambda }_i$. What is the orthogonal basis $\{{\varphi }_i{\}}_i$ though? That depends on the problem and can be a lot of work to calculate.

In the case that our field is stationary on a “nice” domain, though, this can easy — we simply have the Fourier features as the natural basis.