Gaussian process inference by partial updates

1 Subsampling sites

Chen et al. (2020) shows that we can optimise hyperparameters by subsampling sites and performing SGD if the kernels are smooth enough and the batch sizes large enough. At inference time we are still in trouble though.

2 Subsampling observations and sites

Minh (2022) bounds Wasserstein when we subsample observations and sites. e.g. their algorithm 5.1 goes:

• Input: Finite samples $\left\{{\xi }_k^i\left(x_j\right)\right\}$, from $N_i$ realizations ${\xi }_k^i,1\le k\le N_i$, of processes ${\xi }^i,i=1,2$, sampled at $m$ points $x_j,1\le j\le m$

• Procedure:

  1. Form $m\times N_i$ data matrices $Z_i$, with ${\left(Z_i\right)}_{jk}={\xi }_k^i\left(x_j\right),i=1,2,1\le j\le m,1\le k\le N_i$
  2. Compute $m\times m$ empirical covariance matrices ${\hat{K}}^i=\frac{1}{N}{Z}_i{Z}_i^T,i=1,2$
  3. Compute $W=W_2[\mathcal{N}(0,\frac{1}{m}{\hat{K}}^1),\mathcal{N}(0,\frac{1}{m}{\hat{K}}^2)]$ according to $$W_2^2({\nu }_0,{\nu }_1)={\Vert m_0-m_1\Vert }^2+\mathrm{tr}\left(C_0\right)+\mathrm{tr}\left(C_1\right)-2\mathrm{tr}{\left(C_0^{1/2}{C}_1{C}_0^{1/2}\right)}^{1/2}$$

This is pretty much as we would expect to do it naively by plugging in the sample estimates of the target quantity, except they provide bounds for the quality of the estimate we get. AFAICS the bounds are trivial for Wasserstein in infinite-dimensional Hilbert spaces. But if we care about Sinkhorn divergences they seem to have useful bounds?

Anyway, sometimes subsampling is OK, it seems, if we want to approximate some GP in Sinkhorn divergence. Does this tell us anything about the optimisation problem?

3 Random projections

Song et al. (2019) maybe? I wonder if the Slice Score Matching approach is feasible here? Works great in diffusion.

4 References