GP inducing features

TBD

I’m short of time, so I’ll quote a summary from Tiao, Dutordoir, and Picheny (2023), which is not pithy but has the information.

the joint distribution of the model augmented by inducing variables $\mathbf{u}$ is $p(\mathbf{y},\mathbf{f},\mathbf{u})=p(\mathbf{y}\mid \mathbf{f})p(\mathbf{f},\mathbf{u})$ where $p(\mathbf{f},\mathbf{u})=p(\mathbf{f}\mid \mathbf{u})p(\mathbf{u})$ for prior $p(\mathbf{u})=\mathcal{N}(\mathbf{0},{\mathbf{K}}_{\mathbf{u}\mathbf{u}})$ and conditional $$p(\mathbf{f}\mid \mathbf{u})=\mathcal{N}(\mathbf{f}\mid {\mathbf{Q}}_{\mathrm{fu}}\mathbf{u},{\mathbf{K}}_{\mathrm{ff}}-{\mathbf{Q}}_{\mathrm{ff}})$$

where ${\mathbf{Q}}_{\mathrm{ff}}\triangleq {\mathbf{Q}}_{\mathrm{fu}}{\mathbf{K}}_{\mathrm{uu}}{\mathbf{Q}}_{\mathrm{uf}}$ and ${\mathbf{Q}}_{\mathrm{fu}}\triangleq {\mathbf{K}}_{\mathrm{fu}}{\mathbf{K}}_{\mathrm{uu}}^{-1}$. The joint variational distribution is defined as $q(\mathbf{f},\mathbf{u})\triangleq p(\mathbf{f}\mid \mathbf{u})q(\mathbf{u})$, where $q(\mathbf{u})\triangleq \mathcal{N}({\mathbf{m}}_{\mathbf{u}},{\mathbf{C}}_{\mathbf{u}})$ for variational parameters ${\mathbf{m}}_{\mathbf{u}}\in {\mathbb{R}}^M$ and ${\mathbf{C}}_{\mathbf{u}}\in {\mathbb{R}}^{M\times M}$ s.t. ${\mathbf{C}}_{\mathbf{u}}⪰0$. Integrating out $\mathbf{u}$ yields the posterior predictive

$$q\left({\mathbf{f}}_{\ast }\right)=\mathcal{N}({\mathbf{Q}}_{\ast \mathbf{u}}{\mathbf{m}}_{\mathbf{u}},{\mathbf{K}}_{\ast \ast }-{\mathbf{Q}}_{\ast \mathbf{u}}({\mathbf{K}}_{\mathbf{u}\mathbf{u}}-{\mathbf{C}}_{\mathbf{u}}){\mathbf{Q}}_{\mathbf{u}\ast })$$

where parameters ${\mathrm{m}}_{\mathrm{u}}$ and ${\mathrm{C}}_{\mathrm{u}}$ are learned by minimizing the Kullback-Leibler (KL) divergence between the approximate and exact posterior, KL $[q(\mathbf{f})\parallel p(\mathbf{f}\mid \mathbf{y})]$. Thus seen, sVGP has time complexity $\mathcal{O}\left(M^3\right)$ at prediction time and $\mathcal{O}(M^3+$ $M^{2}N)$ during training. In the reproducing kernel Hilbert space (RKHS) associated with $k$, the predictive has a dual representation in which the mean and covariance share the same basis determined by $\mathbf{u}$ (Cheng & Boots, 2017; Salimbeni et al., 2018). More specifically, the basis function is effectively the vector-valued function ${\mathbf{k}}_{\mathrm{u}}:\mathcal{X}\to {\mathbb{R}}^M$ whose $m$-th component is defined as

$${[{\mathbf{k}}_{\mathbf{u}}(\mathbf{x})]}_m\triangleq \mathrm{Cov}(f(\mathbf{x}),u_m)$$

In the standard definition of inducing points, ${[{\mathbf{k}}_{\mathbf{u}}(\mathbf{x})]}_m=$ $k({\mathbf{z}}_m,\mathbf{x})$, so the basis function is solely determined by $k$ and the local influence of pseudo-input ${\mathbf{z}}_m$. Inter-domain inducing features are a generalisation of standard inducing variables in which each variable $u_m\triangleq L_m[f]$ for some linear operator $L_m:{\mathbb{R}}^{\mathcal{X}}\to \mathbb{R}$. A particularly useful operator is the integral transform, $L_m[f]\triangleq$ ${\int }_{\mathcal{X}}f(\mathbf{x}){\varphi }_m(\mathbf{x})\mathrm{d}\mathbf{x}$, which was originally employed by Lázaro-Gredilla & Figueiras-Vidal (2009). Refer to the manuscript of van der Wilk et al. (2020) for a more thorough and contemporary treatment. A closely related form is the scalar projection of $f$ onto some ${\varphi }_m$ in the RKHS $\mathcal{H}$,

$$L_m[f]\triangleq {⟨f,{\varphi }_m⟩}_{\mathcal{H}}$$ and conditional

$$p(\mathbf{f}\mid \mathbf{u})=\mathcal{N}(\mathbf{f}\mid {\mathbf{Q}}_{\mathrm{fu}}\mathbf{u},{\mathbf{K}}_{\mathrm{ff}}-{\mathbf{Q}}_{\mathrm{ff}})$$

where ${\mathbf{Q}}_{\mathrm{ff}}\triangleq {\mathbf{Q}}_{\mathrm{fu}}{\mathbf{K}}_{\mathrm{uu}}{\mathbf{Q}}_{\mathrm{uf}}$ and ${\mathbf{Q}}_{\mathrm{fu}}\triangleq {\mathbf{K}}_{\mathrm{fu}}{\mathbf{K}}_{\mathrm{uu}}^{-1}$. The joint variational distribution is defined as $q(\mathbf{f},\mathbf{u})\triangleq p(\mathbf{f}\mid \mathbf{u})q(\mathbf{u})$, where $q(\mathbf{u})\triangleq \mathcal{N}({\mathbf{m}}_{\mathbf{u}},{\mathbf{C}}_{\mathbf{u}})$ for variational parameters ${\mathbf{m}}_{\mathbf{u}}\in {\mathbb{R}}^M$ and ${\mathbf{C}}_{\mathbf{u}}\in {\mathbb{R}}^{M\times M}$ s.t. ${\mathbf{C}}_{\mathbf{u}}⪰0$. Integrating out $\mathbf{u}$ yields the posterior predictive

$$q\left({\mathbf{f}}_{\ast }\right)=\mathcal{N}({\mathbf{Q}}_{\ast \mathbf{u}}{\mathbf{m}}_{\mathbf{u}},{\mathbf{K}}_{\ast \ast }-{\mathbf{Q}}_{\ast \mathbf{u}}({\mathbf{K}}_{\mathbf{u}\mathbf{u}}-{\mathbf{C}}_{\mathbf{u}}){\mathbf{Q}}_{\mathbf{u}\ast })$$

where parameters ${\mathrm{m}}_{\mathrm{u}}$ and ${\mathrm{C}}_{\mathrm{u}}$ are learned by minimising the Kullback-Leibler (KL) divergence between the approximate and exact posterior, KL $[q(\mathbf{f})\parallel p(\mathbf{f}\mid \mathbf{y})]$. Thus seen, sVGP has time complexity $\mathcal{O}\left(M^3\right)$ at prediction time and $\mathcal{O}(M^3+$ $M^{2}N)$ during training. In the reproducing kernel Hilbert space (RKHS) associated with $k$, the predictive has a dual representation in which the mean and covariance share the same basis determined by $\mathbf{u}$ (Cheng & Boots, 2017; Salimbeni et al., 2018). More specifically, the basis function is effectively the vector-valued function ${\mathbf{k}}_{\mathrm{u}}:\mathcal{X}\to {\mathbb{R}}^M$ whose $m$-th component is defined as

$${[{\mathbf{k}}_{\mathbf{u}}(\mathbf{x})]}_m\triangleq \mathrm{Cov}(f(\mathbf{x}),u_m)$$

In the standard definition of inducing points, ${[{\mathbf{k}}_{\mathbf{u}}(\mathbf{x})]}_m=$ $k({\mathbf{z}}_m,\mathbf{x})$, so the basis function is solely determined by $k$ and the local influence of pseudo-input ${\mathbf{z}}_m$. Inter-domain inducing features are a generalisation of standard inducing variables in which each variable $u_m\triangleq L_m[f]$ for some linear operator $L_m:{\mathbb{R}}^{\mathcal{X}}\to \mathbb{R}$. A particularly useful operator is the integral transform, $L_m[f]\triangleq$ ${\int }_{\mathcal{X}}f(\mathbf{x}){\varphi }_m(\mathbf{x})\mathrm{d}\mathbf{x}$, which was originally employed by Lázaro-Gredilla & Figueiras-Vidal (2009). Refer to the manuscript of van der Wilk et al. (2020) for a more thorough and contemporary treatment. A closely related form is the scalar projection of $f$ onto some ${\varphi }_m$ in the RKHS $\mathcal{H}$,

$$L_m[f]\triangleq {⟨f,{\varphi }_m⟩}_{\mathcal{H}}$$ which leads to ${[{\mathbf{k}}_{\mathbf{u}}(\mathbf{x})]}_m={\varphi }_m(\mathbf{x})$ by the reproducing property of the RKHS. This, in effect, equips the GP approximation with basis functions ${\varphi }_m$ that are not solely determined by the kernel, and suitable choices can lead to sparser representations and considerable computational benefits (Hensman et al., 2018; Burt et al., 2020; Dutordoir et al., 2020; Sun et al., 2021).