Kernel distribution embedding

2.1 Kernel sum rule

In probability theory, the marginal distribution of $X$ can be computed by integrating out $Y$ from the joint density (including the prior distribution on $Y$) $$Q(X)={\int }_{\mathrm{\Omega }}P(X\mid Y)\mathrm{d}\pi (Y)$$

The analogue of this rule in the kernel embedding framework states that ${\mu }_X^{\pi }$, the RKHS embedding of $Q(X)$, can be computed via $${\mu }_X^{\pi }=\mathbb{E}[{\mathcal{C}}_{X\mid Y}\phi (Y)]={\mathcal{C}}_{X\mid Y}\mathbb{E}[\phi (Y)]={\mathcal{C}}_{X\mid Y}{\mu }_Y^{\pi }$$ where ${\mu }_Y^{\pi }$ is the kernel embedding of $\pi (Y)$. In practical implementations, the kernel sum rule takes the following form $${\hat{\mu }}_X^{\pi }={\hat{\mathcal{C}}}_{X\mid Y}{\hat{\mu }}_Y^{\pi }=\mathbf{{\rm Y}}(\mathbf{G}+\lambda \mathbf{I}{)}^{-1}\tilde{\mathbf{G}}\mathit{\alpha }$$ where $${\mu }_Y^{\pi }=\sum _{i=1}^{\tilde{n}}{\alpha }_i\phi \left({\tilde{y}}_i\right)$$ is the empirical kernel embedding of the prior distribution, $\mathit{\alpha }={({\alpha }_1,\dots ,{\alpha }_{\tilde{n}})}^T,\mathbf{{\rm Y}}=(\phi \left(x_1\right),\dots ,\phi \left(x_n\right))$, and $\mathbf{G},\tilde{\mathbf{G}}$ are Gram matrices with entries ${\mathbf{G}}_{ij}=k(y_i,y_j),{\tilde{\mathbf{G}}}_{ij}=k(y_i,{\tilde{y}}_j)$ respectively.

2.2 Kernel chain rule

In probability theory, a joint distribution can be factorised into a product between conditional and marginal distributions $$Q(X,Y)=P(X\mid Y)\pi (Y)$$

The analogue of this rule in the kernel embedding framework states that ${\mathcal{C}}_{XY}^{\pi }$, the joint embedding of $Q(X,Y)$, can be factorised as a composition conditional embedding operator with the auto-covariance operator associated with $\pi (Y)$ $${\mathcal{C}}_{XY}^{\pi }={\mathcal{C}}_{X\mid Y}{\mathcal{C}}_{YY}^{\pi }$$ where $$\begin{array}{rl}{\mathcal{C}}_{XY}^{\pi }& =\mathbb{E}[\phi (X)\otimes \phi (Y)]\\ {\mathcal{C}}_{YY}^{\pi }& =\mathbb{E}[\phi (Y)\otimes \phi (Y)]\end{array}$$

In practical implementations, the kernel chain rule takes the following form $${\hat{\mathcal{C}}}_{XY}^{\pi }={\hat{\mathcal{C}}}_{X\mid Y}{\hat{\mathcal{C}}}_{YY}^{\pi }=\mathbf{{\rm Y}}(\mathbf{G}+\lambda \mathbf{I}{)}^{-1}\tilde{\mathbf{G}}\mathrm{diag}(\mathit{\alpha }){\tilde{\mathbf{\Phi }}}^T$$

2.3 Kernel Bayes’ rule

In probability theory, a posterior distribution can be expressed in terms of a prior distribution and a likelihood function as $$Q(Y\mid x)=\frac{P(x\mid Y)\pi (Y)}{Q(x)}\text{ where }Q(x)={\int }_{\mathrm{\Omega }}P(x\mid y)\mathrm{d}\pi (y)$$

The analogue of this rule in the kernel embedding framework expresses the kernel embedding of the conditional distribution in terms of conditional embedding operators which are modified by the prior distribution $${\mu }_{Y\mid x}^{\pi }={\mathcal{C}}_{Y\mid X}^{\pi }\phi (x)={\mathcal{C}}_{YX}^{\pi }{\left({\mathcal{C}}_{XX}^{\pi }\right)}^{-1}\phi (x)$$ where from the chain rule: $${\mathcal{C}}_{YX}^{\pi }={\left({\mathcal{C}}_{X\mid Y}{\mathcal{C}}_{YY}^{\pi }\right)}^T$$

In practical implementations, the kernel Bayes’ rule takes the following form $${\hat{\mu }}_{Y\mid x}^{\pi }={\hat{\mathcal{C}}}_{YX}^{\pi }{({\left({\hat{\mathcal{C}}}_{XX}\right)}^2+\tilde{\lambda }\mathbf{I})}^{-1}{\hat{\mathcal{C}}}_{XX}^{\pi }\phi (x)=\tilde{\mathbf{\Phi }}{\mathbf{\Lambda }}^T{((\mathbf{D}\mathbf{K}{)}^2+\tilde{\lambda }\mathbf{I})}^{-1}{\mathbf{K}\mathbf{D}\mathbf{K}}_x$$ where $$\mathbf{\Lambda }=(\mathbf{G}+\tilde{\lambda }\mathbf{I}{)}^{-1}\tilde{\mathbf{G}}\mathrm{diag}(\mathit{\alpha }),\mathbf{D}=\mathrm{diag}((\mathbf{G}+\tilde{\lambda }\mathbf{I}{)}^{-1}\tilde{\mathbf{G}}\mathit{\alpha })$$

Two regularisation parameters are used in this framework: $\lambda$ for the estimation of ${\hat{\mathcal{C}}}_{YX}^{\pi },{\hat{\mathcal{C}}}_{XX}^{\pi }=\mathbf{{\rm Y}}\mathbf{D}{\mathbf{{\rm Y}}}^T$ and $\tilde{\lambda }$ for the estimation of the final conditional embedding operator## References

$${\hat{\mathcal{C}}}_{Y\mid X}^{\pi }={\hat{\mathcal{C}}}_{YX}^{\pi }{({\left({\hat{\mathcal{C}}}_{XX}^{\pi }\right)}^2+\tilde{\lambda }\mathbf{I})}^{-1}{\hat{\mathcal{C}}}_{XX}^{\pi }$$

The latter regularisation is done on square of ${\hat{\mathcal{C}}}_{XX}^{\pi }$ because $D$ may not be positive definite.