Gaussian processes



“Gaussian Processes” are stochastic processes/fields with jointly Gaussian distributions of observations. The most familiar of these to many of us is the Gauss-Markov process, a.k.a. the Wiener process, but there are many others. These processes are convenient due to certain useful properties of the multivariate Gaussian distribution e.g. being uniquely specified by first and second moments, nice behaviour under various linear operations, kernel tricks…. Especially famous applications include Gaussian process regression and spatial statistics.

Gauss, with what I believe is possibly the telegraph he invented. That is not the Gaussian process I mean here (Gauss, after all did not invent that) I just think it is cool.

Check out Ti’s Interactive visualization of Gaussian processes.s

Gaussian processes are, specifically, probabilistic distributions over random functions \(\mathcal{T}\to \mathbb{C}\) for some index (or argument) set \(\mathcal{T}\) often taken to be \(\mathcal{T}:=\mathbb{R}^d\).

We typically work with a mean-zero process, in which case for every finite set \(\mathbf{f}:=\{f(t_k);k=1,\dots,K\}\) of observations of that process, the joint distribution is mean-zero Gaussian, \[\begin{aligned} \mathbf{f}(t) &\sim \operatorname{GP}\left(0, \kappa(t, t';\mathbf{\theta})\right) \\ &\Rightarrow\\ p(\mathbf{f}) &=(2\pi )^{-{\frac {K}{2}}}\det({\boldsymbol {\mathrm{K} }})^{-{\frac {1}{2}}}\,e^{-{\frac {1}{2}}\mathbf {f}^{\!{\mathsf {T}}}{\boldsymbol {\mathrm{K} }}^{-1}\mathbf {f}}\\ &=\mathcal{N}(\mathbf{f};0, \mathrm{K}). \end{aligned}\] where \(\mathrm{K}\) is the sample covariance matrix defined such that its entries are given by \(\mathrm{K}_{jk}=\kappa(t_j,t_k).\) That is, this is the covariance kernel that maps from function argument\(t\) — to second moment of function values. In this case, we are specifying only the second moments and this is giving us all the remaining properties of the process.

References

Abrahamsen, Petter. 1997. “A Review of Gaussian Random Fields and Correlation Functions.” http://publications.nr.no/publications.nr.no/directdownload/publications.nr.no/rask/old/917_Rapport.pdf.
Adler, Robert J. 2010. The Geometry of Random Fields. SIAM ed. Philadelphia: Society for Industrial and Applied Mathematics.
Adler, Robert J., and Jonathan E. Taylor. 2007. Random Fields and Geometry. Springer Monographs in Mathematics 115. New York: Springer. https://doi.org/10.1007/978-0-387-48116-6.
Agrell, Christian. n.d. “Gaussian Processes with Linear Operator Inequality Constraints,” 36.
Alexanderian, Alen. 2015. “A Brief Note on the Karhunen-Loève Expansion.” arXiv:1509.07526 [math], October. http://arxiv.org/abs/1509.07526.
Bochner, Salomon. 1959. Lectures on Fourier Integrals. Princeton University Press.
Dym, H., and Henry P. McKean. 2008. Gaussian Processes, Function Theory, and the Inverse Spectral Problem. Dover ed. Dover Books on Mathematics. Mineola, N.Y: Dover Publications.
Kanagawa, Motonobu, and Kenji Fukumizu. 2014. “Recovering Distributions from Gaussian RKHS Embeddings.” In Journal of Machine Learning Research. http://www.jmlr.org/proceedings/papers/v33/kanagawa14.pdf.
Kanagawa, Motonobu, Philipp Hennig, Dino Sejdinovic, and Bharath K. Sriperumbudur. 2018. “Gaussian Processes and Kernel Methods: A Review on Connections and Equivalences.” arXiv:1807.02582 [cs, Stat], July. http://arxiv.org/abs/1807.02582.
Khintchine, A. 1934. “Korrelationstheorie der stationären stochastischen Prozesse.” Mathematische Annalen 109 (1): 604–15. https://doi.org/10.1007/BF01449156.
Liu, Wei, and Michael Röckner. 2015. Stochastic Partial Differential Equations: An Introduction. Springer.
Lukić, Milan, and Jay Beder. 2001. “Stochastic Processes with Sample Paths in Reproducing Kernel Hilbert Spaces.” Transactions of the American Mathematical Society 353 (10): 3945–69. https://doi.org/10.1090/S0002-9947-01-02852-5.
Papoulis, Athanasios. 1984. Probability, Random Variables and Stochastic Processes. https://ui.adsabs.harvard.edu/abs/1984prvs.book…..P.
Rasmussen, Carl Edward, and Hannes Nickisch. 2010. “Gaussian Processes for Machine Learning (GPML) Toolbox.” Journal of Machine Learning Research 11 (Nov): 3011–15. http://www.jmlr.org/papers/v11/rasmussen10a.html.
Rasmussen, Carl Edward, and Christopher K. I. Williams. 2006. Gaussian Processes for Machine Learning. Adaptive Computation and Machine Learning. Cambridge, Mass: MIT Press. http://www.gaussianprocess.org/gpml/.
Yaglom, A. M. 1987. Correlation Theory of Stationary and Related Random Functions. Volume II: Supplementary Notes and References. Springer Series in Statistics. New York, NY: Springer Science & Business Media.

No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.