Simulating Gaussian processes

March 17, 2022 — March 17, 2022

Gaussian
Hilbert space
kernel tricks
Lévy processes
nonparametric
regression
spatial
stochastic processes
time series

Assumed audience:

ML people

How can I simulate a Gaussian Processes with a given covariance? Handy in GP regression, especially GP functional regression and spatial statistics.

Historical overview in Liu et al. (2019).

Figure 1

1 Incoming

2 Lattice tricks

On lattices we can make some computational shortcuts. See GP simulation on lattices.

3 Basis tricks

TBD

4 Simulating from posterior GPs

Probably many tricks, but I know of pathwise GPs.

5 References

Abrahamsen, Kvernelv, and Barker. 2018. “Simulation Of Gaussian Random Fields Using The Fast Fourier Transform (Fft).” In.
Alexanderian. 2015. “A Brief Note on the Karhunen-Loève Expansion.” arXiv:1509.07526 [Math].
Bingham, and Symons. 2022. “Gaussian Random Fields: With and Without Covariances.” Theory of Probability and Mathematical Statistics.
Bolin. 2016. Models and Methods for Random Fields in Spatial Statistics with Computational Efficiency from Markov Properties.
Chan, Grace, and Wood. 1997. “Algorithm AS 312: An Algorithm for Simulating Stationary Gaussian Random Fields.” Journal of the Royal Statistical Society: Series C (Applied Statistics).
Chan, G., and Wood. 1999. “Simulation of Stationary Gaussian Vector Fields.” Statistics and Computing.
Chilès, and Lantuéjoul. 2005. “Prediction by Conditional Simulation: Models and Algorithms.” In Space, Structure and Randomness: Contributions in Honor of Georges Matheron in the Field of Geostatistics, Random Sets and Mathematical Morphology. Lecture Notes in Statistics.
Choromanski, and Sindhwani. 2016. “Recycling Randomness with Structure for Sublinear Time Kernel Expansions.” arXiv:1605.09049 [Cs, Stat].
Cotter, Roberts, Stuart, et al. 2013. “MCMC Methods for Functions: Modifying Old Algorithms to Make Them Faster.” Statistical Science.
Davies, and Bryant. 2013. “On Circulant Embedding for Gaussian Random Fields in R.” Journal of Statistical Software.
Dietrich, and Newsam. 1993. “A Fast and Exact Method for Multidimensional Gaussian Stochastic Simulations.” Water Resources Research.
———. 1997. “Fast and Exact Simulation of Stationary Gaussian Processes Through Circulant Embedding of the Covariance Matrix.” SIAM Journal on Scientific Computing.
Doucet. 2010. “A Note on Efficient Conditional Simulation of Gaussian Distributions.”
Durrande, Adam, Bordeaux, et al. 2019. “Banded Matrix Operators for Gaussian Markov Models in the Automatic Differentiation Era.” In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics.
Ellis, and Lay. 1992. “Factorization of Finite Rank Hankel and Toeplitz Matrices.” Linear Algebra and Its Applications.
Erhel, Oumouni, Pichot, et al. n.d. “Analysis of Continuous Spectral Method for Sampling Stationary Gaussian Random Fields.”
Galy-Fajou, Perrone, and Opper. 2021. “Flexible and Efficient Inference with Particles for the Variational Gaussian Approximation.” Entropy.
Gilboa, Saatçi, and Cunningham. 2015. “Scaling Multidimensional Inference for Structured Gaussian Processes.” IEEE Transactions on Pattern Analysis and Machine Intelligence.
Graham, Kuo, Nuyens, et al. 2017a. “Analysis of Circulant Embedding Methods for Sampling Stationary Random Fields.” arXiv:1710.00751 [Math].
———, et al. 2017b. “Circulant Embedding with QMC — Analysis for Elliptic PDE with Lognormal Coefficients.” arXiv:1710.09254 [Math].
Gray. 2006. Toeplitz and Circulant Matrices: A Review.
Guinness, and Fuentes. 2016. “Circulant Embedding of Approximate Covariances for Inference From Gaussian Data on Large Lattices.” Journal of Computational and Graphical Statistics.
Haran. 2011. “Gaussian Random Field Models for Spatial Data.” In Handbook of Markov Chain Monte Carlo.
Heinig, and Rost. 2011. “Fast Algorithms for Toeplitz and Hankel Matrices.” Linear Algebra and Its Applications.
Hoffman, and Ribak. 1991. “Constrained Realizations of Gaussian Fields-A Simple Algorithm.” The Astrophysical Journal.
Lang, and Potthoff. 2011. “Fast Simulation of Gaussian Random Fields.” Monte Carlo Methods and Applications.
Latz, Eisenberger, and Ullmann. 2019. “Fast Sampling of Parameterised Gaussian Random Fields.” Computer Methods in Applied Mechanics and Engineering.
Liu, Li, Sun, et al. 2019. “Advances in Gaussian Random Field Generation: A Review.” Computational Geosciences.
Murray, Adams, and MacKay. 2010. “Elliptical Slice Sampling.” In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics.
Powell. 2014. “Generating Realisations of Stationary Gaussian Random Fields by Circulant Embedding.” Matrix.
Rue, Havard. 2001. “Fast Sampling of Gaussian Markov Random Fields.” Journal of the Royal Statistical Society. Series B (Statistical Methodology).
Rue, Håvard, and Held. 2005. Gaussian Markov Random Fields: Theory and Applications. Monographs on Statistics and Applied Probability 104.
Sidén. 2020. Scalable Bayesian Spatial Analysis with Gaussian Markov Random Fields. Linköping Studies in Statistics.
Simpson, Daniel Peter. 2008. “Krylov Subspace Methods for Approximating Functions of Symmetric Positive Definite Matrices with Applications to Applied Statistics and Anomalous Diffusion.”
Simpson, Daniel P., Turner, Strickland, et al. 2013. “Scalable Iterative Methods for Sampling from Massive Gaussian Random Vectors.”
Stroud, Stein, and Lysen. 2017. “Bayesian and Maximum Likelihood Estimation for Gaussian Processes on an Incomplete Lattice.” Journal of Computational and Graphical Statistics.
Teichmann, and van den Boogaart. 2016. “Efficient Simulation of Stationary Multivariate Gaussian Random Fields with Given Cross-Covariance.” Applied Mathematics.
Whittle, Peter. 1952. “Some Results in Time Series Analysis.” Scandinavian Actuarial Journal.
Whittle, P. 1953a. “The Analysis of Multiple Stationary Time Series.” Journal of the Royal Statistical Society: Series B (Methodological).
———. 1953b. “Estimation and Information in Stationary Time Series.” Arkiv För Matematik.
Whittle, P. 1954. “On Stationary Processes in the Plane.” Biometrika.
Wilson, Borovitskiy, Terenin, et al. 2021. “Pathwise Conditioning of Gaussian Processes.” Journal of Machine Learning Research.
Yang, Fang, Duan, et al. 2018. “Fast Low-Rank Bayesian Matrix Completion with Hierarchical Gaussian Prior Models.” IEEE Transactions on Signal Processing.
Ye, and Lim. 2016. “Every Matrix Is a Product of Toeplitz Matrices.” Foundations of Computational Mathematics.