Gaussian processes on lattices


Gaussian Processes with a stationary kernel are faster if you are working on a grid of points. I have not used this trick, but I understand it involves various simplifications arising from the structure of Gram matrices which end up being kronecker products of Toeplitz matrices under lexical ordering of the input points (sounds plausible but I have not worked this out on paper). The keyword to highlight this is Kronecker inference in the ML literature . Another keyword is circulant embedding, the approximation of a Toeplitz matrix by a circulant matrix, which apparently enables one to leverage fast Fourier transforms to calculate some quantities of interest, and some other nice linear algebra properties besides. That would imply that this method is has its origins in Whittle likelihoods , if I am not mistaken.

Lattice GP methods complements, perhaps, the trick of filtering Gaussian processes, which can also exploit structure lattice inputs, although the setup is different between these.

TBC.

In regression

OK, so what does this allow us to do with posterior inference in GPs? Apparently quite a lot. The KISS-GP method presumably leverages something similar. So do Saatçi (2012) and Flaxman et al. (2015).

Flaxman et al. (2015) deals with the lattice structure trick in a non-Gaussian likelihood setting using Laplace approximation.

TBC

References

Abrahamsen, P., V. Kvernelv, and D. Barker. 2018. In, 2018:1–14. European Association of Geoscientists & Engineers.
Alexanderian, Alen. 2015. arXiv:1509.07526 [Math], October.
Chan, Grace, and Andrew T.A. Wood. 1997. Journal of the Royal Statistical Society: Series C (Applied Statistics) 46 (1): 171–81.
Chan, G., and A. T. A. Wood. 1999. Statistics and Computing 9 (4): 265–68.
Charlier, Benjamin, Jean Feydy, Joan Alexis Glaunès, François-David Collin, and Ghislain Durif. 2021. Journal of Machine Learning Research 22 (74): 1–6.
Choromanski, Krzysztof, and Vikas Sindhwani. 2016. arXiv:1605.09049 [Cs, Stat], May.
Choudhuri, Nidhan, Subhashis Ghosal, and Anindya Roy. 2004. Biometrika 91 (1): 211–18.
Cotter, S. L., G. O. Roberts, A. M. Stuart, and D. White. 2013. Statistical Science 28 (3): 424–46.
Davies, Tilman M., and David Bryant. 2013. Journal of Statistical Software 55 (9).
Dietrich, C. R., and G. N. Newsam. 1993. Water Resources Research 29 (8): 2861–69.
———. 1997. SIAM Journal on Scientific Computing 18 (4): 1088–1107.
Durrande, Nicolas, Vincent Adam, Lucas Bordeaux, Stefanos Eleftheriadis, and James Hensman. 2019. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, 2780–89. PMLR.
Ellis, Robert L., and David C. Lay. 1992. Linear Algebra and Its Applications 173 (August): 19–38.
Erhel, Jocelyne, Mestapha Oumouni, Géraldine Pichot, and Franck Schoefs. n.d. “Analysis of Continuous Spectral Method for Sampling Stationary Gaussian Random Fields,” 26.
Flaxman, Seth, Andrew Gordon Wilson, Daniel B Neill, Hannes Nickisch, and Alexander J Smola. 2015. “Fast Kronecker Inference in Gaussian Processes with Non-Gaussian Likelihoods.” In, 10.
Gilboa, E., Y. Saatçi, and J. P. Cunningham. 2015. IEEE Transactions on Pattern Analysis and Machine Intelligence 37 (2): 424–36.
Graham, Ivan G., Frances Y. Kuo, Dirk Nuyens, Rob Scheichl, and Ian H. Sloan. 2017a. arXiv:1710.00751 [Math], October.
———. 2017b. arXiv:1710.09254 [Math], October.
Gray, Robert M. 2006. Toeplitz and Circulant Matrices: A Review. Vol. 2.
Guinness, Joseph, and Montserrat Fuentes. 2016. Journal of Computational and Graphical Statistics 26 (1): 88–97.
Heinig, Georg, and Karla Rost. 2011. Linear Algebra and Its Applications 435 (1): 1–59.
Kroese, Dirk P., and Zdravko I. Botev. 2013. arXiv:1308.0399 [Stat], August.
Lang, Annika, and Jürgen Potthoff. 2011. Monte Carlo Methods and Applications 17 (3).
Loper, Jackson, David Blei, John P. Cunningham, and Liam Paninski. 2021. arXiv:2003.05554 [Cs, Stat], October.
Nowak, W., and A. Litvinenko. 2013. Mathematical Geosciences 45 (4): 411–35.
Pleiss, Geoff, Jacob R. Gardner, Kilian Q. Weinberger, and Andrew Gordon Wilson. 2018. In. arXiv.
Powell, Catherine E. 2014. “Generating Realisations of Stationary Gaussian Random Fields by Circulant Embedding.” Matrix 2 (2): 1.
Rue, Havard. 2001. Journal of the Royal Statistical Society. Series B (Statistical Methodology) 63 (2): 325–38.
Rue, Håvard, and Leonhard Held. 2005. Gaussian Markov Random Fields: Theory and Applications. Monographs on Statistics and Applied Probability 104. Boca Raton: Chapman & Hall/CRC.
Saatçi, Yunus. 2012. Ph.D., University of Cambridge.
Saatçi, Yunus, Ryan Turner, and Carl Edward Rasmussen. 2010. In Proceedings of the 27th International Conference on International Conference on Machine Learning, 927–34. ICML’10. Madison, WI, USA: Omnipress.
Sigrist, Fabio, Hans R. Künsch, and Werner A. Stahel. 2015a. Application/pdf. Journal of Statistical Software 63 (14).
———. 2015b. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 77 (1): 3–33.
Stroud, Jonathan R., Michael L. Stein, and Shaun Lysen. 2017. Journal of Computational and Graphical Statistics 26 (1): 108–20.
Teichmann, Jakob, and Karl-Gerald van den Boogaart. 2016. Applied Mathematics 7 (17): 2183–94.
Ubaru, Shashanka, Jie Chen, and Yousef Saad. 2017. SIAM Journal on Matrix Analysis and Applications 38 (4): 1075–99.
Whittle, P. 1954. “On Stationary Processes in the Plane.” Biometrika 41 (3/4): 434–49.
Whittle, P. 1952. Biometrika 39 (3-4): 309–18.
———. 1953a. Journal of the Royal Statistical Society: Series B (Methodological) 15 (1): 125–39.
———. 1953b. Arkiv För Matematik 2 (5): 423–34.
Whittle, Peter. 1952. Scandinavian Actuarial Journal 1952 (1-2): 48–60.
Wilson, Andrew Gordon, Christoph Dann, and Hannes Nickisch. 2015. arXiv:1511.01870 [Cs, Stat], November.
Wilson, Andrew Gordon, and Hannes Nickisch. 2015. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning - Volume 37, 1775–84. ICML’15. Lille, France: JMLR.org.
Wilson, James T, Viacheslav Borovitskiy, Alexander Terenin, Peter Mostowsky, and Marc Peter Deisenroth. 2021. Journal of Machine Learning Research 22 (105): 1–47.
Ye, Ke, and Lek-Heng Lim. 2016. Foundations of Computational Mathematics 16 (3): 577–98.

No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.