A placeholder for learning on curved spaces. Not discussed: learning OF the curvature of spaces.
AFAICT this usually boils down to defining an appropriate stochastic process on a manifold.
Learning on a given manifold
Learning where there is an a priori manifold seems to also be a usage here? For example the manifold of positive definite matrices is treated in depth in Chikuse and 筑瀬 (2003).
See the work of, e.g.
Manifold optimisation implementations:
- pytorch: Lezcano/geotorch: Constrained optimization toolkit for PyTorch (Lezcano Casado 2019)
- MATLAB: manopt,
- Python: pymanopt.
- Julia: Manopt.jl
- Python: Nina Miolane et al’s Geomstats project.
- C++: ROPTLIB (Huang et al. 2018)
- R: ManifoldOptim wrapes ROPTLIB (Martin et al. 2016)
There are at least two textbooks online:
Information Geometry
The unholy offspring of Fisher information and differential geometry, about which I know little except that it sounds like it should be intuitive. It is probably synonymous with some of the other items on this page if I could sort out all this terminology. See information geometry.
Hamiltonian Monte Carlo
You can also discuss Hamiltonian Monte Carlo in this setting. I will not.
Langevin Monte Carlo
Girolami et al discuss Langevin Monte Carlo in this context.
Natural gradient
See natural gradients.
Homogeneous probability
Albert Tarantola’s framing, from his manuscript. How does it relate to information geometry? I don’t know yet. Haven’t had time to read. Also not a common phrasing, which is a danger sign.
Incoming
- Agustinus Kristiadi, Fisher Information Matrix
- Agustinus Kristiadi, Hessian and Curvatures in Machine Learning: A Differential-Geometric View
- Agustinus Kristiadi, Notes on Riemannian Geometry
- Agustinus Kristiadi, Optimization and Gradient Descent on Riemannian Manifolds
References
Absil, P.-A, R Mahony, and R Sepulchre. 2008. Optimization algorithms on matrix manifolds. Princeton, N.J.; Woodstock: Princeton University Press.
Amari, Shun-ichi. 1998. “Natural Gradient Works Efficiently in Learning.” Neural Computation 10 (2): 251–76.
Amari, Shunʼichi. 1987. “Differential Geometrical Theory of Statistics.” In Differential Geometry in Statistical Inference, 19–94.
———. 2001. “Information Geometry on Hierarchy of Probability Distributions.” IEEE Transactions on Information Theory 47: 1701–11.
Aswani, Anil, Peter Bickel, and Claire Tomlin. 2011. “Regression on Manifolds: Estimation of the Exterior Derivative.” The Annals of Statistics 39 (1): 48–81.
Azangulov, Iskander, Andrei Smolensky, Alexander Terenin, and Viacheslav Borovitskiy. 2022. “Stationary Kernels and Gaussian Processes on Lie Groups and Their Homogeneous Spaces I: The Compact Case.” arXiv.
Barndorff-Nielsen, O E. 1987. “Differential and Integral Geometry in Statistical Inference.” In Differential Geometry in Statistical Inference. Sn Aarhus.
Betancourt, Michael, Simon Byrne, Sam Livingstone, and Mark Girolami. 2017. “The Geometric Foundations of Hamiltonian Monte Carlo.” Bernoulli 23 (4A): 2257–98.
Borovitskiy, Viacheslav, Alexander Terenin, Peter Mostowsky, and Marc Peter Deisenroth. 2020. “Matérn Gaussian Processes on Riemannian Manifolds.” arXiv:2006.10160 [Cs, Stat], June.
Boumal, Nicolas. 2013. “On Intrinsic Cramér-Rao Bounds for Riemannian Submanifolds and Quotient Manifolds.” IEEE Transactions on Signal Processing 61 (7): 1809–21.
Boumal, Nicolas, Bamdev Mishra, P.-A. Absil, and Rodolphe Sepulchre. 2014. “Manopt, a Matlab Toolbox for Optimization on Manifolds.” Journal of Machine Learning Research 15: 1455–59.
Boumal, Nicolas, Amit Singer, P.-A. Absil, and Vincent D. Blondel. 2014. “Cramér-Rao Bounds for Synchronization of Rotations.” Information and Inference 3 (1): 1–39.
Carlsson, Gunnar, Tigran Ishkhanov, Vin de Silva, and Afra Zomorodian. 2008. “On the Local Behavior of Spaces of Natural Images.” International Journal of Computer Vision 76 (1): 1–12.
Chen, Minhua, J. Silva, J. Paisley, Chunping Wang, D. Dunson, and L. Carin. 2010. “Compressive Sensing on Manifolds Using a Nonparametric Mixture of Factor Analyzers: Algorithm and Performance Bounds.” IEEE Transactions on Signal Processing 58 (12): 6140–55.
Chikuse, Yasuko, and 筑瀬靖子. 2003. Statistics on Special Manifolds. New York, NY: Springer New York.
Fernández-Martínez, J. L., Z. Fernández-Muñiz, J. L. G. Pallero, and L. M. Pedruelo-González. 2013. “From Bayes to Tarantola: New Insights to Understand Uncertainty in Inverse Problems.” Journal of Applied Geophysics 98 (November): 62–72.
França, Guilherme, Alessandro Barp, Mark Girolami, and Michael I. Jordan. 2021. “Optimization on Manifolds: A Symplectic Approach,” July.
Ge, Rong, and Tengyu Ma. 2017. “On the Optimization Landscape of Tensor Decompositions.” In Advances In Neural Information Processing Systems.
Girolami, Mark, and Ben Calderhead. 2011. “Riemann Manifold Langevin and Hamiltonian Monte Carlo Methods.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 73 (2): 123–214.
Głuch, Grzegorz, and Rüdiger Urbanke. 2021. “Noether: The More Things Change, the More Stay the Same.” arXiv:2104.05508 [Cs, Stat], April.
Hosseini, Reshad, and Suvrit Sra. 2015. “Manifold Optimization for Gaussian Mixture Models.” arXiv Preprint arXiv:1506.07677.
Huang, Wen, P.-A. Absil, Kyle A. Gallivan, and Paul Hand. 2018. “ROPTLIB: An Object-Oriented C++ Library for Optimization on Riemannian Manifolds.” ACM Transactions on Mathematical Software 44 (4): 43:1–21.
Lauritzen, S L. 1987. “Statistical Manifolds.” In Differential Geometry in Statistical Inference, 164. JSTOR.
Ley, Christophe, Slađana Babić, and Domien Craens. 2021. “Flexible Models for Complex Data with Applications.” Annual Review of Statistics and Its Application 8 (1): 369–91.
Lezcano Casado, Mario. 2019. “Trivializations for Gradient-Based Optimization on Manifolds.” In Advances in Neural Information Processing Systems. Vol. 32. Curran Associates, Inc.
Manton, Jonathan H. 2013. “A Primer on Stochastic Differential Geometry for Signal Processing.” IEEE Journal of Selected Topics in Signal Processing 7 (4): 681–99.
Mardia, Kanti V., and Peter E. Jupp. 2009. Directional Statistics. John Wiley & Sons.
Martin, Sean, Andrew M. Raim, Wen Huang, and Kofi P. Adragni. 2016. “ManifoldOptim: An R Interface to the ROPTLIB Library for Riemannian Manifold Optimization,” December.
Miolane, Nina, Johan Mathe, Claire Donnat, Mikael Jorda, and Xavier Pennec. 2018. “Geomstats: A Python Package for Riemannian Geometry in Machine Learning.” arXiv:1805.08308 [Cs, Stat], May.
Mosegaard, Klaus, and Albert Tarantola. 1995. “Monte Carlo Sampling of Solutions to Inverse Problems.” Journal of Geophysical Research: Solid Earth 100 (B7): 12431–47.
Mukherjee, Sayan, Qiang Wu, and Ding-Xuan Zhou. 2010. “Learning Gradients on Manifolds.” Bernoulli 16 (1): 181–207.
Peters, Jan. 2010. “Policy Gradient Methods.” Scholarpedia 5 (11): 3698.
Popov, Andrey Anatoliyevich. 2022. “Combining Data-Driven and Theory-Guided Models in Ensemble Data Assimilation.” ETD. Virginia Tech.
Rao, Vinayak, Lizhen Lin, and David Dunson. n.d. “Bayesian Inference on the Stiefel Manifold,” 33.
Saul, Lawrence K. 2023. “A Geometrical Connection Between Sparse and Low-Rank Matrices and Its Application to Manifold Learning.” Transactions on Machine Learning Research, January.
Seshadhri, C., Aneesh Sharma, Andrew Stolman, and Ashish Goel. 2020. “The Impossibility of Low-Rank Representations for Triangle-Rich Complex Networks.” Proceedings of the National Academy of Sciences 117 (11): 5631–37.
Stam, A. J. 1982. “Limit Theorems for Uniform Distributions on Spheres in High-Dimensional Euclidean Spaces.” Journal of Applied Probability 19 (1): 221–28.
Steinke, Florian, and Matthias Hein. 2009. “Non-Parametric Regression Between Manifolds.” In Advances in Neural Information Processing Systems 21, 1561–68. Curran Associates, Inc.
Townsend, James, Niklas Koep, and Sebastian Weichwald. 2016. “Pymanopt: A Python Toolbox for Optimization on Manifolds Using Automatic Differentiation.” Journal of Machine Learning Research 17 (137): 1–5.
Transtrum, Mark K, Benjamin B Machta, and James P Sethna. 2011. “The Geometry of Nonlinear Least Squares with Applications to Sloppy Models and Optimization.” Physical Review E 83 (3): 036701.
Wang, Yu Guang, and Xiaosheng Zhuang. 2016. “Tight Framelets and Fast Framelet Transforms on Manifolds.” arXiv:1608.04026 [Math], August.
Xifara, T., C. Sherlock, S. Livingstone, S. Byrne, and M. Girolami. 2014. “Langevin Diffusions and the Metropolis-Adjusted Langevin Algorithm.” Statistics & Probability Letters 91 (Supplement C): 14–19.
No comments yet. Why not leave one?