SGD and stochastic gradient (and usually also message passing) meets variational inference.
If we add in amortization as well, we then have a variational autoencoder.
Introductory reading
Wingate and Weber (2013), Ranganath, Gerrish, and Blei (2013), Li and Turner (2016).
References
Blei, David M., Alp Kucukelbir, and Jon D. McAuliffe. 2017. “Variational Inference: A Review for Statisticians.” Journal of the American Statistical Association 112 (518): 859–77.
Detommaso, Gianluca, Tiangang Cui, Alessio Spantini, Youssef Marzouk, and Robert Scheichl. 2018. “A Stein Variational Newton Method.” In Proceedings of the 32nd International Conference on Neural Information Processing Systems, 9187–97. NIPS’18. Red Hook, NY, USA: Curran Associates Inc.
Dhaka, Akash Kumar, and Alejandro Catalina. 2020. “Robust, Accurate Stochastic Optimization for Variational Inference,” 13.
Dhaka, Akash Kumar, Alejandro Catalina, Manushi Welandawe, Michael Riis Andersen, Jonathan Huggins, and Aki Vehtari. 2021. “Challenges and Opportunities in High-Dimensional Variational Inference.” arXiv:2103.01085 [Cs, Stat], March.
Fraccaro, Marco, Sø ren Kaae Sø nderby, Ulrich Paquet, and Ole Winther. 2016. “Sequential Neural Models with Stochastic Layers.” In Advances in Neural Information Processing Systems 29, edited by D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, 2199–2207. Curran Associates, Inc.
Graves, Alex. 2011. “Practical Variational Inference for Neural Networks.” In Proceedings of the 24th International Conference on Neural Information Processing Systems, 2348–56. NIPS’11. USA: Curran Associates Inc.
Hoffman, Matt, David M. Blei, Chong Wang, and John Paisley. 2013. “Stochastic Variational Inference.” arXiv:1206.7051 [Cs, Stat] 14 (1).
Hoffman, Matthew, and David Blei. 2015. “Stochastic Structured Variational Inference.” In PMLR, 361–69.
Jordan, Michael I., Zoubin Ghahramani, Tommi S. Jaakkola, and Lawrence K. Saul. 1999. “An Introduction to Variational Methods for Graphical Models.” Machine Learning 37 (2): 183–233.
Kingma, Diederik P. 2017. “Variational Inference & Deep Learning: A New Synthesis.”
Li, Yingzhen, and Richard E Turner. 2016. “Rényi Divergence Variational Inference.” In Advances in Neural Information Processing Systems, 29:1081–89. Red Hook, NY, USA: Curran Associates, Inc.
Liu, Qiang, and Dilin Wang. 2019. “Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm.” In Advances In Neural Information Processing Systems.
Matthews, Alexander Graeme de Garis. 2017. “Scalable Gaussian Process Inference Using Variational Methods.” Thesis, University of Cambridge.
Plötz, Tobias, Anne S. Wannenwetsch, and Stefan Roth. 2018. “Stochastic Variational Inference with Gradient Linearization.” In CVPR.
Ranganath, Rajesh, Sean Gerrish, and David M. Blei. 2013. “Black Box Variational Inference.” arXiv:1401.0118 [Cs, Stat], December.
Ranganath, Rajesh, Dustin Tran, Jaan Altosaar, and David Blei. 2016. “Operator Variational Inference.” In Advances in Neural Information Processing Systems 29, edited by D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, 496–504. Curran Associates, Inc.
Rezende, Danilo Jimenez, Shakir Mohamed, and Daan Wierstra. 2015. “Stochastic Backpropagation and Approximate Inference in Deep Generative Models.” In Proceedings of ICML.
Salimans, Tim, Diederik Kingma, and Max Welling. 2015. “Markov Chain Monte Carlo and Variational Inference: Bridging the Gap.” In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), 1218–26. ICML’15. Lille, France: JMLR.org.
Titsias, Michalis K., and Miguel Lázaro-Gredilla. 2014. “Doubly Stochastic Variational Bayes for Non-Conjugate Inference.” In Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32, II-1971–II-1980. ICML’14. Beijing, China: JMLR.org.
Wingate, David, and Theophane Weber. 2013. “Automated Variational Inference in Probabilistic Programming.” arXiv:1301.1299 [Cs, Stat], January.
No comments yet. Why not leave one?