SGD and stochastic gradient (and usually also message passing) meets variational inference.
If we add in amortization as well, we then have a variational autoencoder.
Introductory reading
Wingate and Weber (2013), Ranganath, Gerrish, and Blei (2013), Li and Turner (2016).
References
Blei, David M., Alp Kucukelbir, and Jon D. McAuliffe. 2017. βVariational Inference: A Review for Statisticians.β Journal of the American Statistical Association 112 (518): 859β77.
Detommaso, Gianluca, Tiangang Cui, Alessio Spantini, Youssef Marzouk, and Robert Scheichl. 2018. βA Stein Variational Newton Method.β In Proceedings of the 32nd International Conference on Neural Information Processing Systems, 9187β97. NIPSβ18. Red Hook, NY, USA: Curran Associates Inc.
Dhaka, Akash Kumar, and Alejandro Catalina. 2020. βRobust, Accurate Stochastic Optimization for Variational Inference,β 13.
Dhaka, Akash Kumar, Alejandro Catalina, Manushi Welandawe, Michael Riis Andersen, Jonathan Huggins, and Aki Vehtari. 2021. βChallenges and Opportunities in High-Dimensional Variational Inference.β arXiv:2103.01085 [Cs, Stat], March.
Fraccaro, Marco, SΓΈ ren Kaae SΓΈ nderby, Ulrich Paquet, and Ole Winther. 2016. βSequential Neural Models with Stochastic Layers.β In Advances in Neural Information Processing Systems 29, edited by D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, 2199β2207. Curran Associates, Inc.
Graves, Alex. 2011. βPractical Variational Inference for Neural Networks.β In Proceedings of the 24th International Conference on Neural Information Processing Systems, 2348β56. NIPSβ11. USA: Curran Associates Inc.
Hoffman, Matt, David M. Blei, Chong Wang, and John Paisley. 2013. βStochastic Variational Inference.β arXiv:1206.7051 [Cs, Stat] 14 (1).
Hoffman, Matthew, and David Blei. 2015. βStochastic Structured Variational Inference.β In PMLR, 361β69.
Jordan, Michael I., Zoubin Ghahramani, Tommi S. Jaakkola, and Lawrence K. Saul. 1999. βAn Introduction to Variational Methods for Graphical Models.β Machine Learning 37 (2): 183β233.
Kingma, Diederik P. 2017. βVariational Inference & Deep Learning: A New Synthesis.β
Li, Yingzhen, and Richard E Turner. 2016. βRΓ©nyi Divergence Variational Inference.β In Advances in Neural Information Processing Systems, 29:1081β89. Red Hook, NY, USA: Curran Associates, Inc.
Liu, Qiang, and Dilin Wang. 2019. βStein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm.β In Advances In Neural Information Processing Systems.
Matthews, Alexander Graeme de Garis. 2017. βScalable Gaussian Process Inference Using Variational Methods.β Thesis, University of Cambridge.
PlΓΆtz, Tobias, Anne S. Wannenwetsch, and Stefan Roth. 2018. βStochastic Variational Inference with Gradient Linearization.β In CVPR.
Ranganath, Rajesh, Sean Gerrish, and David M. Blei. 2013. βBlack Box Variational Inference.β arXiv:1401.0118 [Cs, Stat], December.
Ranganath, Rajesh, Dustin Tran, Jaan Altosaar, and David Blei. 2016. βOperator Variational Inference.β In Advances in Neural Information Processing Systems 29, edited by D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, 496β504. Curran Associates, Inc.
Rezende, Danilo Jimenez, Shakir Mohamed, and Daan Wierstra. 2015. βStochastic Backpropagation and Approximate Inference in Deep Generative Models.β In Proceedings of ICML.
Salimans, Tim, Diederik Kingma, and Max Welling. 2015. βMarkov Chain Monte Carlo and Variational Inference: Bridging the Gap.β In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), 1218β26. ICMLβ15. Lille, France: JMLR.org.
Titsias, Michalis K., and Miguel LΓ‘zaro-Gredilla. 2014. βDoubly Stochastic Variational Bayes for Non-Conjugate Inference.β In Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32, II-1971βII-1980. ICMLβ14. Beijing, China: JMLR.org.
Wingate, David, and Theophane Weber. 2013. βAutomated Variational Inference in Probabilistic Programming.β arXiv:1301.1299 [Cs, Stat], January.
No comments yet. Why not leave one?