Variational inference

On fitting something not too far from a pretty good model that is not too hard

SGD and stochastic gradient (and usually also message passing) meets variational inference.

If we add in amortization as well, we then have a variational autoencoder.

Introductory reading

Wingate and Weber (2013), Ranganath, Gerrish, and Blei (2013), Li and Turner (2016).


Blei, David M., Alp Kucukelbir, and Jon D. McAuliffe. 2017. β€œVariational Inference: A Review for Statisticians.” Journal of the American Statistical Association 112 (518): 859–77.
Dhaka, Akash Kumar, and Alejandro Catalina. 2020. β€œRobust, Accurate Stochastic Optimization for Variational Inference,” 13.
Dhaka, Akash Kumar, Alejandro Catalina, Manushi Welandawe, Michael Riis Andersen, Jonathan Huggins, and Aki Vehtari. 2021. β€œChallenges and Opportunities in High-Dimensional Variational Inference.” arXiv:2103.01085 [Cs, Stat], March.
Fraccaro, Marco, SΓΈ ren Kaae SΓΈ nderby, Ulrich Paquet, and Ole Winther. 2016. β€œSequential Neural Models with Stochastic Layers.” In Advances in Neural Information Processing Systems 29, edited by D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, 2199–2207. Curran Associates, Inc.
Graves, Alex. 2011. β€œPractical Variational Inference for Neural Networks.” In Proceedings of the 24th International Conference on Neural Information Processing Systems, 2348–56. NIPS’11. USA: Curran Associates Inc.
Hoffman, Matt, David M. Blei, Chong Wang, and John Paisley. 2013. β€œStochastic Variational Inference.” arXiv:1206.7051 [Cs, Stat] 14 (1).
Hoffman, Matthew, and David Blei. 2015. β€œStochastic Structured Variational Inference.” In PMLR, 361–69.
Jordan, Michael I., Zoubin Ghahramani, Tommi S. Jaakkola, and Lawrence K. Saul. 1999. β€œAn Introduction to Variational Methods for Graphical Models.” Machine Learning 37 (2): 183–233.
Kingma, Diederik P. 2017. β€œVariational Inference & Deep Learning: A New Synthesis.”
Li, Yingzhen, and Richard E Turner. 2016. β€œRΓ©nyi Divergence Variational Inference.” In Advances in Neural Information Processing Systems, 29:1081–89. Red Hook, NY, USA: Curran Associates, Inc.
Matthews, Alexander Graeme de Garis. 2017. β€œScalable Gaussian Process Inference Using Variational Methods.” Thesis, University of Cambridge.
PlΓΆtz, Tobias, Anne S. Wannenwetsch, and Stefan Roth. 2018. β€œStochastic Variational Inference with Gradient Linearization.” In CVPR.
Ranganath, Rajesh, Sean Gerrish, and David M. Blei. 2013. β€œBlack Box Variational Inference.” arXiv:1401.0118 [Cs, Stat], December.
Ranganath, Rajesh, Dustin Tran, Jaan Altosaar, and David Blei. 2016. β€œOperator Variational Inference.” In Advances in Neural Information Processing Systems 29, edited by D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, 496–504. Curran Associates, Inc.
Rezende, Danilo Jimenez, Shakir Mohamed, and Daan Wierstra. 2015. β€œStochastic Backpropagation and Approximate Inference in Deep Generative Models.” In Proceedings of ICML.
Salimans, Tim, Diederik Kingma, and Max Welling. 2015. β€œMarkov Chain Monte Carlo and Variational Inference: Bridging the Gap.” In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), 1218–26. ICML’15. Lille, France:
Titsias, Michalis K., and Miguel LΓ‘zaro-Gredilla. 2014. β€œDoubly Stochastic Variational Bayes for Non-Conjugate Inference.” In Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32, II-1971–II-1980. ICML’14. Beijing, China:
Wingate, David, and Theophane Weber. 2013. β€œAutomated Variational Inference in Probabilistic Programming.” arXiv:1301.1299 [Cs, Stat], January.

No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.