Variational inference

On fitting something not too far from a pretty good model that is not too hard

March 22, 2016 — February 10, 2022

approximation
metrics
optimization
probabilistic algorithms
probability
statistics
Figure 1:

SGD and stochastic gradient (and usually also message passing) meet variational inference.

If we add in amortization as well, we then have a variational autoencoder.

1 Introductory reading

Wingate and Weber (2013), Ranganath, Gerrish, and Blei (2013), Li and Turner (2016).

2 References

Blei, Kucukelbir, and McAuliffe. 2017. “Variational Inference: A Review for Statisticians.” Journal of the American Statistical Association.
Detommaso, Cui, Spantini, et al. 2018. “A Stein Variational Newton Method.” In Proceedings of the 32nd International Conference on Neural Information Processing Systems. NIPS’18.
Dhaka, and Catalina. 2020. “Robust, Accurate Stochastic Optimization for Variational Inference.”
Dhaka, Catalina, Welandawe, et al. 2021. “Challenges and Opportunities in High-Dimensional Variational Inference.” arXiv:2103.01085 [Cs, Stat].
Fraccaro, Sø nderby, Paquet, et al. 2016. “Sequential Neural Models with Stochastic Layers.” In Advances in Neural Information Processing Systems 29.
Graves. 2011. “Practical Variational Inference for Neural Networks.” In Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11.
Hoffman, Matthew, and Blei. 2015. “Stochastic Structured Variational Inference.” In PMLR.
Hoffman, Matt, Blei, Wang, et al. 2013. “Stochastic Variational Inference.” arXiv:1206.7051 [Cs, Stat].
Jordan, Ghahramani, Jaakkola, et al. 1999. “An Introduction to Variational Methods for Graphical Models.” Machine Learning.
Kingma. 2017. “Variational Inference & Deep Learning: A New Synthesis.”
Li, and Turner. 2016. “Rényi Divergence Variational Inference.” In Advances in Neural Information Processing Systems.
Liu, and Wang. 2019. “Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm.” In Advances In Neural Information Processing Systems.
Matthews. 2017. “Scalable Gaussian Process Inference Using Variational Methods.”
Plötz, Wannenwetsch, and Roth. 2018. “Stochastic Variational Inference with Gradient Linearization.” In CVPR.
Ranganath, Gerrish, and Blei. 2013. “Black Box Variational Inference.” arXiv:1401.0118 [Cs, Stat].
Ranganath, Tran, Altosaar, et al. 2016. “Operator Variational Inference.” In Advances in Neural Information Processing Systems 29.
Rezende, Mohamed, and Wierstra. 2015. “Stochastic Backpropagation and Approximate Inference in Deep Generative Models.” In Proceedings of ICML.
Salimans, Kingma, and Welling. 2015. “Markov Chain Monte Carlo and Variational Inference: Bridging the Gap.” In Proceedings of the 32nd International Conference on Machine Learning (ICML-15). ICML’15.
Titsias, and Lázaro-Gredilla. 2014. “Doubly Stochastic Variational Bayes for Non-Conjugate Inference.” In Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32. ICML’14.
Wingate, and Weber. 2013. “Automated Variational Inference in Probabilistic Programming.” arXiv:1301.1299 [Cs, Stat].
Yin, and Zhou. 2018. “Semi-Implicit Variational Inference.” In Proceedings of the 35th International Conference on Machine Learning.