A variational autoencoder uses a limited latent distribution to approximate a complex posterior distribution
A method at the intersection of stochastic variational inference and probabilistic neural nets where we presume that the model is generated by a low-dimensional latent space, which is, if you squint at it, kind of the information bottleneck trick but in a probabilistic setting. To my mind it is a sorta-kinda nonparametric approximate Bayes method.
There is a lot more going on here than I have time to explain, let alone that which I cannot have not even understood for myself.
TBD: connection to reparameterization tricks.
To explore: Relative complexity of these methods e.g. how long does it take to train a variational autoencoder for a given task compared to a similarly expressive GAN?
For now, check out some of the many tutorials, e.g.
- Jaan Altosaarβs VAE tutorial.
- Adam Kosiorek: What is wrong with VAEs?
- Dustin Tran, Denoising Criterion for Variational Auto-Encoding Framework
Incoming
Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images, Child 2020
Jaan Altosaar, Tutorial - What is a variational autoencoder?
This year I realized that VAEs are non-parametrically consistent as models of the observed data even when the encoder is held fixed and arbitrary. This is best demonstrated with a nonstandard derivation of VAEs bypassing the ELBO.
References
Abbasnejad, Ehsan, Anthony Dick, and Anton van den Hengel. 2016. βInfinite Variational Autoencoder for Semi-Supervised Learning.β In Advances in Neural Information Processing Systems 29.
Ambrogioni, Luca, Umut GΓΌΓ§lΓΌ, Yagmur GΓΌΓ§lΓΌtΓΌrk, Max Hinne, Eric Maris, and Marcel A. J. van Gerven. 2018. βWasserstein Variational Inference.β In Proceedings of the 32Nd International Conference on Neural Information Processing Systems, 2478β87. NIPSβ18. USA: Curran Associates Inc.
Arjovsky, Martin, Soumith Chintala, and LΓ©on Bottou. 2017. βWasserstein Generative Adversarial Networks.β In International Conference on Machine Learning, 214β23.
Bamler, Robert, and Stephan Mandt. 2017. βStructured Black Box Variational Inference for Latent Time Series Models.β arXiv:1707.01069 [Cs, Stat], July.
Berg, Rianne van den, Leonard Hasenclever, Jakub M. Tomczak, and Max Welling. 2018. βSylvester Normalizing Flows for Variational Inference.β In UAI18.
Bora, Ashish, Ajil Jalal, Eric Price, and Alexandros G. Dimakis. 2017. βCompressed Sensing Using Generative Models.β In International Conference on Machine Learning, 537β46.
Bowman, Samuel R., Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Jozefowicz, and Samy Bengio. 2015. βGenerating Sentences from a Continuous Space.β arXiv:1511.06349 [Cs], November.
Burda, Yuri, Roger Grosse, and Ruslan Salakhutdinov. 2016. βImportance Weighted Autoencoders.β In arXiv:1509.00519 [Cs, Stat].
Caterini, Anthony L., Arnaud Doucet, and Dino Sejdinovic. 2018. βHamiltonian Variational Auto-Encoder.β In Advances in Neural Information Processing Systems.
Chen, Tian Qi, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. 2018. βNeural Ordinary Differential Equations.β In Advances in Neural Information Processing Systems 31, edited by S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, 6572β83. Curran Associates, Inc.
Chen, Xi, Diederik P. Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. 2016. βVariational Lossy Autoencoder.β In PRoceedings of ICLR.
Chung, Junyoung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Bengio. 2015. βA Recurrent Latent Variable Model for Sequential Data.β In Advances in Neural Information Processing Systems 28, edited by C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, 2980β88. Curran Associates, Inc.
Cremer, Chris, Xuechen Li, and David Duvenaud. 2018. βInference Suboptimality in Variational Autoencoders.β arXiv:1801.03558 [Cs, Stat], January.
Cutajar, Kurt, Edwin V. Bonilla, Pietro Michiardi, and Maurizio Filippone. 2017. βRandom Feature Expansions for Deep Gaussian Processes.β In PMLR.
Dupont, Emilien, Arnaud Doucet, and Yee Whye Teh. 2019. βAugmented Neural ODEs.β arXiv:1904.01681 [Cs, Stat], April.
Fabius, Otto, and Joost R. van Amersfoort. 2014. βVariational Recurrent Auto-Encoders.β In Proceedings of ICLR.
Garnelo, Marta, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J. Rezende, S. M. Ali Eslami, and Yee Whye Teh. 2018. βNeural Processes,β July.
Grathwohl, Will, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. 2018. βFFJORD: Free-Form Continuous Dynamics for Scalable Reversible Generative Models.β arXiv:1810.01367 [Cs, Stat], October.
He, Junxian, Daniel Spokoyny, Graham Neubig, and Taylor Berg-Kirkpatrick. 2019. βLagging Inference Networks and Posterior Collapse in Variational Autoencoders.β In PRoceedings of ICLR.
Hegde, Pashupati, Markus Heinonen, Harri LΓ€hdesmΓ€ki, and Samuel Kaski. 2018. βDeep Learning with Differential Gaussian Process Flows.β arXiv:1810.04066 [Cs, Stat], October.
Hoffman, Matthew D, and Matthew J Johnson. 2016. βELBO Surgery: Yet Another Way to Carve up the Variational Evidence Lower Bound.β In Advances In Neural Information Processing Systems, 4.
Hsu, Wei-Ning, Yu Zhang, and James Glass. 2017. βUnsupervised Learning of Disentangled and Interpretable Representations from Sequential Data.β In arXiv:1709.07902 [Cs, Eess, Stat].
Hu, Zhiting, Zichao Yang, Ruslan Salakhutdinov, and Eric P. Xing. 2018. βOn Unifying Deep Generative Models.β In arXiv:1706.00550 [Cs, Stat].
Huang, Chin-Wei, David Krueger, Alexandre Lacoste, and Aaron Courville. 2018. βNeural Autoregressive Flows.β arXiv:1804.00779 [Cs, Stat], April.
Husain, Hisham, Richard Nock, and Robert C. Williamson. 2019. βA Primal-Dual Link Between GANs and Autoencoders.β In Advances in Neural Information Processing Systems, 32:415β24.
Kim, Yoon, Sam Wiseman, Andrew C. Miller, David Sontag, and Alexander M. Rush. 2018. βSemi-Amortized Variational Autoencoders.β arXiv:1802.02550 [Cs, Stat], February.
Kingma, Diederik P. 2017. βVariational Inference & Deep Learning: A New Synthesis.β
Kingma, Diederik P., Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. 2016. βImproving Variational Inference with Inverse Autoregressive Flow.β In Advances in Neural Information Processing Systems 29. Curran Associates, Inc.
Kingma, Diederik P., Tim Salimans, and Max Welling. 2015. βVariational Dropout and the Local Reparameterization Trick.β In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2, 2575β83. NIPSβ15. Cambridge, MA, USA: MIT Press.
Kingma, Diederik P., and Max Welling. 2014. βAuto-Encoding Variational Bayes.β In ICLR 2014 Conference.
βββ. 2019. An Introduction to Variational Autoencoders. Vol. 12. Foundations and Trends in Machine Learning. Now Publishers, Inc.
Kingma, Durk P, and Prafulla Dhariwal. 2018. βGlow: Generative Flow with Invertible 1x1 Convolutions.β In Advances in Neural Information Processing Systems 31, edited by S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, 10236β45. Curran Associates, Inc.
Knop, Szymon, PrzemysΕaw Spurek, Jacek Tabor, Igor Podolak, Marcin Mazur, and StanisΕaw JastrzΔbski. 2020. βCramer-Wold Auto-Encoder.β Journal of Machine Learning Research 21 (164): 1β28.
Larsen, Anders Boesen Lindbo, SΓΈren Kaae SΓΈnderby, Hugo Larochelle, and Ole Winther. 2015. βAutoencoding Beyond Pixels Using a Learned Similarity Metric.β arXiv:1512.09300 [Cs, Stat], December.
Lee, Holden, Rong Ge, Tengyu Ma, Andrej Risteski, and Sanjeev Arora. 2017. βOn the Ability of Neural Nets to Express Distributions.β In arXiv:1702.07028 [Cs].
Liang, Dawen, Rahul G. Krishnan, Matthew D. Hoffman, and Tony Jebara. 2018. βVariational Autoencoders for Collaborative Filtering.β In Proceedings of the 2018 World Wide Web Conference, 689β98. WWW β18. Republic and Canton of Geneva, CHE: International World Wide Web Conferences Steering Committee.
Louizos, Christos, Uri Shalit, Joris M Mooij, David Sontag, Richard Zemel, and Max Welling. 2017. βCausal Effect Inference with Deep Latent-Variable Models.β In Advances in Neural Information Processing Systems 30, edited by I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, 6446β56. Curran Associates, Inc.
Louizos, Christos, and Max Welling. 2017. βMultiplicative Normalizing Flows for Variational Bayesian Neural Networks.β In PMLR, 2218β27.
Luo, Yin-Jyun, Kat Agres, and Dorien Herremans. 2019. βLearning Disentangled Representations of Timbre and Pitch for Musical Instrument Sounds Using Gaussian Mixture Variational Autoencoders.β In Proceedings of the 20th Conference of the International Society for Music Information Retrieval.
Mathieu, Emile, Tom Rainforth, N. Siddharth, and Yee Whye Teh. 2019. βDisentangling Disentanglement in Variational Autoencoders.β In International Conference on Machine Learning, 4402β12. PMLR.
Meent, Jan-Willem van de, Brooks Paige, Hongseok Yang, and Frank Wood. 2021. βAn Introduction to Probabilistic Programming.β arXiv:1809.10756 [Cs, Stat], October.
Ng, Ignavier, Zhuangyan Fang, Shengyu Zhu, Zhitang Chen, and Jun Wang. 2020. βMasked Gradient-Based Causal Structure Learning.β arXiv:1910.08527 [Cs, Stat], February.
Ng, Ignavier, Shengyu Zhu, Zhitang Chen, and Zhuangyan Fang. 2019. βA Graph Autoencoder Approach to Causal Structure Learning.β In Advances In Neural Information Processing Systems.
Papamakarios, George, Iain Murray, and Theo Pavlakou. 2017. βMasked Autoregressive Flow for Density Estimation.β In Advances in Neural Information Processing Systems 30, edited by I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, 2338β47. Curran Associates, Inc.
Rakesh, Vineeth, Ruocheng Guo, Raha Moraffah, Nitin Agarwal, and Huan Liu. 2018. βLinked Causal Variational Autoencoder for Inferring Paired Spillover Effects.β In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, 1679β82. CIKM β18. New York, NY, USA: Association for Computing Machinery.
Rezende, Danilo Jimenez, and Shakir Mohamed. 2015. βVariational Inference with Normalizing Flows.β In International Conference on Machine Learning, 1530β38. ICMLβ15. Lille, France: JMLR.org.
Rezende, Danilo Jimenez, Shakir Mohamed, and Daan Wierstra. 2015. βStochastic Backpropagation and Approximate Inference in Deep Generative Models.β In Proceedings of ICML.
Richter, Lorenz, Ayman Boustati, Nikolas NΓΌsken, Francisco J. R. Ruiz, and Γmer Deniz Akyildiz. 2020. βVarGrad: A Low-Variance Gradient Estimator for Variational Inference.β arXiv.
Rippel, Oren, and Ryan Prescott Adams. 2013. βHigh-Dimensional Probability Estimation with Deep Density Models.β arXiv:1302.5125 [Cs, Stat], February.
Roberts, Adam, Jesse Engel, Colin Raffel, Curtis Hawthorne, and Douglas Eck. 2018. βA Hierarchical Latent Vector Model for Learning Long-Term Structure in Music.β arXiv:1803.05428 [Cs, Eess, Stat], March.
Roeder, Geoffrey, Paul K. Grant, Andrew Phillips, Neil Dalchau, and Edward Meeds. 2019. βEfficient Amortised Bayesian Inference for Hierarchical and Nonlinear Dynamical Systems.β arXiv:1905.12090 [Cs, Stat], May.
Ruiz, Francisco J. R., Michalis K. Titsias, and David M. Blei. 2016. βThe Generalized Reparameterization Gradient.β In Advances In Neural Information Processing Systems.
Salimans, Tim, Diederik Kingma, and Max Welling. 2015. βMarkov Chain Monte Carlo and Variational Inference: Bridging the Gap.β In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), 1218β26. ICMLβ15. Lille, France: JMLR.org.
Spantini, Alessio, Daniele Bigoni, and Youssef Marzouk. 2017. βInference via Low-Dimensional Couplings.β Journal of Machine Learning Research 19 (66): 2639β709.
Tait, Daniel J., and Theodoros Damoulas. 2020. βVariational Autoencoding of PDE Inverse Problems.β arXiv:2006.15641 [Cs, Stat], June.
Tran, Dustin, Rajesh Ranganath, and David M. Blei. 2015. βThe Variational Gaussian Process.β In Proceedings of ICLR.
Ullrich, K. 2020. βA Coding Perspective on Deep Latent Variable Models.β
Wang, Prince Zizhuang, and William Yang Wang. 2019. βRiemannian Normalizing Flow on Variational Wasserstein Autoencoder for Text Modeling.β In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 284β94. Minneapolis, Minnesota: Association for Computational Linguistics.
Yang, Mengyue, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. 2020. βCausalVAE: Disentangled Representation Learning via Neural Structural Causal Models.β arXiv:2004.08697 [Cs, Stat], July.
Zahm, Olivier, Paul Constantine, ClΓ©mentine Prieur, and Youssef Marzouk. 2018. βGradient-Based Dimension Reduction of Multivariate Vector-Valued Functions.β arXiv:1801.07922 [Math], January.
No comments yet. Why not leave one?