Deep generative models

December 10, 2020 — November 11, 2021

approximation
Bayes
generative
likelihood free
Monte Carlo
neural nets
optimization
probabilistic algorithms
probability
statistics
unsupervised

Certain famous models in neural nets are generative — informally, they produce samples some distribution, in training the distribution of those samples is tweaked until its distribution resembles, in some sense, the distribution of our observed data. There are many attempts now to unify fancy generative techniques such as GANs and VAEs and neural diffusiong into a single unified method, or at least a cordial family of methods, so I had better devise a page for that.

Here I mean generative in the sense that “this model will (approximately) simulate from the true distribution of interest”, which is somewhat weaker that the requirements of, e.g., MC Bayesian inference, where we assume that we can access likelihoods, or at least likelihood gradients. In such a case, we might have no likelihood at all, or variational approximations to likelihood or whatever.

1 Philosophical diversion: probability is a weird abstraction

Tangent: Learning problems involve composition of differentiating and integrating various terms that measure various properties of how well you have approximated the state of the world. Probabilistic neural networks leverage combinations of integrals that we can solve by Monte Carlo, and derivatives that we can solve via automatic differentiation, which are both fast-ish on modern hardware In cunning combination these find approximate solutions to some very interesting problems in calculus. Although… There is something odd about that setup. From this perspective the generative models (such as GANs and autoencoders) solve an intractable integral by simulating samples probabilistically from it, in lieu of processing the continuous, unknowable, intractable integral that we actually wish to solve. But that continuous intractable integral was in any case a contrivance, a thought experiment imagining a world populated with such weird Platonic objects as integrals-over-possible-states-of-the-world which only mathematicians would consider reasonable. The world we live in has, as far as I know, no such thing. We do not have a world where the things we observe are stochastic samples from an ineffable probability density, but rather the observations themselves are the phenomena, and the probability density over them is a weird abstraction. It must look deeply odd from the outside when we to talk about how we are solving integrals by looking at data, instead of solving data by looking at integrals.

3 References

Anderson. 1982. Stochastic Processes and Their Applications.
Arjovsky, and Bottou. 2017. arXiv:1701.04862 [Stat].
Arjovsky, Chintala, and Bottou. 2017. In International Conference on Machine Learning.
Arora, Ge, Liang, et al. 2017. arXiv:1703.00573 [Cs].
Arora, Liang, and Ma. 2015. arXiv:1511.05653 [Cs].
Bach, He, Ratner, et al. 2017. In Proceedings of the 34th International Conference on Machine Learning.
Bahadori, Chalupka, Choi, et al. 2017. arXiv:1702.02604 [Cs, Stat].
Baydin, Shao, Bhimji, et al. 2019. In arXiv:1907.03382 [Cs, Stat].
Bora, Jalal, Price, et al. 2017. In International Conference on Machine Learning.
Bowman, Vilnis, Vinyals, et al. 2015. arXiv:1511.06349 [Cs].
Burda, Grosse, and Salakhutdinov. 2016. In arXiv:1509.00519 [Cs, Stat].
Caterini, Doucet, and Sejdinovic. 2018. In Advances in Neural Information Processing Systems.
Chen, Duan, Houthooft, et al. 2016. In Advances in Neural Information Processing Systems 29.
Chen, Kingma, Salimans, et al. 2016. In PRoceedings of ICLR.
Dasgupta, Yoshizumi, and Osogami. 2016. arXiv:1610.01989 [Cs, Stat].
Denton, Chintala, Szlam, et al. 2015. arXiv:1506.05751 [Cs].
Dhariwal, and Nichol. 2021. arXiv:2105.05233 [Cs, Stat].
Donahue, McAuley, and Puckette. 2019. In ICLR 2019.
Dosovitskiy, Springenberg, Tatarchenko, et al. 2014. arXiv:1411.5928 [Cs].
Dutordoir, Hensman, van der Wilk, et al. 2021. In arXiv:2105.04504 [Cs, Stat].
Dutordoir, Saul, Ghahramani, et al. 2022.
Dziugaite, Roy, and Ghahramani. 2015. In Proceedings of the Thirty-First Conference on Uncertainty in Artificial Intelligence. UAI’15.
Engel, Agrawal, Chen, et al. 2019. In Seventh International Conference on Learning Representations.
Engel, Hantrakul, Gu, et al. 2019. In.
Engel, Resnick, Roberts, et al. 2017. In PMLR.
Frühstück, Alhashim, and Wonka. 2019. arXiv:1904.12795 [Cs].
Gal, and Ghahramani. 2015. “On Modern Deep Learning and Variational Inference.” In Advances in Approximate Bayesian Inference Workshop, NIPS.
Genevay, Peyré, and Cuturi. 2017. arXiv:1706.00292 [Stat].
Goodfellow, Ian, Pouget-Abadie, Mirza, et al. 2014. In Advances in Neural Information Processing Systems 27. NIPS’14.
Goodfellow, Ian J., Shlens, and Szegedy. 2014. arXiv:1412.6572 [Cs, Stat].
Gulrajani, Ahmed, Arjovsky, et al. 2017. arXiv:1704.00028 [Cs, Stat].
Guo, Hong, Lin, et al. 2017. arXiv:1705.07164 [Cs, Stat].
Han, Zheng, and Zhou. 2022.
He, Wang, and Hopcroft. 2016. In Advances in Neural Information Processing Systems.
Hinton. 2007. Trends in Cognitive Sciences.
Hoffman, and Johnson. 2016. In Advances In Neural Information Processing Systems.
Ho, Jain, and Abbeel. 2020. arXiv:2006.11239 [Cs, Stat].
Hoogeboom, Gritsenko, Bastings, et al. 2021. arXiv:2110.02037 [Cs, Stat].
Husain. 2020. arXiv:2006.04349 [Cs, Stat].
Husain, Nock, and Williamson. 2019. In Advances in Neural Information Processing Systems.
Huszár. 2015. arXiv:1511.05101 [Cs, Math, Stat].
———. 2017.
Hu, Yang, Salakhutdinov, et al. 2018. In arXiv:1706.00550 [Cs, Stat].
Hyvärinen. 2005. The Journal of Machine Learning Research.
Isola, Zhu, Zhou, et al. 2017. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Jalal, Arvinte, Daras, et al. 2021. In Advances in Neural Information Processing Systems.
Jayaram, and Thickstun. 2020. arXiv:2002.07942 [Cs, Stat].
Jetchev, Bergmann, and Vollgraf. 2016. In Advances in Neural Information Processing Systems 29.
Ji, and Liang. 2018.
Jolicoeur-Martineau, Piché-Taillefer, Mitliagkas, et al. 2022. In.
Karaletsos. 2016.
Karras, Laine, and Aila. 2018. arXiv:1812.04948 [Cs, Stat].
Kim, Wiseman, Miller, et al. 2018. arXiv:1802.02550 [Cs, Stat].
Kingma, and Dhariwal. 2018. In Advances in Neural Information Processing Systems 31.
Kodali, Abernethy, Hays, et al. 2017. arXiv:1705.07215 [Cs].
Krishnan, Shalit, and Sontag. 2017. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence.
Kulkarni, Whitney, Kohli, et al. 2015. arXiv:1503.03167 [Cs].
Lee, Holden, Ge, Ma, et al. 2017. In arXiv:1702.07028 [Cs].
Lee, Honglak, Grosse, Ranganath, et al. 2009. In Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09.
Liang, Krishnan, Hoffman, et al. 2018. In Proceedings of the 2018 World Wide Web Conference. WWW ’18.
Li, Chang, Cheng, et al. 2017. In Advances in Neural Information Processing Systems 30.
Liu, Luo, Xu, et al. 2023.
Louizos, and Welling. 2016. In arXiv Preprint arXiv:1603.04733.
Mirza, and Osindero. 2014. arXiv:1411.1784 [Cs, Stat].
Miyato, Kataoka, Koyama, et al. 2018. In ICLR 2018.
Mnih, and Gregor. 2014. In Proceedings of The 31st International Conference on Machine Learning. ICML’14.
Mohamed, A. r, Dahl, and Hinton. 2012. IEEE Transactions on Audio, Speech, and Language Processing.
Mohamed, Shakir, and Lakshminarayanan. 2016.
Mohamed, Shakir, and Rezende. 2015. “Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning.” In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2. NIPS’15.
Nichol, and Dhariwal. 2021. In Proceedings of the 38th International Conference on Machine Learning.
Panaretos, and Zemel. 2019. Annual Review of Statistics and Its Application.
Papamakarios, Nalisnick, Rezende, et al. 2021. Journal of Machine Learning Research.
Pascual, Serrà, and Bonafonte. 2019. arXiv:1904.03418 [Cs, Eess].
Pfau, and Vinyals. 2016. arXiv:1610.01945 [Cs, Stat].
Poole, Alemi, Sohl-Dickstein, et al. 2016. In Advances in Neural Information Processing Systems 29.
Prenger, Valle, and Catanzaro. 2018. arXiv:1811.00002 [Cs, Eess, Stat].
Radford, Metz, and Chintala. 2015. In arXiv:1511.06434 [Cs].
Ramasinghe, Ranasinghe, Khan, et al. 2020. In.
Ranganath, Tran, Altosaar, et al. 2016. In Advances in Neural Information Processing Systems 29.
Rezende, Mohamed, and Wierstra. 2015. In Proceedings of ICML.
Salakhutdinov. 2015. Annual Review of Statistics and Its Application.
Salimans, Kingma, and Welling. 2015. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15). ICML’15.
Sohl-Dickstein, Weiss, Maheswaranathan, et al. 2015. arXiv:1503.03585 [Cond-Mat, q-Bio, Stat].
Song, Yang, Durkan, Murray, et al. 2021. In Advances in Neural Information Processing Systems.
Song, Yang, and Ermon. 2020a. In Advances In Neural Information Processing Systems.
———. 2020b. In Advances In Neural Information Processing Systems.
Song, Yang, Garg, Shi, et al. 2019.
Song, Jiaming, Meng, and Ermon. 2021. arXiv:2010.02502 [Cs].
Song, Yang, Shen, Xing, et al. 2022. In.
Song, Yang, Sohl-Dickstein, Kingma, et al. 2022. In.
Sun, Liu, Zhang, et al. 2016. arXiv:1611.05416 [Cs].
Sutherland, Tung, Strathmann, et al. 2017. In Proceedings of ICLR.
Swersky, Ranzato, Buchman, et al. 2011. “On Autoencoders and Score Matching for Energy Based Models.” In Proceedings of the 28th International Conference on Machine Learning (ICML-11).
Theis, and Bethge. 2015. arXiv:1506.03478 [Cs, Stat].
Tran, Hoffman, Saurous, et al. 2017. In ICLR.
Tran, Ranganath, and Blei. 2017. In Advances in Neural Information Processing Systems 30.
Ullrich. 2020.
Uppal, Stensbo-Smidt, Boomsma, et al. 2023.
Oord, Aaron van den, Dieleman, Zen, et al. 2016. In 9th ISCA Speech Synthesis Workshop.
Oord, Aäron van den, Kalchbrenner, and Kavukcuoglu. 2016. arXiv:1601.06759 [Cs].
Vincent. 2011. Neural Computation.
Wang, Chuang, Hu, and Lu. 2019. arXiv:1805.08349 [Cond-Mat, Stat].
Wang, Prince Zizhuang, and Wang. 2019. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).
Winkler, Worrall, Hoogeboom, et al. 2023.
Wu, Rosca, and Lillicrap. 2019. In International Conference on Machine Learning.
Xie, Gao, Nijkamp, et al. 2020. Annual Review of Statistics and Its Application.
Yang, Li-Chia, Chou, and Yang. 2017. In Proceedings of the 18th International Society for Music Information Retrieval Conference (ISMIR’2017), Suzhou, China.
Yang, Mengyue, Liu, Chen, et al. 2020. arXiv:2004.08697 [Cs, Stat].
Yang, Liu, Zhang, and Karniadakis. 2020. SIAM Journal on Scientific Computing.
Yang, Ling, Zhang, Song, et al. 2023. ACM Computing Surveys.
Yıldız, Heinonen, and Lähdesmäki. 2019. arXiv:1905.10994 [Cs, Stat].
Zhou, Horgan, Kumar, et al. 2018. arXiv:1808.08311 [Cs, Eess].
Zhu, B., Jiao, and Tse. 2020. IEEE Transactions on Information Theory.
Zhu, Jun-Yan, Krähenbühl, Shechtman, et al. 2016. In Proceedings of European Conference on Computer Vision.