# Deep generative models

December 10, 2020 — November 11, 2021

Certain famous models in neural nets are generative — informally, they produce samples some distribution, in training the distribution of those samples is tweaked until its distribution resembles, in some sense, the distribution of our observed data. There are many attempts now to unify fancy generative techniques such as GANs and VAEs and neural diffusiong into a single unified method, or at least a cordial family of methods, so I had better devise a page for that.

Here I mean generative in the sense that “this model will (approximately) simulate from the true distribution of interest”, which is somewhat weaker that the requirements of, e.g., MC Bayesian inference, where we assume that we can access likelihoods, or at least likelihood gradients. In such a case, we might have no likelihood at all, or variational approximations to likelihood or whatever.

## 1 Philosophical diversion: probability is a weird abstraction

Tangent: Learning problems involve composition of differentiating and integrating various terms that measure various properties of how well you have approximated the state of the world. Probabilistic neural networks leverage combinations of integrals that we can solve by Monte Carlo, and derivatives that we can solve via automatic differentiation, which are both fast-ish on modern hardware In cunning combination these find approximate solutions to some very interesting problems in calculus. Although… There *is* something odd about that setup. From this perspective the generative models (such as GANs and autoencoders) solve an intractable integral by simulating samples probabilistically from it, in lieu of processing the continuous, unknowable, intractable integral that we actually wish to solve. But that continuous intractable integral was in any case a contrivance, a thought experiment imagining a world populated with such weird Platonic objects as integrals-over-possible-states-of-the-world which only mathematicians would consider reasonable. The world we live in has, as far as I know, no such thing. We do not have a world where the things we observe are stochastic samples from an ineffable probability density, but rather the observations themselves are the phenomena, and the probability density over them is a weird abstraction. It must look deeply odd from the outside when we to talk about how we are solving integrals by looking at data, instead of solving data by looking at integrals.

## 2 Generative flow nets

See this page.

## 3 References

*Stochastic Processes and Their Applications*.

*arXiv:1701.04862 [Stat]*.

*International Conference on Machine Learning*.

*arXiv:1703.00573 [Cs]*.

*arXiv:1511.05653 [Cs]*.

*Proceedings of the 34th International Conference on Machine Learning*.

*arXiv:1702.02604 [Cs, Stat]*.

*arXiv:1907.03382 [Cs, Stat]*.

*International Conference on Machine Learning*.

*arXiv:1511.06349 [Cs]*.

*arXiv:1509.00519 [Cs, Stat]*.

*Advances in Neural Information Processing Systems*.

*Advances in Neural Information Processing Systems 29*.

*PRoceedings of ICLR*.

*arXiv:1610.01989 [Cs, Stat]*.

*arXiv:1506.05751 [Cs]*.

*arXiv:2105.05233 [Cs, Stat]*.

*ICLR 2019*.

*arXiv:1411.5928 [Cs]*.

*arXiv:2105.04504 [Cs, Stat]*.

*Proceedings of the Thirty-First Conference on Uncertainty in Artificial Intelligence*. UAI’15.

*Seventh International Conference on Learning Representations*.

*PMLR*.

*arXiv:1904.12795 [Cs]*.

*Advances in Approximate Bayesian Inference Workshop, NIPS*.

*arXiv:1706.00292 [Stat]*.

*Advances in Neural Information Processing Systems 27*. NIPS’14.

*arXiv:1412.6572 [Cs, Stat]*.

*arXiv:1704.00028 [Cs, Stat]*.

*arXiv:1705.07164 [Cs, Stat]*.

*Advances in Neural Information Processing Systems*.

*Trends in Cognitive Sciences*.

*Advances In Neural Information Processing Systems*.

*arXiv:2006.11239 [Cs, Stat]*.

*arXiv:2110.02037 [Cs, Stat]*.

*arXiv:2006.04349 [Cs, Stat]*.

*Advances in Neural Information Processing Systems*.

*arXiv:1511.05101 [Cs, Math, Stat]*.

*arXiv:1706.00550 [Cs, Stat]*.

*The Journal of Machine Learning Research*.

*2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*.

*Advances in Neural Information Processing Systems*.

*arXiv:2002.07942 [Cs, Stat]*.

*Advances in Neural Information Processing Systems 29*.

*arXiv:1812.04948 [Cs, Stat]*.

*arXiv:1802.02550 [Cs, Stat]*.

*Advances in Neural Information Processing Systems 31*.

*arXiv:1705.07215 [Cs]*.

*Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence*.

*arXiv:1503.03167 [Cs]*.

*arXiv:1702.07028 [Cs]*.

*Proceedings of the 26th Annual International Conference on Machine Learning*. ICML ’09.

*Proceedings of the 2018 World Wide Web Conference*. WWW ’18.

*Advances in Neural Information Processing Systems 30*.

*arXiv Preprint arXiv:1603.04733*.

*arXiv:1411.1784 [Cs, Stat]*.

*ICLR 2018*.

*Proceedings of The 31st International Conference on Machine Learning*.

*IEEE Transactions on Audio, Speech, and Language Processing*.

*arXiv:2102.09672 [Cs, Stat]*.

*Annual Review of Statistics and Its Application*.

*Journal of Machine Learning Research*.

*arXiv:1904.03418 [Cs, Eess]*.

*arXiv:1610.01945 [Cs, Stat]*.

*Advances in Neural Information Processing Systems 29*.

*arXiv:1811.00002 [Cs, Eess, Stat]*.

*arXiv:1511.06434 [Cs]*.

*Advances in Neural Information Processing Systems 29*.

*Proceedings of ICML*.

*Annual Review of Statistics and Its Application*.

*Proceedings of the 32nd International Conference on Machine Learning (ICML-15)*. ICML’15.

*arXiv:1503.03585 [Cond-Mat, q-Bio, Stat]*.

*Advances in Neural Information Processing Systems*.

*Advances In Neural Information Processing Systems*.

*Advances In Neural Information Processing Systems*.

*arXiv:2010.02502 [Cs]*.

*arXiv:1611.05416 [Cs]*.

*Proceedings of ICLR*.

*Proceedings of the 28th International Conference on Machine Learning (ICML-11)*.

*arXiv:1506.03478 [Cs, Stat]*.

*ICLR*.

*9th ISCA Speech Synthesis Workshop*.

*arXiv:1601.06759 [Cs]*.

*Neural Computation*.

*arXiv:1805.08349 [Cond-Mat, Stat]*.

*Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*.

*International Conference on Machine Learning*.

*Annual Review of Statistics and Its Application*.

*Proceedings of the 18th International Society for Music Information Retrieval Conference (ISMIR’2017), Suzhou, China*.

*arXiv:2004.08697 [Cs, Stat]*.

*SIAM Journal on Scientific Computing*.

*arXiv:1905.10994 [Cs, Stat]*.

*arXiv:1808.08311 [Cs, Eess]*.

*IEEE Transactions on Information Theory*.

*Proceedings of European Conference on Computer Vision*.