Deep generative models

Philosophical diversion: probability is a weird abstraction

Tangent: Learning problems involve composition of differentiating and integrating various terms that measure various properties of how well you have approximated the state of the world. Probabilistic neural networks leverage combinations of integrals that we can solve by Monte Carlo, and derivatives that we can solve via automatic differentiation, which are both fast-ish on modern hardware In cunning combination these find approximate solutions to some very interesting problems in calculus. Although… There is something odd about that setup. From this perspective the generative models (such as GANs and autoencoders) solve an intractable integral by simulating samples probabilistically from it, in lieu of processing the continuous, unknowable, intractable integral that we actually wish to solve. But that continuous intractable integral was in any case a contrivance, a thought experiment imagining a world populated with such weird Platonic objects as integrals-over-possible-states-of-the-world which only mathematicians would consider reasonable. The world we live in has, as far as I know, no such thing. We do not have a world where the things we observe are stochastic samples from an ineffable probability density, but rather the observations themselves are the phenomena, and the probability density over them is a weird abstraction. It must look deeply odd from the outside when we to talk about how we are solving integrals by looking at data, instead of solving data by looking at integrals.

References