# Probabilistic neural nets

Inferring distributions in neural nets

January 12, 2017 — April 7, 2022

Inferring densities and distributions in a massively parameterised deep learning setting.

This is not intrinsically a Bayesian thing to do but in practice much of the demand to do probabilistic nets comes from the demand for Bayesian posterior inference for neural nets. Bayesian inference is, however, not the only way to do uncertainty quantification.

Neural networks are very far from simple exponential families where conjugate distributions might help, and so typically rely upon approximations or luck to approximate our true target of interest.

Closely related: Generative models where we train a process to generate a (possibly stochastic) phenomenon of interest.

## 1 Backgrounders

Jospin et al. (2022) is a modern high-speed intro and summary of many approaches.

There was a tutorial in 2020: by Dustin Tran, Jasper Snoek, Balaji Lakshminarayanan: Practical Uncertainty Estimation & Out-of-Distribution Robustness in Deep Learning.

## 2 Mixture density networks

Nothing to say for now but here are some recommendations I received about this classic (C. Bishop 1994) method.

## 3 Ensemble methods

Deep learning has its own variants model averaging and bagging: Neural ensembles. Yarin Gal’s PhD Thesis (Gal 2016) summarizes some implicit approximate approaches (e.g. the Bayesian interpretation of dropout) although dropout as he frames it has become highly controversial these days as a means of inference.

## 4 Practicalities

The computational toolsets for “neural” probabilistic programming and vanilla probabilistic programming are converging. See the tool listing under probabilistic programming.

## 5 Incoming

## 6 References

*Advances in Neural Information Processing Systems 29*.

*arXiv:2005.12998 [Math]*.

*SIAM Journal on Scientific Computing*.

*Proceedings of the 39th International Conference on Machine Learning*.

*arXiv:2110.11216 [Cs, Math, Stat]*.

*arXiv:1511.07367 [Stat]*.

*Inverse Problems*.

*arXiv:1907.03382 [Cs, Stat]*.

*Microsoft Research*.

*Pattern Recognition and Machine Learning*. Information Science and Statistics.

*Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37*. ICML’15.

*International Conference on Machine Learning*.

*Journal of the American Statistical Association*.

*arXiv:1703.04818 [Cs]*.

*Mathematics of Computation*.

*arXiv:2105.04471 [Cs, Stat]*.

*Proceedings of the 35th International Conference on Machine Learning*.

*Advances in Neural Information Processing Systems 31*.

*PMLR*.

*Artificial Intelligence and Statistics*.

*arXiv:2012.07244 [Cs]*.

*arXiv:2106.14806 [Cs, Stat]*.

*Computer Physics Communications*.

*Advances in Neural Information Processing Systems 28*. NIPS’15.

*arXiv:1801.10395 [Stat]*.

*arXiv:2012.00152 [Cs, Stat]*.

*Journal of Machine Learning Research*.

*arXiv:1904.01681 [Cs, Stat]*.

*2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*.

*arXiv:2105.04504 [Cs, Stat]*.

*arXiv:1703.11008 [Cs]*.

*Advances in Neural Information Processing Systems 30*.

*Proceedings of ICLR*.

*Advances in Neural Information Processing Systems 31*.

*arXiv:1704.04110 [Cs, Stat]*.

*arXiv:1906.11537 [Cs, Stat]*.

*Advances in Approximate Bayesian Inference Workshop, NIPS*.

*Advances in Approximate Bayesian Inference Workshop, NIPS*.

*Proceedings of the 33rd International Conference on Machine Learning (ICML-16)*.

*arXiv:1512.05287 [Stat]*.

*4th International Conference on Learning Representations (ICLR) Workshop Track*.

*arXiv:1506.02157 [Stat]*.

*arXiv:1705.07832 [Stat]*.

*arXiv:1807.01613 [Cs, Stat]*.

*arXiv:1902.10298 [Cs]*.

*IEEE Transactions on Signal Processing*.

*Proceedings of the 24th International Conference on Neural Information Processing Systems*. NIPS’11.

*arXiv:1308.0850 [Cs]*.

*2013 IEEE International Conference on Acoustics, Speech and Signal Processing*.

*arXiv:1502.04623 [Cs]*.

*Advances in Neural Information Processing Systems 28*.

*Proceedings of ICLR*.

*arXiv:1805.08034 [Cs, Math]*.

*Advances in Neural Information Processing Systems*.

*PMLR*.

*arXiv:1809.09505 [Cs, Math, Stat]*.

*arXiv:1706.00550 [Cs, Stat]*.

*Proceedings of the 38th International Conference on Machine Learning*.

*International Conference on Artificial Intelligence and Statistics*.

*Spatial Statistics*, Spatial Statistics Miami,.

*Proceedings of The 35th Uncertainty in Artificial Intelligence Conference*.

*arXiv:2007.06823 [Cs, Stat]*.

*Proceedings of ICLR*.

*arXiv:1906.01930 [Cs, Stat]*.

*Advances in Neural Information Processing Systems 29*.

*ICLR 2014 Conference*.

*Inverse Problems*.

*UAI17*.

*arXiv Preprint arXiv:1511.05121*.

*ICML 2020*.

*Advances in Neural Information Processing Systems*.

*Uncertainty in Artificial Intelligence*.

*CoRR*.

*arXiv:1512.09300 [Cs, Stat]*.

*Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS)*. Proceedings of Machine Learning Research.

*ICLR*.

*Technometrics*.

*arXiv Preprint arXiv:1705.10306*.

*Journal of Statistical Software*.

*Advances In Neural Information Processing Systems*.

*Journal of the American Statistical Association*.

*Workshop on Learning to Generate Natural Language*.

*Computer Methods in Applied Mechanics and Engineering*.

*Advances in Neural Information Processing Systems 30*.

*arXiv Preprint arXiv:1603.04733*.

*PMLR*.

*Neural Computation*.

*Information Theory, Inference & Learning Algorithms*.

*arXiv Preprint arXiv:1705.09279*.

*JMLR*.

*arXiv:2004.12550 [Stat]*.

*Proceedings of the 32nd International Conference on Machine Learning*.

*arXiv:1804.11271 [Cs, Stat]*.

*arXiv:1610.08733 [Stat]*.

*Proceedings of ICML*.

*Probabilistic Machine Learning: Advanced Topics*.

*Proceedings of the 28th International Conference on Machine Learning (ICML-11)*.

*Journal of Biomedical Informatics*.

*Technometrics*.

*Extremes*.

*Proceedings of the 33rd International Conference on Neural Information Processing Systems*.

*arXiv:2111.08239 [Cs, Stat]*.

*IEEE Transactions on Neural Networks*.

*Advances in Neural Information Processing Systems 30*.

*International Conference on Artificial Intelligence and Statistics*.

*Russian Mathematical Surveys*.

*Journal of Computational Physics*.

*Gaussian Processes for Machine Learning*. Adaptive Computation and Machine Learning.

*International Conference on Machine Learning*. ICML’15.

*Machine Learning and the Physical Sciences Workshop at the 33rd Conference on Neural Information Processing Systems (NeurIPS)*.

*arXiv:2105.14594 [Cs, Stat]*.

*arXiv:1604.00860 [Stat]*.

*Advances In Neural Information Processing Systems*.

*arXiv:1802.03335 [Stat]*.

*Machine Learning and the Physical Sciences Workshop at the 33rd Conference on Neural Information Processing Systems (NeurIPS)*.

*arXiv:1404.5886 [Math, Stat]*.

*Journal of the Royal Statistical Society: Series B (Statistical Methodology)*.

*Proceedings of the 32nd International Conference on Machine Learning*.

*Statistics and Computing*.

*arXiv:2107.10885 [Math, Stat]*.

*arXiv:2006.11695 [Cs, Stat]*.

*Advances in Neural Information Processing Systems*.

*ICLR*.

*arXiv:1610.09787 [Cs, Stat]*.

*Advances in Neural Information Processing Systems*.

*Journal of Machine Learning Research*.

*UAI18*.

*arXiv:1701.07989 [Math]*.

*New Directions in Statistical Signal Processing*.

*NeurIPS Workshop on Bayesian Deep Learning*.

*ICLR*.

*Proceedings of the 37th International Conference on Machine Learning*.

*arXiv:2011.11955 [Cs, Math]*.

*arXiv:2101.12353 [Cs, Math, Stat]*.

*Neural Networks: The Official Journal of the International Neural Network Society*.