- Backgrounders
- MC sampling of weights by low-rank Matheron updates
- Mixture density networks
- Variational autoencoders
- Sampling via Monte Carlo
- Stochastic Gradient Descent as MC inference
- Laplace approximation
- Via random projections
- In Gaussian process regression
- Via measure transport
- Via infinite-width random nets
- Via NTK
- Ensemble methods
- Practicalities
- References

Inferring densities and distributions in a massively parameterised deep learning setting.

This is not intrinsically a Bayesian thing to do but in practice much of the demand to do probabilistic nets comes from the demand for Bayesian posterior inference for neural nets. Bayesian inference is, however, not the only way to do uncertainty quantification.

Neural networks are very far from simple exponential families where conjugate distributions might help, and so typically rely upon approximations or luck to approximate our true target of interest.

Closely related: Generative models where we train a process to generate a (possibly stochastic) phenomenon of interest.

## Backgrounders

Radford Neal’s thesis (Neal 1996) is a foundational Bayesian use of neural networks in the wide NN and MCMC sampling settings. Diederik P. Kingma’s thesis is a blockbuster in the more recent variational tradition.

Alex Graves’ poster of his paper (Graves 2011) of a simplest prior uncertainty thing for recurrent nets - (diagonal Gaussian weight uncertainty) I found elucidating. (There is a 3rd party quick and dirty implementation.)

One could refer to the 2019 NeurIPS Bayes deep learning workshop site which will have some more modern positioning. There was a tutorial in 2020: by Dustin Tran, Jasper Snoek, Balaji Lakshminarayanan: Practical Uncertainty Estimation & Out-of-Distribution Robustness in Deep Learning.

Generative methods are useful here, e.g. the variational autoencoder and affiliated reparameterization trick. Likelihood free methods seems to be in the air too.

We are free to consider classic neural network inference as sort-of a special case of Bayes inference. Specifically, we interpret the loss function \(\mathcal{L}\) of a net \(f:\mathbb{R}^n\times\mathbb{R}^d\to\mathbb{R}^k\) in the likelihood setting \[ \begin{aligned} \mathcal{L}(\theta) &:=-\sum_{i=1}^{m} \log p\left(y_{i} \mid f\left(x_{i} ; \theta\right)\right)-\log p(\theta) \\ &=-\log p(\theta \mid \mathcal{D}). \end{aligned} \]

Obviously a few things are different here; the parameter vector \(\theta\) is not interpretable, so what do posterior distributions over it even mean? What are sensible priors? Choosing priors over by-design-uninterpretable parameters such as NN weights is a whole fraught thing that we will mostly ignore for now. Usually it is by default something like \[ p(\theta)=\mathcal{N}\left(0, \lambda^{-1} I\right) \] for want of a better idea. This ends up being equivalent to the “weight decay” regularisation.

Sweeping those qualms aside, we could do the usual stuff for Bayes inference, like considering the predictive posterior \[ p(y \mid x, \mathcal{D})=\int p(y \mid f(x ; \theta)) p(\theta \mid \mathcal{D}) d \theta \]

Usually this turns out to be intractable to calculate in the very high dimension spaces of an NN, so we encode the net as a simple maximum a posteriori estimate \[ \theta_{\mathrm{MAP}}:=\operatorname{argmin}_{\theta} \mathcal{L}(\theta). \] In this case we have recovered the classic training of non-Bayes nets. But we have no notion of predictive uncertainty in that setting.

Usually the model will also have many symmetries so we know that it has many optima, which makes approximations that leverage particular modes like a MAP estimate.

Somewhere between the full belt-and-braces Bayes approach and the MAP point estimate there are various approximations to Bayes inference we might try.

🏗 To discuss: so many options for predictive uncertainty, but fewer for inverse uncertainty.

What follows is an non-exhaustive smörgåsbord of options to do probabilistic inference in neural nets.

## MC sampling of weights by low-rank Matheron updates

Needs a shorter names but looks cool (Ritter et al. 2021).

microsoft/bayesianize: Bayesianize: A Bayesian neural network wrapper in pytorch

## Mixture density networks

Nothing to say for now but here are some recommendations I received about this classic (C. Bishop 1994) method.

## Variational autoencoders

## Sampling via Monte Carlo

TBD. For now, if the number of parameters is smallish see Hamiltonian Monte Carlo.

## Stochastic Gradient Descent as MC inference

I have a vague memory that this argument is leveraged in Neal (1996)? But see the version in Mandt, Hoffman, and Blei (2017) for a highly developed modern take:

Stochastic Gradient Descent with a constant learning rate (constant SGD) simulates a Markov chain with a stationary distribution. With this perspective, we derive several new results.

- We show that constant SGD can be used as an approximate Bayesian posterior inference algorithm. Specifically, we show how to adjust the tuning parameters of constant SGD to best match the stationary distribution to a posterior, minimizing the Kullback-Leibler divergence between these two distributions.
- We demonstrate that constant SGD gives rise to a new variational EM algorithm that optimizes hyperparameters in complex probabilistic models.
- We also propose SGD with momentum for sampling and show how to adjust the damping coefficient accordingly.
- We analyze MCMC algorithms. For Langevin Dynamics and Stochastic Gradient Fisher Scoring, we quantify the approximation errors due to finite learning rates. Finally,
- we use the stochastic process perspective to give a short proof of why Polyak averaging is optimal. Based on this idea, we propose a scalable approximate MCMC algorithm, the Averaged Stochastic Gradient Sampler.

A popular recent version of this is the Stochastic Weight Averaging family (Izmailov et al. 2018, 2020; Maddox et al. 2019; Wilson and Izmailov 2020), which I am interested in. See Andrew G Wilson’s web page for a brief description of the sub methods here, since he seems to have been involved in all of them.

## Laplace approximation

A Laplace approximation locally approximates the posterior using a Gaussian \[ p(\theta \mid \mathcal{D}) \approx \mathcal{N}\left(\theta_{\mathrm{MAP}}, \Sigma\right). \] Such an approach is classic for neural nets (David J. C. MacKay 1992). There are many variants of this technique for different assumptions. Laplace approximations have the attractive feature of providing estimates for forward and inverse problems (Foong et al. 2019; Immer, Korzepa, and Bauer 2021) by leveraging the delta method.

The basic idea is that we hold \(x \in \mathbb{R}^{n}\) fixed and use the Jacobian matrix \(J(x):=\left.\nabla_{\theta} f(x ; \theta)\right|_{\theta_{\mathrm{MAP}}} \in \mathbb{R}^{d \times k}\), to the network as
\[
f(x ; \theta) \approx f\left(x ; \theta_{\mathrm{MAP}}\right)+J(x)^{\top}\left(\theta-\theta_{\mathrm{MAP}}\right)
\]
where the variance is now justifed as a Taylor expansion.
Under this approximation, since \(\theta\) is a posteriori distributed as Gaussian \(\mathcal{N}\left(\theta_{\mathrm{MAP}}, \Sigma\right)\), it follows that the marginal distribution over the network output \(f(x)\) is also Gaussian, given by
\[
p(f(x) \mid x, \mathcal{D}) \sim \mathcal{N}\left(f\left(x ; \theta_{\mathrm{MAP}}\right), J(x)^{\top} \Sigma J(x)\right).
\]
For more on this, see (C. M. Bishop 2006, 5.167, 5.188).
It is essentially a gratis Laplace approximation in the sense that if I have fit the networks I can already calculate those Jacobians so I am probably 1 line of code away from getting some kind of uncertainty estimate.
However, I have no particular guarantees to hope that it is well calibrated, because the simplifications were chosen *a priori* and might not be appropriate to the combination of model and data that I actually have.

### Learnable Laplace approximations

Agustinus Kristiadi and team have created various methods for low-overhead neural uncertainty quantification via Laplace approximation that have greater flexibility for adaptively choosing the type and manner of approximation. See, e.g. Painless Uncertainty for Deep Learning and their papers (Kristiadi, Hein, and Hennig 2020, 2021b).

One interesting variant is that of Kristiadi, Hein, and Hennig (2021b) which generalises to learnable uncertainty to, for example, allow the distribution to reflect uncertainty about outlier datapoints.
They define an augmented *Learnable Uncertainty Laplace Approximation* (LULA) network \(\tilde{f}\) with more parameters \(\tilde{\theta}=\theta_{\mathrm{MAP}}, \hat{\theta}.\)

Let \(f: \mathbb{R}^{n} \times \mathbb{R}^{d} \rightarrow \mathbb{R}^{k}\) be an \(L\)-layer neural network with a MAP-trained parameters \(\theta_{\text {MAP }}\) and let \(\widetilde{f}: \mathbb{R}^{n} \times \mathbb{R}^{\widetilde{d}} \rightarrow \mathbb{R}^{k}\) along with \(\widetilde{\theta}_{\text {MAP }}\) be obtained by adding LULA units. Let \(q(\widetilde{\theta}):=\mathcal{N}\left(\tilde{\theta}_{\mathrm{MAP}}, \widetilde{\Sigma}\right)\) be the Laplace-approximated posterior and \(p\left(y \mid x, \mathcal{D} ; \widetilde{\theta}_{\mathrm{MAP}}\right)\) be the (approximate) predictive distribution under the LA. Furthermore, let us denote the dataset sampled i.i.d. from the data distribution as \(\mathcal{D}_{\text {in }}\) and that from some outlier distribution as \(\mathcal{D}_{\text {out }}\), and let \(H\) be the entropy functional. We construct the following loss function to induce high uncertainty on outliers while maintaining high confidence over the data (inliers): \[ \begin{array}{rl} \mathcal{L}_{\text {LULA }}\left(\widetilde{\theta}_{\text {MAP }}\right)&:=\frac{1}{\left|\mathcal{D}_{\text {in }}\right|} \sum_{x_{\text {in }} \in \mathcal{D}_{\text {in }}} H\left[p\left(y \mid x_{\text {in }}, \mathcal{D} ; \widetilde{\theta}_{\text {MAP }}\right)\right] \\ &-\frac{1}{\left|\mathcal{D}_{\text {out }}\right|} \sum_{x_{\text {out }} \in \mathcal{D}_{\text {out }}} H\left[p\left(y \mid x_{\text {out }}, \mathcal{D} ; \widetilde{\theta}_{\text {MAP }}\right)\right] \end{array} \] and minimize it w.r.t. the free parameters \(\widehat{\theta}\).

I am assuming here that by the *entropy functional* they mean the entropy of the normal distribution,
\[
H(\mathcal{N}(\mu, \sigma)) = {\frac {1}{2}}\ln \left((2\pi \mathrm {e} )^{k}\det \left({\boldsymbol {\Sigma }}\right)\right)
\]
but this looks expensive due to that determinant calculation in a (large) \(d\times d\) matrix.
Or possibly they mean some general entropy with respect to some density \(p\)
\[H(p)=\mathbb{E}_{p}\left[-\log p( x)\right]\]
which I suppose one could estimate as
\[H(p)=\frac{1}{N}\sum_{i=1}^N \left[-\log p(x_i)\right]\] without taking that normal Laplace approximation at this step, if we could find the density, and assuming the \(x_i\) were drawn from it.

The result is a slightly weird hybrid fitting procedure that requires two loss functions and which feels a little *ad hoc*, but maybe it works?

### By stochastic weight averaging.

A Bayesian extension of Stochastic Weight Averaging. Izmailov et al. (2018); Izmailov et al. (2020); Maddox et al. (2019); Wilson and Izmailov (2020)

## Via random projections

I do not have a single paper about this, but I have seen random projection pop up as a piece of the puzzle in other methods. TBC.

## In Gaussian process regression

See kernel learning.

## Via measure transport

See reparameterization.

## Via infinite-width random nets

See wide NN.

## Via NTK

How does this work? He, Lakshminarayanan, and Teh (2020).

## Ensemble methods

Deep learning has its own twists on model averaging and bagging: Neural ensembles. Yarin Gal’s PhD Thesis (Gal 2016) summarizes some implicit approximate approaches (e.g. the Bayesian interpretation of dropout) although dropout as he frames it has become highly controversial these days as a means of inference..

## Practicalities

The computational toolsets for “neural” probabilistic programming and vanilla probabilistic programming are converging. See the tool listing under probabilistic programming.

## References

*Advances in Neural Information Processing Systems 29*. http://arxiv.org/abs/1611.07800.

*arXiv:1511.07367 [stat]*, November. http://arxiv.org/abs/1511.07367.

*Inverse Problems*36 (11): 115003. https://doi.org/10.1088/1361-6420/abb447.

*arXiv:1907.03382 [cs, Stat]*. http://arxiv.org/abs/1907.03382.

*Uai18*. http://arxiv.org/abs/1803.05649.

*Microsoft Research*, January. https://www.microsoft.com/en-us/research/publication/mixture-density-networks/.

*Pattern Recognition and Machine Learning*. Information Science and Statistics. New York: Springer.

*International Conference on Machine Learning*, 537–46. http://arxiv.org/abs/1703.03208.

*arXiv:1703.04818 [cs]*, March. http://arxiv.org/abs/1703.04818.

*Computer Physics Communications*244 (November): 170–79. https://doi.org/10.1016/j.cpc.2019.06.007.

*Advances in Neural Information Processing Systems 31*, edited by S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, 6572–83. Curran Associates, Inc. http://papers.nips.cc/paper/7892-neural-ordinary-differential-equations.pdf.

*PMLR*. http://proceedings.mlr.press/v70/cutajar17a.html.

*Artificial Intelligence and Statistics*, 207–15. http://proceedings.mlr.press/v31/damianou13a.html.

*arXiv:2012.07244 [cs]*, March. http://arxiv.org/abs/2012.07244.

*Advances in Neural Information Processing Systems 28*, 1414–22. NIPS’15. Cambridge, MA, USA: MIT Press. http://dl.acm.org/citation.cfm?id=2969239.2969397.

*arXiv:1801.10395 [stat]*, January. http://arxiv.org/abs/1801.10395.

*arXiv:2012.00152 [cs, Stat]*, November. http://arxiv.org/abs/2012.00152.

*Journal of Machine Learning Research*19 (1): 2100–2145. http://jmlr.org/papers/v19/18-015.html.

*arXiv:1904.01681 [cs, Stat]*, April. http://arxiv.org/abs/1904.01681.

*arXiv:2105.04504 [cs, Stat]*, May. http://arxiv.org/abs/2105.04504.

*Advances in Neural Information Processing Systems 30*, edited by I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, 5309–19. Curran Associates, Inc. http://papers.nips.cc/paper/7115-identification-of-gaussian-process-state-space-models.pdf.

*Proceedings of ICLR*. http://arxiv.org/abs/1412.6581.

*Advances in Neural Information Processing Systems 31*, edited by S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, 441–52. Curran Associates, Inc. http://papers.nips.cc/paper/7326-implicit-reparameterization-gradients.pdf.

*arXiv:1704.04110 [cs, Stat]*, April. http://arxiv.org/abs/1704.04110.

*arXiv:1906.11537 [cs, Stat]*, June. http://arxiv.org/abs/1906.11537.

*Advances in Approximate Bayesian Inference Workshop, NIPS*.

*Advances in Approximate Bayesian Inference Workshop, NIPS*.

*Proceedings of the 33rd International Conference on Machine Learning (ICML-16)*. http://arxiv.org/abs/1506.02142.

*arXiv:1512.05287 [stat]*. http://arxiv.org/abs/1512.05287.

*4th International Conference on Learning Representations (ICLR) Workshop Track*. http://arxiv.org/abs/1506.02158.

*arXiv:1506.02157 [stat]*, May. http://arxiv.org/abs/1506.02157.

*arXiv:1705.07832 [stat]*, May. http://arxiv.org/abs/1705.07832.

*arXiv:1807.01613 [cs, Stat]*, July, 10. https://arxiv.org/abs/1807.01613v1.

*arXiv:1902.10298 [cs]*, February. http://arxiv.org/abs/1902.10298.

*IEEE Transactions on Signal Processing*64 (13): 3444–57. https://doi.org/10.1109/TSP.2016.2546221.

*Journal of Applied Econometrics*8 (December): S85–118. http://www.jstor.org/stable/2285076.

*Proceedings of the 24th International Conference on Neural Information Processing Systems*, 2348–56. NIPS’11. USA: Curran Associates Inc. https://papers.nips.cc/paper/4329-practical-variational-inference-for-neural-networks.pdf.

*arXiv:1308.0850 [cs]*, August. http://arxiv.org/abs/1308.0850.

*2013 IEEE International Conference on Acoustics, Speech and Signal Processing*. https://doi.org/10.1109/ICASSP.2013.6638947.

*arXiv:1502.04623 [cs]*, February. http://arxiv.org/abs/1502.04623.

*Advances in Neural Information Processing Systems 28*, edited by C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, 2629–37. Curran Associates, Inc. http://papers.nips.cc/paper/5961-neural-adaptive-sequential-monte-carlo.pdf.

*Proceedings of ICLR*. https://arxiv.org/abs/1511.05176v3.

*Advances in Neural Information Processing Systems*. Vol. 33. https://proceedings.neurips.cc//paper_files/paper/2020/hash/0b1ec366924b26fc98fa7b71a9c249cf-Abstract.html.

*PMLR*, 361–69. http://proceedings.mlr.press/v38/hoffman15.html.

*arXiv:1706.00550 [cs, Stat]*. http://arxiv.org/abs/1706.00550.

*International Conference on Artificial Intelligence and Statistics*, 703–11. PMLR. https://proceedings.mlr.press/v130/immer21a.html.

*Proceedings of The 35th Uncertainty in Artificial Intelligence Conference*, 1169–79. PMLR. https://proceedings.mlr.press/v115/izmailov20a.html.

*Proceedings of ICLR*. http://arxiv.org/abs/1605.06432.

*Advances in Neural Information Processing Systems 29*. Curran Associates, Inc. http://arxiv.org/abs/1606.04934.

*ICLR 2014 Conference*. http://arxiv.org/abs/1312.6114.

*Uai17*. http://arxiv.org/abs/1610.05392.

*arXiv Preprint arXiv:1511.05121*. https://arxiv.org/abs/1511.05121.

*ICML 2020*. http://arxiv.org/abs/2002.10118.

*arXiv:2010.02709 [cs, Stat]*, May. http://arxiv.org/abs/2010.02709.

*Uncertainty in Artificial Intelligence*. http://arxiv.org/abs/2010.02720.

*arXiv:1512.09300 [cs, Stat]*, December. http://arxiv.org/abs/1512.09300.

*Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS)*, 54:1338–48. Proceedings of Machine Learning Research. Fort Lauderdale, FL, USA: PMLR. http://arxiv.org/abs/1610.09900.

*arXiv Preprint arXiv:1705.10306*. https://arxiv.org/abs/1705.10306.

*Technometrics*44 (3): 230–41. https://doi.org/10.1198/004017002188618419.

*ICLR*. http://arxiv.org/abs/1711.00165.

*Journal of the American Statistical Association*0 (0): 1–18. https://doi.org/10.1080/01621459.2020.1863223.

*Workshop on Learning to Generate Natural Language*. http://arxiv.org/abs/1708.00077.

*Advances in Neural Information Processing Systems 30*, edited by I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, 6446–56. Curran Associates, Inc. http://papers.nips.cc/paper/7223-causal-effect-inference-with-deep-latent-variable-models.pdf.

*arXiv Preprint arXiv:1603.04733*, 1708–16. http://arxiv.org/abs/1603.04733.

*PMLR*, 2218–27. http://proceedings.mlr.press/v70/louizos17a.html.

*Neural Computation*4 (3): 448–72. https://doi.org/10.1162/neco.1992.4.3.448.

*Information Theory, Inference & Learning Algorithms*. Cambridge University Press.

*arXiv Preprint arXiv:1705.09279*. https://arxiv.org/abs/1705.09279.

*JMLR*, April. http://arxiv.org/abs/1704.04289.

*arXiv:1804.11271 [cs, Stat]*, August. http://arxiv.org/abs/1804.11271.

*arXiv:1610.08733 [stat]*, October. http://arxiv.org/abs/1610.08733.

*Proceedings of ICML*. http://arxiv.org/abs/1701.05369.

*Proceedings of the 28th International Conference on Machine Learning (ICML-11)*, 1105–12. http://machinelearning.wustl.edu/mlpapers/paper_files/ICML2011Ngiam_557.pdf.

*Technometrics*59 (1): 80–92. https://doi.org/10.1080/00401706.2015.1125391.

*Advances in Neural Information Processing Systems 30*, edited by I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, 2338–47. Curran Associates, Inc. http://papers.nips.cc/paper/6828-masked-autoregressive-flow-for-density-estimation.pdf.

*International Conference on Artificial Intelligence and Statistics*, 1126–36. PMLR. http://proceedings.mlr.press/v108/peluchetti20a.html.

*Journal of Computational Physics*378 (February): 686–707. https://doi.org/10.1016/j.jcp.2018.10.045.

*Gaussian Processes for Machine Learning*. Adaptive Computation and Machine Learning. Cambridge, Mass: MIT Press. http://www.gaussianprocess.org/gpml/.

*International Conference on Machine Learning*, 1530–38. ICML’15. Lille, France: JMLR.org. http://arxiv.org/abs/1505.05770.

*Machine Learning and the Physical Sciences Workshop at the 33rd Conference on Neural Information Processing Systems (NeurIPS)*, 6.

*arXiv:2105.14594 [cs, Stat]*, May. http://arxiv.org/abs/2105.14594.

*Advances In Neural Information Processing Systems*. http://arxiv.org/abs/1610.02287.

*arXiv:1802.03335 [stat]*, February. http://arxiv.org/abs/1802.03335.

*Machine Learning and the Physical Sciences Workshop at the 33rd Conference on Neural Information Processing Systems (NeurIPS)*, 11.

*Journal of the Royal Statistical Society: Series B (Statistical Methodology)*77 (1): 3–33. https://doi.org/10.1111/rssb.12061.

*ICLR*. http://arxiv.org/abs/1701.03757.

*arXiv:1610.09787 [cs, Stat]*, October. http://arxiv.org/abs/1610.09787.

*New Directions in Statistical Signal Processing*. Vol. 155. MIT Press.

*ICLR*. http://arxiv.org/abs/2002.06715.

*arXiv:2011.11955 [cs, Math]*. http://arxiv.org/abs/2011.11955.

*arXiv:2101.12353 [cs, Math, Stat]*, January. http://arxiv.org/abs/2101.12353.

*Neural Networks: The Official Journal of the International Neural Network Society*10 (1): 99–109. https://doi.org/10.1016/S0893-6080(96)00037-8.

## No comments yet. Why not leave one?