Ensembling neural nets

Monte Carlo

One of the practical forms of Bayesian inference for massively parameterised networks by model averaging.

Explicit ensembles

Train a collection of networks and calculate empirical means and variances to estimate means posterior predictive (He, Lakshminarayanan, and Teh 2020; Huang et al. 2016; Lakshminarayanan, Pritzel, and Blundell 2017; Wen, Tran, and Ba 2020; Xie, Xu, and Chuang 2013). This is neat and on one hand we might think there is nothing special to do here since it’s already more or less classical model ensembling, as near as I can tell. But in practice there are lots of tricks needed to make this go in a neural network context, in particular because models are already supposed to be so big that they strain the GPU; having many such models is presumably ridiculous. You need tricks. There are various such tricks. BatchEnsemble is one (Wen, Tran, and Ba 2020).

Cute: Justin Domke, in The human regression ensemble, creates ensembles of curves that he drew through datapoints on a PDF and gets pretty good results.


Dropout is an implicit ensembling method. Or maybe the implicit ensembling method; I am not aware of others. Recommended reading: Foong et al. (2019);Gal, Hron, and Kendall (2017);Kingma, Salimans, and Welling (2015).

A popular kind of noise layer which randomly zeroes out some coefficients in the net when training (and optionally while predicting.) A coarse resemblance to random forests etc is pretty immediate, and indeed you can just use those instead. Here, however, we are trying to average over strong learners, not weak learners.

The key insight here is that dropout can be rationalized, apparently, as model averaging and thence as a kind of implicit probabilistic learning because in the limit it approaches a certain deep Gaussian process (Kingma, Salimans, and Welling 2015; Gal and Ghahramani 2016b, 2015). Leveraging this argument there are some papers that claim to approximate Bayesian inference by randomizing dropout (M. Kasim et al. 2019; M. F. Kasim et al. 2020).

AFAICT current consensus seems to be that highly cited and very simple model of Gal and Ghahramani (2015) is flawed, and that the rather more onerous approach of Kingma, Salimans, and Welling (2015) is how you would use dropout as a more reasonable posterior; So much was said in a seminar, but I have not really used either paper in practice so I cannot comment.

Alternate model combinations

Should we stop weighting hypotheses and start β€œstacking”? Yao et al. (2018) (also how is that different?)

The widely recommended procedure of Bayesian model averaging is flawed in the M-open setting in which the true data-generating process is not one of the candidate models being fit. We take the idea of stacking from the point estimation literature and generalize to the combination of predictive distributions, extending the utility function to any proper scoring rule, using Pareto smoothed importance sampling to efficiently compute the required leave-one-out posterior distributions and regularization to get more stability.


So apparently you can train a model to emulate an ensemble of similar models? Great terminology refer to distilling of dark knowledge.

See Bubeck on this: Three mysteries in deep learning: Ensemble, knowledge distillation, and self-distillation.


How does this work? He, Lakshminarayanan, and Teh (2020).


These methods focus generally on the posterior predictive. How do I find posteriors for parameter values in my model without including them in my predictive loss explicitly? If many of my parameters are not interpretable I am naturally tempted to fit some by Maximum Likelihood, take them as given then update posteriors over the remainder, but this does not look like a principled inference procedure.


Alquier, Pierre. 2021. β€œUser-Friendly Introduction to PAC-Bayes Bounds.” arXiv:2110.11216 [Cs, Math, Stat], October.
Chada, Neil, and Xin Tong. 2022. β€œConvergence Acceleration of Ensemble Kalman Inversion in Nonlinear Settings.” Mathematics of Computation 91 (335): 1247–80.
Chipman, Hugh A, Edward I George, and Robert E Mcculloch. 2006. β€œBayesian Ensemble Learning.” In, 8.
Clarke, Bertrand. 2003. β€œComparing Bayes Model Averaging and Stacking When Model Approximation Error Cannot Be Ignored.” The Journal of Machine Learning Research 4 (null): 683–712.
Dandekar, Raj, Karen Chung, Vaibhav Dixit, Mohamed Tarek, Aslan Garcia-Valadez, Krishna Vishal Vemula, and Chris Rackauckas. 2021. β€œBayesian Neural Ordinary Differential Equations.” arXiv:2012.07244 [Cs], March.
Daxberger, Erik, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. 2021. β€œLaplace Redux β€” Effortless Bayesian Deep Learning.” In arXiv:2106.14806 [Cs, Stat].
Durasov, Nikita, Timur Bagautdinov, Pierre Baque, and Pascal Fua. 2021. β€œMasksembles for Uncertainty Estimation.” In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). arXiv.
Foong, Andrew Y K, David R Burt, Yingzhen Li, and Richard E Turner. 2019. β€œPathologies of Factorised Gaussian and MC Dropout Posteriors in Bayesian Neural Networks.” In 4th Workshop on Bayesian Deep Learning (NeurIPS 2019), 17.
Gal, Yarin, and Zoubin Ghahramani. 2015. β€œDropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning.” In Proceedings of the 33rd International Conference on Machine Learning (ICML-16).
β€”β€”β€”. 2016a. β€œA Theoretically Grounded Application of Dropout in Recurrent Neural Networks.” In arXiv:1512.05287 [Stat].
β€”β€”β€”. 2016b. β€œBayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference.” In 4th International Conference on Learning Representations (ICLR) Workshop Track.
β€”β€”β€”. 2016c. β€œDropout as a Bayesian Approximation: Appendix.” arXiv:1506.02157 [Stat], May.
Gal, Yarin, Jiri Hron, and Alex Kendall. 2017. β€œConcrete Dropout.” arXiv:1705.07832 [Stat], May.
Haber, Eldad, Felix Lucka, and Lars Ruthotto. 2018. β€œNever Look Back - A Modified EnKF Method and Its Application to the Training of Neural Networks Without Back Propagation.” arXiv:1805.08034 [Cs, Math], May.
He, Bobby, Balaji Lakshminarayanan, and Yee Whye Teh. 2020. β€œBayesian Deep Ensembles via the Neural Tangent Kernel.” In Advances in Neural Information Processing Systems. Vol. 33.
Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. 2015. β€œDistilling the Knowledge in a Neural Network.” arXiv:1503.02531 [Cs, Stat], March.
Huang, Gao, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. 2016. β€œDeep Networks with Stochastic Depth.” In Computer Vision – ECCV 2016, edited by Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, 646–61. Lecture Notes in Computer Science. Cham: Springer International Publishing.
Kasim, M. F., D. Watson-Parris, L. Deaconu, S. Oliver, P. Hatfield, D. H. Froula, G. Gregori, et al. 2020. β€œUp to Two Billion Times Acceleration of Scientific Simulations with Deep Neural Architecture Search.” arXiv:2001.08055 [Physics, Stat], January.
Kasim, Muhammad, J Topp-Mugglestone, P Hatfield, D H Froula, G Gregori, M Jarvis, E Viezzer, and Sam Vinko. 2019. β€œA Million Times Speed up in Parameters Retrieval with Deep Learning.” In, 5.
Kingma, Diederik P., Tim Salimans, and Max Welling. 2015. β€œVariational Dropout and the Local Reparameterization Trick.” In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2, 2575–83. NIPS’15. Cambridge, MA, USA: MIT Press.
Kovachki, Nikola B., and Andrew M. Stuart. 2019. β€œEnsemble Kalman Inversion: A Derivative-Free Technique for Machine Learning Tasks.” Inverse Problems 35 (9): 095005.
Lakshminarayanan, Balaji, Alexander Pritzel, and Charles Blundell. 2017. β€œSimple and Scalable Predictive Uncertainty Estimation Using Deep Ensembles.” In Proceedings of the 31st International Conference on Neural Information Processing Systems, 6405–16. NIPS’17. Red Hook, NY, USA: Curran Associates Inc.
Le, Tri, and Bertrand Clarke. 2017. β€œA Bayes Interpretation of Stacking for M-Complete and M-Open Settings.” Bayesian Analysis 12 (3): 807–29.
Mandt, Stephan, Matthew D. Hoffman, and David M. Blei. 2017. β€œStochastic Gradient Descent as Approximate Bayesian Inference.” JMLR, April.
Minka, Thomas P. 2002. β€œBayesian Model Averaging Is Not Model Combination.”
Molchanov, Dmitry, Arsenii Ashukha, and Dmitry Vetrov. 2017. β€œVariational Dropout Sparsifies Deep Neural Networks.” In Proceedings of ICML.
Papadopoulos, G., P.J. Edwards, and A.F. Murray. 2001. β€œConfidence Estimation Methods for Neural Networks: A Practical Comparison.” IEEE Transactions on Neural Networks 12 (6): 1278–87.
Pearce, Tim, Mohamed Zaki, and Andy Neely. 2018. β€œBayesian Neural Network Ensembles.” Third Workshop on Bayesian Deep Learning (NeurIPS 2018), MontrΓ©al, Canada., 5.
Ritter, Hippolyt, Martin Kukla, Cheng Zhang, and Yingzhen Li. 2021. β€œSparse Uncertainty Representation in Deep Learning with Inducing Weights.” arXiv:2105.14594 [Cs, Stat], May.
Sheikh, Hassam Ullah, Mariano Phielipp, and Ladislau Boloni. 2022. β€œMaximizing Ensemble Diversity in Deep Reinforcement Learning,” 25.
Wang, Xiaofang, Dan Kondratyuk, Eric Christiansen, Kris M. Kitani, Yair Alon, and Elad Eban. 2021. β€œWisdom of Committees: An Overlooked Approach To Faster and More Accurate Models.” arXiv:2012.01988 [Cs], October.
Wen, Yeming, Dustin Tran, and Jimmy Ba. 2020. β€œBatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning.” In ICLR.
Wilson, Andrew Gordon, and Pavel Izmailov. 2020. β€œBayesian Deep Learning and a Probabilistic Perspective of Generalization,” February.
Wortsman, Mitchell, Maxwell Horton, Carlos Guestrin, Ali Farhadi, and Mohammad Rastegari. 2021. β€œLearning Neural Network Subspaces.” arXiv:2102.10472 [Cs], February.
Xie, Jingjing, Bing Xu, and Zhang Chuang. 2013. β€œHorizontal and Vertical Ensemble with Deep Representation for Classification.” arXiv:1306.2759 [Cs, Stat], June.
Yao, Yuling, Aki Vehtari, Daniel Simpson, and Andrew Gelman. 2018. β€œUsing Stacking to Average Bayesian Predictive Distributions.” Bayesian Analysis 13 (3): 917–1007.

No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.