# Model averaging, model stacking, model ensembling

On keeping many incorrect hypotheses and using them all as one goodish one

June 20, 2017 — July 24, 2023

Train a bunch of different models and use them all. Fashionable in the form of *blending*, *stacking* or *staging* in machine learning competitions, but also popular in classic frequentist inference as model averaging or bagging, or in e.g. posterior predictives in Bayes inference, which especially in the MCMC methods are easy to interpret as weighted ensembles.

I’ve seen the idea pop up in disconnected areas recently. Specifically: a Bayesian heuristic for dropout in neural nets, AIC for frequentist model averaging, Neural net ensembles, boosting/bagging, and in a statistical learning context for optimal time series prediction.

This vexingly incomplete article points out that something like model averaging might work for any convex loss thanks to Jensen’s inequality.

Two articles (Clarke 2003; Minka 2002) point out that model averaging and *combination* are not the same and the difference is acute in the M-open setting.

## 1 Mixtures of models

See mixtue models.

## 2 Stacking

Alternate fun branding: “super learning”. Not actually model averaging, but looks pretty similar if you squint.

Breiman (1996); Clarke (2003); T. Le and Clarke (2017); Naimi and Balzer (2018); Ting and Witten (1999); Wolpert (1992); Yao et al. (2022); Y. Zhang et al. (2022)

## 3 Bayesian stacking

As above, but Bayesian. Motivates suggestive invocation of M-open machinery. (Clarke 2003; Clyde and Iversen 2013; Hoeting et al. 1999; T. Le and Clarke 2017; T. M. Le and Clarke 2022; Minka 2002; Naimi and Balzer 2018; Polley 2010; Ting and Witten 1999; Wolpert 1992; Yao et al. 2022, 2018)

## 4 Forecasting

Time series prediction? Try ensemble methods for time series.

## 5 References

*arXiv:2110.11216 [Cs, Math, Stat]*.

*Journal of the Operational Research Society*.

*arXiv:1212.2447 [Cs, Stat]*.

*Machine Learning*.

*Biometrics*.

*Proceedings of the Conference on Fairness, Accountability, and Transparency*.

*Model Selection and Model Averaging*. Cambridge Series in Statistical and Probabilistic Mathematics.

*The Journal of Machine Learning Research*.

*Statistical Science*.

*Bayesian Theory and Applications*.

*arXiv:1509.08864 [Stat]*.

*The Computer Journal*.

*Engineering Applications of Artificial Intelligence*.

*Econometrica*.

*Advances in Neural Information Processing Systems*.

*arXiv:1503.02531 [Cs, Stat]*.

*Journal of the American Statistical Association*.

*Statistical Science*.

*The Canadian Journal of Statistics / La Revue Canadienne de Statistique*.

*Biometrika*.

*Bayesian Analysis*.

*Journal of Machine Learning Research*.

*IEEE Transactions on Information Theory*.

*European Journal of Epidemiology*.

*Journal of Business & Economic Statistics*.

*Statistics and Computing*.

*U.C. Berkeley Division of Biostatistics Working Paper Series*.

*arXiv:2106.02589 [Math, Stat]*.

*Journal of the American Statistical Association*.

*arXiv:2106.03253 [Cs]*.

*Journal of Artificial Intelligence Research*.

*Statistical Applications in Genetics and Molecular Biology*.

*arXiv:2012.01988 [Cs]*.

*Journal of Systems Science and Complexity*.

*Advances in Neural Information Processing Systems*.

*Neural Networks*.

*Bayesian Analysis*.

*Bayesian Analysis*.

*arXiv:2012.02130 [Cs, Stat]*.

*The Annals of Statistics*.

*GIScience & Remote Sensing*.