# Bayes for beginners

May 30, 2016 — July 23, 2022

Bayes
generative
how do science
Monte Carlo
statistics

Even for the most currmudgeonly frequentist it is sometimes refreshing to move your effort from deriving frequentist estimators for intractable models, to using the damn Bayesian ones, which fail in different and interesting ways than you are used to. If it works and you are feeling fancy you might then justify your Bayesian method on frequentist grounds, which washes away the sin.

Here are some scattered tidbits about getting into it. No attempt is made to be comprehensive, novel, or to even expert.

## 1 Prior choice

Is weird and important. Here are some argumentative and disputed rules of thumb.

## 2 Teaching

### 2.1 Course material

So many! Too many. Actually I kinda like McElreath’s stuff to teach from; You get practical quite quickly.

• Bayesian Data Analysis course - GSU 2022

• How to measure anything

• The milieu around Andrew Gelman . These are very good courses for the kind of statistics most people need, including people who think they need different statistics. Bayesian Data Analysis is online

• McElreath (2020) is a cult textbook which various people haver reimplemented in various languages. It is remarkable how far this takes some very simple computational tools.

• Cameron Davidson-Pilon, Probabilistic Programming & Bayesian Methods for Hackers (source) is an interesting one; does what is says on the tin. IMO McElreath is just a bit better, even for hackers, but this is cheaper and still a good start.

## 3 Linear regression

This workhorse pops up everywhere.

Deisenroth and Zafeiriou, Mathematics for Inference and Machine Learning give an ML perspective.

## 4 Workflow

If we want to use Bayesian tools to do science there is a principled workflow that we need to be thinking about. For a fun rant read Shalizi on Praxis and Ideology in Bayesian Data Analysis, about Gelman and Shalizi (2013).

The visualisation howto from, basically, the Stan team, is a deeper than it sounds and highly recommended .

Michael Betancourt’s examples, for example his workflow tips, are a good start for practical work, incorporating the inevitable collision of statistical and computational difficulties.

See also BAT the Bayesian Analysis Toolkit, which does sophisticated Bayes modelling although AFAICT uses a fairly basic Sampler?

Notes on Rao-Blackwellisation for doing faster MCMC inference, and even handling discrete parameters in Stan.

## 5 Nonparametrics

Dirichlet processes, Gaussian Process regression etc. 🏗

## 10 References

Alquier. 2021. arXiv:2110.11216 [Cs, Math, Stat].
Bacchus, Kyburg, and Thalos. 1990. “Against Conditionalization.” Synthese.
Barbier, and Macris. 2017. arXiv:1705.02780 [Cond-Mat].
Bernardo, and Smith. 2000. Bayesian Theory.
Carpenter, Hoffman, Brubaker, et al. 2015. arXiv Preprint arXiv:1509.07164.
Caruana. 1998. In Learning to Learn.
Deisenroth, and Zafeiriou. 2017. “Mathematics for Inference and Machine Learning.” Dept. Comput., Imperial College London, London, UK, Tech. Rep., Accessed on Jul.
Diaconis, and Ylvisaker. 1979. The Annals of Statistics.
Domingos. 2020. arXiv:2012.00152 [Cs, Stat].
Fink. 1997.
Gabry, Simpson, Vehtari, et al. 2019. Journal of the Royal Statistical Society: Series A (Statistics in Society).
Gelman. 2006. Bayesian Analysis.
Gelman, Carlin, Stern, et al. 2013. Bayesian Data Analysis. Chapman & Hall/CRC texts in statistical science.
Gelman, Hill, and Vehtari. 2021. Regression and other stories.
Gelman, and Nolan. 2017. Teaching Statistics: A Bag of Tricks.
Gelman, and Rubin. 1995. Sociological Methodology.
Gelman, and Shalizi. 2013. British Journal of Mathematical and Statistical Psychology.
Gelman, and Yao. 2021. Journal of Physics G: Nuclear and Particle Physics.
Goodman, Mansinghka, Roy, et al. 2012. arXiv:1206.3255.
Goodrich, Gelman, Hoffman, et al. 2017. Journal of Statistical Software.
Howard. 1970. Proceedings of the IEEE.
Hubbard. 2014. How to Measure Anything: Finding the Value of Intangibles in Business.
Khan, and Rue. 2023.
Li, and Dunson. 2016. arXiv:1611.01241 [Stat].
Mackay. 1995. Network: Computation in Neural Systems.
MacKay. 1999. Neural Computation.
Ma, Kording, and Goldreich. 2022. Bayesian Models of Perception and Action.
Mandt, Hoffman, and Blei. 2017. JMLR.
Martin, Frazier, and Robert. 2020. arXiv:2004.06425 [Stat].
Nguyen, Low, and Jaillet. 2020. In Advances in Neural Information Processing Systems.
O’Hagan. 2010. Kendall’s Advanced Theory of Statistics: Bayesian Inference. Volume 2B.
Raftery. 1995. Sociological Methodology.
Robert. 2007. The Bayesian choice: from decision-theoretic foundations to computational implementation. Springer texts in statistics.
Schervish. 2012. Theory of Statistics. Springer Series in Statistics.
Stuart. 2010. Acta Numerica.
van de Schoot, Depaoli, King, et al. 2021. Nature Reviews Methods Primers.
van der Linden, and Chryst. 2017. Frontiers in Applied Mathematics and Statistics.
Zellner. 1988. The American Statistician.
———. 2002. Journal of Econometrics, Information and Entropy Econometrics,.