Bayesians vs frequentists

Just because we both get the same answer doesn’t mean neither of us is wrong


Sundry schools thought in how to stitch mathematics to the world, brief notes and questions thereto. Justin Domke wrote a Dummy guide to risk and decision theory which explains the different assumptions underlying each methodology from the risk and decision theory angle.

Bayesian statistics is controversial amongst some frequentists, and vice versa. Sometimes this is for purely terminological reasons, and sometimes for profound philosophical ones. I am not particular invested in this dispute myself.

Avoiding the whole damn issue

You are a frequentist and want to use a Bayesian estimator because it’s tractable and simple? No problem. Discuss prior beliefs in terms of something other than probability, use the Bayesian formalism, then produce a frequentist justification.

Now everyone is happy, apart from you, because you had to miss your family’s weekend in the countryside, and cannot remember the name of your new niece.

This is the best option; just be clear about which guarantees your method of choice will give you. There is a diversity of such guarantees across different fields of statistics, and no free lunches. You know, just like you’d expect.

Frequentist vs Bayesian acrimony

Would I prefer to spend time in an interminable and, to outsiders, useless debate? Is there someone I wish to irritate at the next faculty meeting?

Well then, why not try to use your current data set as a case study to answer the following questions:

Can I recycle Bayes belief updating formalism as a measures of certainty for a hypothesis, or not? Which bizarre edge case can I demonstrate by assuming I can? Or by assuming I can’t? Can I straw-man the “other side” into sounding like idiots?

If I can phrase an estimator in terms of Bayesian belief updates, does it mean that anyone who doesn’t phrase an estimator in terms of Bayesian belief updates is doing it wrong and I need tell them so? If someone produces a perfectly good estimator by belief updating, do I regard it as broken if it uses the language of probabilities to describe belief, even when it still satisfies frequentist desiderata such as admissibility? If I can find a Bayesian rationale for a given frequentist method – say, regularisation – does it mean that what the frequentist is “really” doing is the Bayesian thing I just rationalised, but they are ignorant for not describing it in terms of priors?

That should give me some controversies. Now, I can weigh in!

Here is a sampling of expert opinions probably more expert than mine:

  • (Jaynes and Bretthorst 2003)

    More or less, claims “Baysian statistical practice IS science”. Makes frequentists angry.

  • Deborah Mayo as a philosopher of science and especiallly of the practice of frequentism, has more than you could possibly wish to know about the details of statistical practice, as well as rhetorical dissection of the F-vs-B debate, and says BTW that “Baysian statistics ARE NOT science”. Makes Bayesians angry.

  • Larry Wasserman: Freedman’s neglected theorem

    In this post I want to review an interesting result by David Freedman […]

    The result gets very little attention. Most researchers in statistics and machine learning seem to be unaware of the result. The result says that, “almost all” Bayesian prior distributions yield inconsistent posteriors, in a sense we’ll make precise below. The math is uncontroversial but, as you might imagine, the interpretation of the result is likely to be controversial.

    […] as Freedman says in his paper:

    “ … it is easy to prove that for essentially any pair of Bayesians, each thinks the other is crazy.”

    Is that theorem one of these? (Diaconis and Freedman 1986; Freedman 1999)

  • (Gelman 2011)

    I am told I should look at Andrew Gelman’s model of Bayesian methodology, which is supposed to be reasonable even to frequentists (‘I always feel that people who like Gelman would prefer to have no Bayes at all.’)

  • (Shalizi 2004, 2009)

    Mathematical invective from Shalizi, showing that stubbornly applying Bayesian methods to a sufficiently un-cooperative problem with a sufficiently bad model is effectively producing a replicator system. Which is to say, the failure modes are interesting. (Question: Is this behaviour much worse than in a mis-specified dependent frequentist parametric model? I should read it and find out.)

    Chatty summary here.

  • (Sims 2010)

    Sims has a Nobel Memorial Prize, so he gets to speak on behalf of Bayesian econometrics I guess.

  • nostalgebraist grumps about strong Bayes as a methodology for science. Key point: you do not have all the possible models, and you do not have the computational resource to force them to be consistent if you did have them.

Bacchus, F, H E Kyburg, and M Thalos. 1990. “Against Conditionalization.” Synthese 85 (3): 475–506.

Bernardo, Jose M, and Universitat de Valencia. 2006. “A Bayesian Mathematical Statistics Primer,” 6.

Bernardo, José M., and Adrian F. M. Smith. 2000. Bayesian Theory. 1 edition. Chichester: Wiley.

Diaconis, Persi, and David Freedman. 1986. “On the Consistency of Bayes Estimates.” The Annals of Statistics 14 (1): 1–26.

Freedman, David. 1999. “Wald Lecture: On the Bernstein-von Mises Theorem with Infinite-Dimensional Parameters.” The Annals of Statistics 27 (4): 1119–41. https://doi.org/10.1214/aos/1017938917.

Gelman, Andrew. 2011. “Induction and Deduction in Bayesian Data Analysis.” Rationality, Markets and Morals 2 (67-78). http://www.stat.columbia.edu/~gelman/research/unpublished/philosophy_online4.pdf.

Gelman, Andrew, and Cosma Rohilla Shalizi. 2013. “Philosophy and the Practice of Bayesian Statistics.” British Journal of Mathematical and Statistical Psychology 66 (1): 8–38. https://doi.org/10.1111/j.2044-8317.2011.02037.x.

Jaynes, Edwin Thompson. 1963. “Information Theory and Statistical Mechanics.” In Statistical Physics. Vol. 3. Brandeis University Summer Institute Lectures in Theoretical Physics.

Jaynes, Edwin Thompson, and G Larry Bretthorst. 2003. Probability Theory: The Logic of Science. Cambridge, UK; New York, NY: Cambridge University Press.

Mayo, D G, and A Spanos. 2011. “Error Statistics.” Philosophy of Statistics 7: 153.

Shalizi, Cosma Rohilla. 2004. “The Backwards Arrow of Time of the Coherently Bayesian Statistical Mechanic.”

———. 2009. “Dynamics of Bayesian Updating with Dependent Data and Misspecified Models.” Electronic Journal of Statistics 3: 1039–74. https://doi.org/10.1214/09-EJS485.

Sims, C. 2010. “Understanding Non-Bayesians.” Unpublished Chapter, Department of Economics, Princeton University. http://sims.princeton.edu/yftp/UndrstndgNnBsns/GewekeBookChpter.pdf.