Bayes inference in an open world
Realizability, infrabayesianism, M-open, M-closed, mis-specification
2016-05-30 — 2026-01-14
Wherein misspecified Bayes is treated as M-open, and predictive mixtures are formed by LOO cross-validation with PSIS, while likelihoods are tempered by η to restrain overconfidence.
It turns out all models are wrong. We’re used to that, but we rarely account for it in Bayesian inference. If my model is “wrong” in the sense that the ground truth isn’t in the hypothesis class, can I treat it “as if” it were correct and still recover something like “nearly good” inference? Does it really matter if I didn’t account for my model being misspecified?
The answer: it depends. M-open is a term used to describe different relationships between our hypothesis class and reality, and to help us reason about how bad it is that our simplifications aren’t perfect. Infrabayesianism is a set of strategies for doing principled reasoning with oversimplified models.
Here are my notes on this.
For now, consider Christian Robert’s brief intro:
A few thoughts (and many links to my blog entries!) about that meme that all models are wrong:
- While the hypothetical model is indeed almost invariably and irremediably wrong, it still makes sense to act in an efficient or coherent manner with respect to this model if this is the best one can do. The resulting inference produces an evaluation of the formal model that is the “closest” to the actual data-generating model (if any);
- There exist Bayesian approaches that can do without the model, a most recent example being the papers by Bissiri et al. (with my comments) and by Watson and Holmes (which I discussed with Judith Rousseau);
- In a connected way, there exists a whole branch of Bayesian statistics dealing with M-open inference;
- And yet another direction I like a lot is the SafeBayes approach of Peter Grünwald, who takes into account model misspecification to replace the likelihood with a downgraded version expressed as a power of the original likelihood.
- The very recent Read Paper by Gelman and Hennig addresses this issue, albeit in a convoluted manner (and I added some comments on my blog). I presume you could gather material for a discussion from the entries about your question.
- In a sense, Bayesians should be the least concerned among statisticians and modellers about this aspect since the sampling model is to be taken as one of several prior assumptions and the outcome is conditional or relative to all those prior assumptions.
1 M-open Bayes
Fancy folks write M-open as \(\mathcal{M}\)-open, but life’s too short for fancy typography.
Le and Clarke (2017) summarises:
For the sake of completeness, we recall that Bernardo and Smith (Bernardo and Smith 2000) define M-closed problems as those for which a true model can be identified and written down but is one amongst finitely many models from which an analyst has to choose. By contrast, M-complete problems are those in which a true model (sometimes called a belief model) exists but is inaccessible in the sense that even though it can be conceptualised it cannot be written down or at least cannot be used directly. Effectively this means that other surrogate models must be identified and used for inferential purposes. M-open problems according to Bernardo and Smith (2000) are those problems where a true model exists but cannot be specified at all.
They also mention Clyde and Iversen (2013) as a useful resource.
My understanding is as follows: In statistical modelling, we often operate under a convenient fiction: that somewhere within our set of candidate models lies the “true” process that generated our data. This is known as the M-closed setting. There’s also M-complete in the mix, but it doesn’t seem to be a popular category in practice, so I won’t disambiguate it here. But what happens when we acknowledge that our models are, at best, useful approximations of a reality far more complex than they can capture?
This brings us to the M-open setting, where we accept that the true data-generating process is fundamentally outside our model class. This is, of course, the state of the world for most complex, real-world systems, because the map is not the territory, which is surprisingly not always acknowledged.
If no model is “true,” the goal of inference is no longer to identify that true model. Instead, we focus on predictive performance and robust decision-making under (unavoidable) model misspecification. The archetypal question changes from “Which model is right?” to “Which model is most useful, and how can we mitigate the risks of it being wrong?”
Related: likelihood principle, decision-theory, black swans, …
Bayesians have developed a pragmatic set of tools for navigating the M-open world, emphasizing practical performance over theoretical purity. The next few sections explore popular alternatives.
2 Ignore misspecification
The default, and very popular in practice.
3 Stacking
The most common approach in early M-open applications is to use Bayesian model stacking.
If we can’t trust any single model, why not combine them in such a way that the models make up for each other’s deficiencies? Instead of traditional Bayesian model averaging, M-open practice favours stacking. Stacking uses cross-validation to find the optimal weights for combining multiple models into a single predictive distribution that performs best on out-of-sample data.
Specifically, we use Leave-one-out cross-validation (LOO) and its efficient approximation, PSIS (Pareto smoothed importance sampling). These methods provide a robust estimate of a model’s predictive accuracy, helping practitioners choose and combine models in a way that is explicitly geared for performance in the face of misspecification. The focus is on building a better predictive engine (Le and Clarke 2017), not on finding some imaginary ground truth in the model set.
4 Generalized and Gibbs Posteriors
Standard Bayesian updating can behave poorly when the model is misspecified, sometimes becoming over-confidently wrong as more data comes in. Generalized Bayesian posteriors somewhat address this by “tempering” the likelihood with a learning rate parameter (η). This down-weights the influence of the likelihood, preventing the model from becoming too concentrated on a flawed representation of the world.
This approach, also known as a Gibbs posterior, helps repair some of the statistical inconsistencies that arise under misspecification. Some methods, like SafeBayes (Thomas and Corander 2019), even learn the optimal tempering rate from the data itself, offering a more adaptive way to handle the mismatch between model and reality.
My read on this approach is that it’s a pragmatic robustification procedure, but not terribly theoretically satisfying.
5 Alternative Bayes foundations
Infrabayesianism, maximin expected utility, and other approaches rebuild the foundations of reasoning to handle misspecification from the ground up. See Imprecise Bayesianism.
6 Gibbs posteriors
Relatedly, Gibbs posteriors, if we squint, look like an attempt to address the M-open problem by removing the need for a valid likelihood. AFAICT they always imply a valid likelihood, though, so they don’t directly address the problem.
