“Generalized” Bayesian inference

We follow the reasoning in Jewson, Smith, and Holmes (2018).

In the M-closed scenario (where the true data-generating process is within the model class), Bayesian updating and maximum likelihood estimation are justified because they minimize the KL divergence between the true distribution and the model.

But! when the models are mis-specified, KL divergence—the mathematical backbone of these methods— is not such a slam dunk. High speed tour follows.

The Kullback-Leibler divergence between distributions P and Q is defined as:

$$D_{KL}(P\parallel Q)=\int p(x)\log \frac{p(x)}{q(x)}dx$$

In the well-specified case (“M-closed”), this divergence is exactly what maximum likelihood estimation minimizes. When we maximize the likelihood:

$${\hat{\theta }}_{MLE}=\arg \underset{\theta }{max}\sum _{i=1}^n\log p(x_i|\theta )$$

we’re implicitly minimizing $D_{KL}(P_{true}\parallel P_{\theta })$ where $P_{true}$ is the true data-generating distribution. This is why MLE is “optimal”—it finds the model closest to reality in KL terms.

But here’s the problem: what if $P_{true}$ isn’t in our model class at all? This is the M-open world. And, in fact, no model every include the truth; if we ignore that fact and pretend, weird stuff can happen (Shalizi 2009).

White (1982) showed that under misspecification, MLE still converges to something meaningful—the parameter that minimizes KL divergence between the true distribution and our model class:

$${\theta }^{\ast }=\arg \underset{\theta }{min}{D}_{KL}(P_{true}\parallel P_{\theta })$$

This was reassuring… is that reassuring? a small KL discrepancy can still be pretty bad in terms of making terrible decisions.

Bissiri, Holmes, and Walker (2016) asked: why restrict ourselves to KL divergence at all? They developed a general Bayesian framework using arbitrary divergence measures $d(\cdot ,\cdot )$:

$$\pi (\theta |data)\propto \pi (\theta )\exp (-\eta \cdot d(data,model(\theta )))$$

where $\eta$ controls the learning rate. When $d$ is the log-likelihood, we recover standard Bayesian updating. But we can choose other divergences based on what aspects of the data matter most for our specific problem.

TODO: this looks a lot like Zellner’s argument (Zellner 1988) used in Bayes-by-backprop.

Multiple research groups (Ghosh and Basu 2018; Hooker and Vidyashankar 2014) developed “minimum divergence estimation” approaches, minimising other divergences:

• Hellinger distance: $H(P,Q)=\frac{1}{2}\int (\sqrt{p(x)}-\sqrt{q(x)}{)}^{2}dx$
• Total variation: $TV(P,Q)=\frac{1}{2}\int |p(x)-q(x)|dx$
• α-divergences: $D_{\alpha }(P\parallel Q)=\frac{1}{\alpha (\alpha -1)}\int p(x)[{\left(\frac{p(x)}{q(x)}\right)}^{\alpha -1}-1]dx$

Each targets different features of the data and offers different robustness properties under misspecification.

Nothing dominates (although actually optimal transport is a strong contender), and the main insight is that the choice of divergence should match your inferential or decision-theoretical goals.

So, how do we practically do inference that minimizes these divergences?