Calibration of probabilistic forecasts

Intuitively speaking, we need to ensure that if our prediction is 80% certain, we are wrong as close to 20% of the time as possible. The same applies to all other certainties.

Placeholder.

I do not know much about this, but I could probably start from the compact lit review in Gneiting and Raftery (2007), or chapter 2 of Neyman (2024) which generalises from calibration to all sort of interesting topics in Bayesian epistemics. The same scoring-rule primitive also underwrites the loss-as-voting framing in AI alignment to collective values, the truth-from-strategic-agents formalisms in learning from the madness of crowds, and the futarchy / prediction-market line in utopian governance.

The idea is that we choose a rule for paying an expert which, assuming no side payments, incentivizes that expert to report their “best” prediction of the truth.

Suppose I ask my expert to predict an outcome \(y\) (e.g. the result of a future event) and they report their prediction in the form of a probability distribution \(p(y)\) over possible outcomes. If a verifiable \(y\) eventually arrives, we can evaluate a \(p\) with a strictly proper scoring rule \(S(p,y)\) so that accurate predictions maximize expected score (Gneiting and Raftery 2007). Formally, for any belief \(q\) about \(y\), \[ \mathbb{E}_{y\sim q}[S(q,y)] \;\ge\; \mathbb{E}_{y\sim q}[S(p,y)] \] with equality iff \(p=q\). Now suppose we pay the expert \(\$S(p,y)\) for their prediction. It should be plausible that they will maximize expected payment by attempting to report as close as possible to the optimal \(p=q\). Classic examples include the log score (\(-\log p(y)\)) and the Brier score (Gneiting and Raftery 2007).