Bayesian elicitation
Information elicitation, incentive mechanisms for truth, proper scoring rules…
2025-05-05 — 2026-07-08
Wherein the Design of Incentive Mechanisms for Truthful Reporting Is Examined, With Particular Attention Being Paid to Proper Scoring Rules and Their Connection to Machine-Learning Loss Functions.
One of many topics on which I am not an expert but find it useful to track. How should I design a system to optimize for truthfulness?
Frongillo and Waggoner (2024):
This note provides a survey for the Economics and Computation community of some recent trends in the field of information elicitation. At its core, the field concerns the design of incentives for strategic agents to provide accurate and truthful information. Such incentives are formalized as proper scoring rules, and turn out to be the same object as loss functions in machine-learning settings, providing many connections. More broadly, the field concerns the design of mechanisms to obtain information from groups of agents and aggregate it or use it for decision making. Recently, work on information elicitation has expanded and been connected to online no-regret learning, mechanism design, fair division, and more.
This page is about the intersection of Bayes and mechanism design.
This idea demands a notebook than many to track because the scientific literature in this area is opaque and the popular material is of highly variable quality — e.g. rationalists like to talk off-handedly about prediction markets, but every time I try to chase that concept down I find myself deep in a LessWrong rabbit hole where the low-quality takes crowd out the good ones and I become discouraged. Thus! Here we are.
I am currently reading, and strongly recommend, Chapter 2 of Neyman (2024), which is a well-written introduction to many of the key concepts in this domain. In particular, it explains beautifully the connection between Bregman divergences and proper scoring rules, and incentive mechanisms for truthfulness. Potential applications to epistemic communities, prediction markets etc are immediate. One excellent insight is that we can think about loss functions as scoring rules for groups also, and this idea will be helpful in the design of mechanisms to obtain information from groups of experts.
1 Proper scoring rules as payments
Refresher on calibration: The idea is that we choose a rule for paying an expert which, assuming no side payments, incentivizes that expert to report their best prediction of the truth.
Suppose I ask my expert to predict an outcome \(y\) (e.g. the result of a future event) and they report their prediction in the form of a probability distribution \(p(y)\) over possible outcomes. If we eventually observe \(y\), we can reward a supplied \(p\) with a strictly proper scoring rule \(S(p,y)\) so that accurate predictions maximize expected score (Gneiting and Raftery 2007). Formally, for any belief \(q\) about \(y\), \[ \mathbb{E}_{y\sim q}[S(q,y)] \;\ge\; \mathbb{E}_{y\sim q}[S(p,y)] \] with equality iff \(p=q\). Now suppose we pay the expert \(\$S(p,y)\) for their prediction. It should be plausible that they will maximize expected payment by attempting to report as close as possible to the optimal \(p=q\). Classic examples include the log score (\(-\log p(y)\)) and the Brier score (Gneiting and Raftery 2007).
2 Truth-elicitation
We can get more elaborate: more than one expert, no observable outcome, objectives other than accuracy, interactive communication…
Disclosure: I am especially interested in this for peer review, due to being elbows-deep in the alignment journal.
2.1 The planner’s problem
Suppose I am trying to coax truth out of a group of experts, each of whom has some private information about a state of the world \(\omega\). Because of the connection to mechanism design, I will call myself the planner. The experts are the agents of classic mechanism design.
- Each expert \(i\) privately observes a signal \(s_i\) about the state \(\omega\); I write \(s=(s_1,\dots,s_n)\) for the pooled signals, which I never see.
- The experts send me messages \(m=(m_1,\dots,m_n)\), and I pay expert \(i\) according to a payment rule \(S_i(m)\).
What should success mean here? To a statistician, the obvious target is to get the posterior right: Write \(q(\cdot\mid s)\) for the posterior over \(\omega\) that I would hold if I could observe the pooled signals directly, and \(\hat q(\cdot\mid m)\) for the elicited posterior, the one I actually construct by aggregating the messages (\(q\) as in the refresher; the hat marks an estimate). Belief alignment asks that the two agree,
\[\hat q(\cdot\mid m) \approx q(\cdot\mid s).\]
If \(\omega\) is observable, this would reduce to choosing a proper scoring rule, copy-pasted a few times. Pay each expert a strictly proper score against the realized outcome, and truthful reporting is a best response for each of them separately.
2.2 Elicitation without observation
Now we drop observability. We never observe \(\omega\), so payments can depend only on the messages themselves. With one expert I am stuck — no payment rule can distinguish a truthful report from an invented one. With several experts, though, the signals are correlated through \(\omega\), and I can score each expert’s report against the others’. Different choices of message space and payment rule get us different mechanisms.
- Peer Prediction (PP): pay experts using others’ reports so truthful reporting is a Bayes–Nash equilibrium (Miller, Resnick, and Zeckhauser 2005; Witkowski and Parkes 2012a).
- Bayesian Truth Serum (BTS) and variants: reward reports and meta-predictions about the distribution of others’ reports—truthful reporting is incentivized under mild common-prior/correlation conditions (Prelec 2004; Prelec, Seung, and McCoy 2017).
- Robust BTS / Minimal PP: relax prior/knowledge assumptions or handle non-binary or finite-sample regimes (Witkowski and Parkes 2012b; Radanovic and Faltings 2013; Dasgupta and Ghosh 2013).
TIL about the related surprisingly popular method (Prelec, Seung, and McCoy 2017), which elicits the same meta-predictions as BTS but uses them for aggregation rather than payment: ask people what they think other people will think, then favour answers that are more popular than predicted.
3 Action-oriented elicitation
So much for learning truth. Sometimes we want to learn something else: the best action to take, given the experts’ information. Let \(\mathcal{A}\) be the set of actions available to me; the aggregation/decision map \(\delta\) takes a message profile \(m\) to an action \(\delta(m)\in\mathcal{A}\), doing the work of both summarizing the evidence and acting on it. I now judge success by an objective \(W(a;\omega)\) — the payoff of taking action \(a\) when the state turns out to be \(\omega\) — rather than by \(\hat q\) itself: “I don’t care about the belief itself, just so long as it works”. For any belief \(q\) about \(\omega\), write \(a^\star(q)\in\arg\max_{a\in\mathcal{A}}\mathbb{E}_{\omega\sim q}[W(a;\omega)]\) for the Bayes act, the best available action for someone who believes \(q\). The matching target is \[\delta(m^\star(s)) = a^\star(q(\cdot\mid s)),\]
where \(m^\star(s)\) denotes the equilibrium message profile under the payments. If we attain this, we have decision alignment: the action I take is the one that would be best if I could see the pooled signals directly.
Now, we can recover the truth elicitation case by trivializing: Take the action space \(\mathcal{A}\) to be the set of probability distributions over \(\omega\), so that “acting” just means publishing a forecast \(p\), and take the objective to be a strictly proper scoring rule, \(W(p;\omega)=S(p,\omega)\) — the state \(\omega\) now playing the role the outcome \(y\) played in the refresher. Then, for every belief \(q\), \[a^\star(q)=\arg\max_p\ \mathbb{E}_{\omega\sim q}[S(p,\omega)]=q\] by strict properness — the best forecast to publish is the belief itself — and the decision-alignment condition reads \(\hat q(\cdot\mid m^\star(s))=q(\cdot\mid s)\): belief alignment again (Savage 1971)!
This is also the universal case — the posterior is a sufficient statistic for every downstream decision problem, so belief alignment delivers every decision alignment by post-processing.
3.1 Coarsening the target
A general \(W\) changes the picture in two ways: the target coarsens, which helps me, and the action determines which evidence I ever see, which hinders me
Keep the outcome observable for a moment. If the realized welfare \(W(a;\omega)\) is observable and transferable, I can pay a single expert the objective itself; recommending their own Bayes act then maximizes their expected payment, so \(W\) acts as a proper scoring rule for the report “which action is best” (Y. Chen et al. 2014).
Note that I no longer need the whole posterior, only the statistic of it that picks the action. The Bayes act \(a^\star(q)\) is a property of the belief \(q\) in the sense of Lambert, Pennock, and Shoham (2008) — a functional of the distribution, like a mean or a quantile. Some properties admit direct elicitation via a scoring rule on some reduced report (means, quantiles, expectiles); others do not, and force us to elicit more of the distribution than the decision ultimately uses (survey in Frongillo and Waggoner (2024) on elicitability and elicitation complexity.) So the action target may be coarser than the belief target: cheaper to communicate, sometimes cheaper to score.
3.2 Decision markets
The second change works against me. I ask the experts “what happens if we do \(a\)?” for each candidate action, choose \(a=\delta(m)\), and observe only the outcome of the action I took; the counterfactuals never materialize, so I can never score them. That censoring reopens the manipulation problem: my decision depends on the reports, so an expert can shade their report to steer me toward the action where they expect to score well, or away from one that would expose their ignorance.
Surprisingly-to-me, deterministic follow-the-advice rules are not incentive compatible even when the downstream scoring rule is strictly proper (Othman and Sandholm 2010). The fix is randomization: commit to a decision rule with full support over actions, and reweight the realized score by the inverse probability of the action taken; strictly proper decision markets exist when the decision rule has full support (Y. Chen et al. 2014). The price is that I must sometimes take actions my own elicited beliefs rank as inferior, to keep every conditional forecast scoreable.
This problem arises in futarchy and conditional prediction markets.
3.3 Relaxations and metrics
Exact alignment is a lot to ask; the relaxations quantify the shortfall.
- ε-CIC (elicitation slack): bound the maximum gain from misreporting under the payment rule by \(\varepsilon\) (in utility units).
- Decision regret: the welfare gap \(\mathbb{E}[W(a^\star(q(\cdot\mid s));\omega) - W(\delta(m);\omega)]\), for whichever \(W\) I care about.
- Posterior divergence: a probability distance between \(\hat q(\cdot\mid m)\) and \(q(\cdot\mid s)\) — KL, total variation, Wasserstein.
The \(W=S\) reduction makes the last two (truth, action) dissolve into one. Then, the decision regret becomes \[\mathbb{E}\big[S(q(\cdot\mid s),\omega)-S(\hat q(\cdot\mid m),\omega)\big]=\mathbb{E}\,D_G\big(q(\cdot\mid s),\hat q(\cdot\mid m)\big),\] which is the Bregman divergence of the scoring rule’s convex expected-score potential \(G\), with KL as the log-score case (Gneiting and Raftery 2007). Posterior divergence is decision regret evaluated at the universal decision problem, and the choice of scoring rule is a choice of how to evaluate posterior errors.
3.4 Communication vs persuasion
Not all communication seeks the truth. Bayesian persuasion designs signal structures to move the receiver’s action toward the sender’s objective (Kamenica and Gentzkow 2011). Cheap talk shows when informative communication is impossible or coarse (Crawford and Sobel 1982). TODO