Learning from the madness of crowds

1.1 Certainty

I like to claim to be a Good Bayesian and thus I never Believe Things with certainty. No, I say, my mind is too well schooled in the arts of Optimal Updating, and I am too steeped in the arts of subjective probability to do any of that certainty business. No, I believe nothing with iron-clad certainty; rather I hold all hypotheses in mind, weighted by my current prior probability. Except, of course, the mechanisms of Bayesian updating itself.

This is of course bollocks and some things I treat as if they were true. I believe in gravity, and the existence of apples, and so on. I might conceivably be shown, eventually, to be wrong about these things, but I will be totally surprised and will have made no long-shot contingency plans against the absence of gravity or apples; not so much as a small amulet with the face of Newton to bless me in my travails.

TODO: Hyland and Albarracin (2025).

2.1 Crowdsourcing models

Due to Dawid-Skene and descendants.

In the classical setup, there is a latent ground truth label, and \(n\) noisy annotators each report with an unknown confusion matrix. There exist methods (expectation-maximisation) to estimate these matrices; spectral methods (Zhang et al. 2016) can identify annotator quality without ground truth. The limitation for our purposes is that Dawid-Skene-type methods assume all annotators label the same items. Internet authors write about different things. Still, the mathematical machinery — latent truth, heterogeneous noise, identification via redundancy — might be useful.

2.2 Robust statistics

We can estimate the mean of a distribution even when an \(\varepsilon\)-fraction of samples are adversarially corrupted, as long as the clean distribution has bounded moments. This is because, in high dimensions, adversarial corruption distorts the covariance in detectable ways. Specifically, it introduces “spurious” eigenvalues, which we can then filter out. By analogy we might hope that even if \(\varepsilon\) of the internet is adversarially generated, if the “clean” distribution over text had enough structure, we could learn through the noise. The book to read is Diakonikolas & Kane’s Algorithmic High-Dimensional Robust Statistics (Diakonikolas and Kane 2023).

This is an analogy only, once again. We would need other things to make it do useful things. For example, what are the “bounded moments” assumptions in the text setting? What does “adversarial corruption” even mean when the data has sequential structure? 🏗

2.3 Performative prediction

Predictions that change the distribution they predict. Relevant because: internet content is generated in response to existing discourse. An LLM trained on internet text is predicting a reflexive process. The performative prediction framework gives conditions under which retraining converges to a stable fixed point despite this feedback loop. The limitation: performative prediction is about a single predictor affecting a population, not about extracting signal from a population of strategic reporters.

2.4 Multi-agent inverse reinforcement learning

If the corpus consists of demonstrations from agents with heterogeneous (unknown) reward functions, next-token prediction is a kind of multi-agent IRL. We’re trying to infer… what? A shared world model? An aggregated reward function? The reward decomposition question matters: is there a “common world model” component that all agents share (physics, grammar, entity knowledge) plus agent-specific “bias” components (goals, political views, persuasive intent)? If so, can we identify conditions under which the common component is recoverable? Yu, Song, and Ermon (2019) propose adversarial multi-agent IRL but the text-corpus setting hasn’t been addressed, AFAIK. 🏗

2.5 Data Shapley and data valuation

(Ghorbani2019Data?) (“Data Shapley”) and descendants. Which data points contribute what to model capabilities? This connects to “what are the votes for.” Recent work applies Shapley values specifically to LLM fine-tuning data. The conceptual contribution: treating data points as players in a cooperative game, where the “value” of each is its marginal contribution to model performance. But this is retrospective (evaluating existing data) rather than generative (telling us how the learner extracts signal).

2.6 The “internet as latent mixture model”

More carefully: model each document as generated by first sampling a “type” (topic \(\times\) intent \(\times\) competence \(\times\) honesty), then generating text conditionally. The LLM learns the full joint. At inference time, conditioning on a well-crafted prompt (e.g. “a careful, accurate, well-sourced explanation of X”) selects from the subpopulation of types that would have produced such a prompt, which (if the type space is rich enough) filters out the biased types. This is just Bayesian conditioning in the latent type space. It explains why instruction-tuning and RLHF work at all: they’re shifting the posterior over types toward “helpful, honest, harmless.” But what determines whether the latent type space has enough resolution to make this filtering effective? 🏗

2.7 Free energy and active inference multi-agent models.

Hyland et al. (2024) proposes modelling interactions between boundedly-rational agents as free-energy equilibria. Walters et al. (2025) applies free-energy risk metrics to multi-agent systems. The active-inference framing says each agent minimises its own free energy (surprise), and the question becomes: under what conditions does a collection of surprise-minimising agents produce an aggregate that is itself low-surprise in a useful sense? Crutchfield and Jurgens (2025) on “agentic information theory” might be relevant here — intrinsic semantics of information processes. This feels potentially deep but I haven’t worked through the maths. 🏗

2.8 Differentiable social choice

An and Du (2026) surveys this directly; the implications for ML training pipelines are developed in AI alignment to collective values. If we can frame the aggregation as a differentiable function of the inputs, we can learn the mechanism itself. Fish et al. (2025) on “generative social choice” is also relevant — using generative models as the output of a social choice process (producing text that represents a group’s position).

4.1 Key positive and negative results

Negative:

• Speed of learning in network equilibria is bounded by a constant depending only on private signal distributions, independent of network size (Huang, Strack, and Tamuz 2024). As networks grow, almost all private information is lost. Naive information sharing fails even at scale. This is a strong constraint on any “just let agents talk” approach.
• Arrow/Gibbard-Satterthwaite: aggregation mechanisms are either dictatorial or manipulable.
• Information aggregation under ambiguity (Galanis, Ioannou, and Kotronis 2024) — ambiguity aversion distorts aggregation even in designed mechanisms.

Positive:

• Prediction markets converge to rational expectations under certain conditions. The prices are the aggregation.
• Proper scoring rules (Gneiting and Raftery 2007) incentivise truthful reporting, and connect to Bregman divergences.
• Surprisingly popular (Prelec 2004; Prelec, Seung, and McCoy 2017) extracts minority-correct answers using meta-knowledge.
• Robust statistics can handle \(\varepsilon\)-corruption if the clean distribution has structure.
• Collina et al. (2025): collaborative prediction via iterated agreement, sharing only predictions (not features), with communication cost independent of data dimensionality.

4.2 Formalisms opened up by the prescriptive framing

Prediction markets as the canonical case. The cleanest existing mechanism for extracting beliefs from strategic agents. A proper scoring rule incentivises truthful reporting; a prediction market aggregates via prices. Limitation everyone knows: thin markets, manipulation, subsidisation costs. But the mathematical structure — agents reveal through bets, the mechanism aggregates — is the prescriptive framing in its purest form. The question: can we design something market-like for general knowledge extraction rather than just binary event prediction? Sudhir and Tran-Thanh (2025) on market-based architectures in RL is suggestive. Olckers and Walsh (2024) surveys what goes wrong.

Opponent shaping (LOLA, M-FOS, COLA). Foerster et al.’s “Learning with Opponent-Learning Awareness” and descendants. Instead of best-responding to other agents’ current strategies, model how they’ll adapt to ours, and optimise for the resulting trajectory. If our learner queries/interacts with biased agents repeatedly, it should account for how those agents will respond to being queried. A naive learner that just asks questions gets gamed; an opponent-shaping learner can steer the interaction toward more informative equilibria. The connection holds if we think of the learner as playing a repeated game against information sources with incentives to mislead.

Bayesian persuasion, inverted. Kamenica & Gentzkow’s framework (introduced in Bayesian epistemics as the dual of truth-elicitation) has the sender designing an information structure to influence a receiver. Our learner is the receiver trying to extract information despite the sender’s strategic design. Understanding the sender’s optimal strategy tells us what the worst-case bias structure looks like.¹ This is useful because it gives us the adversary’s optimal play, which is exactly what we need to design against.

Information design meets mechanism design. S. Chen et al. (2023): proper scoring rules meet principal-agent models. Altman and Tennenholtz (2007): incentive-compatible ranking systems. Y. Chen and Yu (2024): scoring rule design under partial knowledge. The prescriptive question: design a protocol where agents’ self-interested behaviour is the information extraction mechanism. Carey et al. (2025) on incentives for responsiveness and instrumental control is relevant too — what incentives does the mechanism create for the agents being queried?

Kalai and Lehrer (1993): rational learning leads to Nash equilibrium. Agents who update beliefs rationally will converge to equilibrium play. The question is whether learning about source reliability converges fast enough to be useful before the information environment shifts.

4.3 The implicit market hypothesis

The internet isn’t a designed mechanism, but it has market-like properties. Agents compete for attention (a kind of currency). If we could design a better attention-allocation mechanism — one that rewards informativeness rather than engagement — we’d have a prediction-market-like structure for general knowledge. This is, IMO, the most buildable version of the prescriptive research programme: take the mathematical structure of information markets (proper scoring rules, market makers, sequential trade) and adapt it to the problem of producing reliable knowledge from distributed agents with heterogeneous incentives.

What makes this hard: in a prediction market, there’s an eventual ground truth (did the event happen?). For general knowledge, the “resolution” mechanism is unclear. Dasgupta and Ghosh (2013) on crowdsourced judgement elicitation with endogenous proficiency and Carvalho (2010) on sharing rewards based on subjective opinions address this gap — mechanisms for eliciting and aggregating beliefs where no resolution event exists.

4.4 What would we build? (prescriptive version)

1. Market-based knowledge aggregation protocol. Design a mechanism where agents submit probabilistic claims and are scored by a rule that doesn’t require ground truth (the peer-prediction family — see Bayesian epistemics — and Miller, Resnick, and Zeckhauser (2005), Witkowski and Parkes (2012)). Study whether the mechanism produces better calibrated beliefs than unstructured information sharing. This is an experiment, not a product — but it has the shape of something that could become a product.

2. Opponent-shaping query strategies. Implement a learner that models source incentives and adapts its queries to extract maximum information. Compare against naive querying on a simulated population of strategic agents with known bias structures. The LOLA/COLA machinery provides the gradient-based update rule; the question is whether it helps in an information-extraction setting rather than the usual cooperation/competition settings.

3. Robust aggregation with reputation. Combine robust mean estimation (Diakonikolas and Kane 2023) with an online reputation system (bandit-like credibility tracking). Each round: query sources, aggregate robustly, update reputations. Study convergence rates as a function of corruption fraction and reputation learning speed.

4. Information market for LLM alignment. Instead of RLHF’s Borda-ish aggregation, implement a market mechanism where annotators “bet” on which response is better, with payoffs determined by peer prediction. Compare alignment quality to standard RLHF. This connects back to Analogy A above but with the prescriptive mechanism-design lens.

4.5 Garrabrant’s logical induction as a prototype

Garrabrant et al. (2020) proposes: treat logical truths as things we have credences over, and let a market of “traders” (each polynomial-time computable) bid on logical sentences. A sentence’s stock is worth $1 if true, $0 otherwise. The logical induction criterion: no poly-time trading strategy with finite risk tolerance earns unbounded profit. A computable algorithm satisfying this criterion exists, and its prices converge to truth, become coherent (obey probability axioms), and learn to predict patterns of truth “often long before having the resources to evaluate the statements, so long as the patterns can be written down in polynomial time.”

The load-bearing insight: different traders specialise in detecting different patterns. The market aggregates their heterogeneous computational capabilities through prices. This is a division of computational labour, not just informational labour. The ensemble can collectively track truths that no individual trader can verify.

4.6 Simon’s docility as the demand side

Herbert Simon’s concept of docility (from Administrative Behavior (Herbert A. Simon 1997) and later works (Herbert A. Simon 1990)): agents who are computationally bounded rationally accept influence from their social environment because figuring everything out from scratch is too expensive. Docility is adaptive under bounded compute — importing a peer’s conclusion is cheaper than re-deriving it. The failure mode: docility makes agents manipulable. The design question: when is it rational to be docile, and toward whom?

4.7 The allocation problem

Smashing these together: agents are bounded in both information (they see different parts of the world) and compute (they can verify different logical consequences). Each agent has comparative advantage in certain inferences. The question: when should I reason for myself and when should I import a peer’s conclusion?

This is structurally an explore-exploit tradeoff over epistemic actions:

• “Explore” = do my own reasoning/observation (costly, first-hand, reliable)
• “Exploit” = adopt a peer’s conclusion (cheap, second-hand, uncertain reliability)

The optimal policy depends on uncertainty about the peer’s reliability, the cost of verification, and the value of getting the right answer. But it’s richer than a standard bandit because:

• Verifying one claim gives information about the peer’s reliability on other claims (correlated types)
• The peer’s conclusions aren’t independent of ours — if we publish our conclusions, the peer updates (reflexivity)
• The verification itself consumes compute that could have been spent on other claims (opportunity cost in a shared budget)

4.8 Related formalisms

Rational inattention (Sims 2003; Matějka and McKay 2015). Agents have finite information-processing capacity (measured in bits/mutual information) and must choose what to attend to. Optimal attention allocation depends on the payoff structure. This captures the “limited compute” side — the agent can’t process all available information. Standard rational inattention doesn’t model the social aspect (importing conclusions from peers), but the mathematical framework (channel capacity as a constraint on inference) is right.

Division of cognitive labour (Weisberg and Muldoon 2009; Zollman 2010; Thoma 2015). Agent-based models of scientific communities exploring “epistemic landscapes.” Key results: cognitive diversity gives epistemic communities an advantage (Weisberg and Muldoon 2009), but the right goal is transient diversity — explore different hypotheses initially, then converge (Zollman 2010). Too much communication can cause premature convergence; too little wastes information. Network structure matters. Muldoon (2013) surveys the formal models.

Zollman’s transient-diversity result (Zollman 2010) is particularly interesting here: division of cognitive labour can be maintained either by limiting information or by endowing agents with extreme beliefs, but if both are present, agents fail to converge. So we want a mechanism that supports diversity of investigation while still allowing convergence of belief — which is exactly what a well-designed market does.

Proof markets (speculative extension of Garrabrant). Imagine making Garrabrant’s framework strategic: traders have incentives to mislead the market (not just honest computational limitations). Can the logical induction criterion be maintained under adversarial traders? This would combine Garrabrant’s computational division of labour with the adversarial robustness of prediction markets. I don’t know of anyone who has done this. 🏗

4.9 The synthesis I think is missing

No existing formalism simultaneously models agents who are:

1. Bounded in compute (can verify only some claims)
2. Bounded in information (see only part of the world)
3. Strategic (may benefit from misleading others)
4. Embedded in a network (hear from some peers, not all)
5. Making allocation decisions (how much effort to verify vs. import)

Garrabrant handles 1 and 4 but assumes honest traders. Prediction markets handle 3 but assume unbounded compute and a single resolved event. Rational inattention handles 1-2 but assumes a single agent. Division-of-cognitive-labour models handle 1-2 and 4-5 but typically assume non-strategic agents.

The closest compound might be: a Garrabrant-style market with strategic traders, partial information, and a network structure — where the “logical sentences” are replaced by empirical claims about the world, and traders can both observe and compute, with both being costly.

4.10 What would we build? (bounded-compute version)

5. Verification allocation game. Design a simulated environment: \(n\) agents, each can observe \(k\) facts and verify \(m\) inferences per round. Agents can also read peers’ published conclusions at low cost. Study: what allocation strategies (how much to observe vs. verify vs. import) lead to best aggregate knowledge? What mechanisms (market prices, reputation scores, peer prediction) best incentivise efficient allocation?

6. Strategic logical induction. Extend Garrabrant’s framework with strategic traders who can profit from misleading the market. Study whether the logical induction criterion can be approximately maintained, or whether adversarial traders can permanently distort prices. The key parameter: what fraction of adversarial traders can the market tolerate?

7. Docility-as-bandit. Model each agent’s trust in each peer as a bandit arm. Pulling the arm = importing the peer’s conclusion. Reward = whether the conclusion turns out to be consistent/useful (a proxy for truth, since ground truth is unavailable). Study convergence of trust estimates and aggregate belief quality. Compare against: always verify (too expensive), always import from majority (vulnerable to cascades), import from most-consistent-with-own-observations (computationally cheap consistency check).