Wisdom from the madness of crowds
God help me I need to extract truth from the internet
2026-04-19 — 2026-06-15
Wherein the Surprisingly Popular Algorithm Is Examined as a Model for Extracting Minority-Correct Beliefs From Biased Informants, Alongside Market-Based Mechanisms as Alternatives to Passive Corpus Learning.
Starting questions for better formulations of social belief.
1 Why do I believe things?
2 Certainty
I like to claim to be a Good Bayesian and thus I never Believe Things with absolute certainty. No, I say, my refined intellect is too well schooled in the arts of Optimal Updating, and too steeped in the arts of subjective probability to do any of that sordid absolute business. Rather, the certainty with which I believe anything can never be said to be absolute. I virtuously hold all hypotheses in mind, weighted by my current prior probability— with the sole and righteous exception of the mechanisms of Bayesian updating itself, of course. They are 100% true.
You will ofc realise this is bollocks. Obviously some things I treat as if they were true. I believe in gravity, and the existence of apples, and so on. I might conceivably be shown, eventually, to be wrong about these things, but I will be totally surprised and will have made no long-shot contingency plans against the absence of gravity or apples; not so much as a small amulet with the face of Newton to bless me.
Indeed, full Bayesianism is impossible (and worse than that, plain difficult), for a bounded agent like me.
TODO: Hyland and Albarracin (2025) and Herbert A. Simon (1990).
3 Slop lit review
What follows is a lit review of formalisms that impinge upon aspects of this problem. An LLM seeded it but I have now deleted some of the most glaring insanity. Proceed with care
It is organised by how directly they address the question: how can a learner extract a good world model from data generated by strategic, biased, or adversarial agents? The learner isn’t just filtering noise — the learner’s actions (queries, publications, bets) change what other agents produce. Further, the learner can design its interactions to extract more information.
3.1 Variational belief strategies for bounded agents
In Bayes lingo variational means approximate in some principled way. I am very sorry about this stupid terminology. It is too late now.
3.1.1 Docility
Herbert Simon’s concept of docility (Herbert A. Simon 1997; Herbert A. Simon 1990): agents who are computationally bounded ‘rationally’ accept influence from their social environment because figuring everything out from scratch is too expensive. Docility is adaptive under bounded compute — importing a peer’s conclusion is cheaper than re-deriving it.
At the same time, we might imagine that docility makes agents manipulable.
So, when is it rational to be docile, and toward whom?
3.1.2 Rational inattention
(Sims 2003; Matějka and McKay 2015). Agents have finite information-processing capacity (measured in bits) and must choose what to attend to.
This is not directly a “limited compute” model, but rather a “limited data/precision” model — channel capacity as a constraint on inference, but we might imagine they coincide sometimes. Standard rational inattention doesn’t model the social aspect (importing conclusions from peers), but the mathematical framework () is right.
3.2 Crowd meta-knowledge
We can hope to extract truth from crowds by thinking about incentives, Bayes, and meta-knowledge. Key works here are Prelec (2004) and descendants (Witkowski and Parkes 2012; Miller, Resnick, and Zeckhauser 2005). The “surprisingly popular” (SP) algorithm (Prelec 2004; Prelec, Seung, and McCoy 2017) seems like a good start: it extracts truth from crowds by finding answers that are more popular than predicted. Concretely: people who hold the correct-but-minority view often know their view is rare, so they predict lower support for it from everyone else. The SP algorithm uses this gap between actual and predicted popularity as a signal — the gap between first-order beliefs (what I think) and second-order beliefs (what I think others think). Does the structure of the internet corpus contain enough second-order information to support something like this? Blog posts often respond to other positions, explicitly modelling what “they” believe. This meta-discursive structure might be informative in the SP sense — the internet running a massively distributed, ramshackle version of the same trick, finding coherent patterns that show up more than we’d expect if the corpus were just noise (cf. Collina et al. (2025) on collaborative prediction by sharing predictions, not data).
3.3 The “internet as latent mixture model”
Model each document as generated by first sampling a “type” (topic \(\times\) intent \(\times\) competence \(\times\) honesty), then generating text conditionally. The LLM learns the full joint. At inference time, conditioning on a well-crafted prompt (e.g. “a careful, accurate, well-sourced explanation of X”) selects from the subpopulation of types that would have produced such a prompt, which (if the type space is rich enough) filters out the biased types. This is just Bayesian conditioning in the latent type space. It explains why instruction-tuning and RLHF work at all: they’re shifting the posterior over types toward “helpful, honest, harmless.” 🏗 I’m not sure there is anything here per se; The internet as a mixture of truth and lies sounds less profound when you spell it out that way.
3.4 Markets and scoring rules
3.4.1 Prediction-market-alikes
The cleanest mechanisms for prising beliefs out of strategic agents are the designed ones, and the textbook pair is the proper scoring rule and the prediction market. A proper scoring rule (Gneiting and Raftery 2007) pays a reporter so that stating their true probability maximizes expected reward — truthful reporting becomes the dominant strategy. A prediction market goes one step further and aggregates: agents reveal beliefs by betting, and the price does the averaging. This leans into action — we don’t read beliefs off a corpus, we build a mechanism in which self-interested behaviour is the elicitation.
Markets may notoriously fail by being thin, subject to manipulation, and/or require a costly market maker to subsidize liquidity (Olckers and Walsh 2024).
Interestingly, Sudhir and Tran-Thanh (2025) ports market-based structure into reinforcement learning.
3.4.2 Implicit inference markets
The internet has market-like properties. Agents compete for attention, which looks a bit like currency, and the “price” of a claim is how much attention it gets. The most buildable version of the prescriptive programme: take the mathematical structure of information markets (proper scoring rules, market makers, sequential trade) and adapt it into a better attention-allocation mechanism — one that rewards informativeness rather than engagement — for producing reliable knowledge from distributed agents with heterogeneous incentives.
What makes this hard: in a prediction market, there’s an eventual ground truth (did the event happen?). For general knowledge, the “resolution” mechanism is unclear. Dasgupta and Ghosh (2013) on crowdsourced judgement elicitation with endogenous proficiency and Carvalho (2010) on sharing rewards based on subjective opinions address this gap — mechanisms for eliciting and aggregating beliefs where no resolution event exists.
3.4.3 Logical induction
Garrabrant et al. (2020) proposes: treat logical truths as things we have credences over, and let a market of “traders” (each polynomial-time computable) bid on logical sentences. A sentence’s stock is worth $1 if true, $0 otherwise. The logical induction criterion: no poly-time trading strategy with finite risk tolerance earns unbounded profit. A computable algorithm satisfying this criterion exists, and its prices converge to truth, become coherent (obey probability axioms), and learn to predict patterns of truth “often long before having the resources to evaluate the statements, so long as the patterns can be written down in polynomial time.”
Different traders are incentivised to specialise in detecting different patterns. The market aggregates their heterogeneous computational capabilities through prices. This is a division of computational labour, not just informational labour. The ensemble can collectively track truths that no individual trader can verify.
3.4.4 Proof markets
Speculative extension of Garrabrant. Imagine making Garrabrant’s framework strategic: traders have incentives to mislead the market (not just honest computational limitations). Can the logical induction criterion be maintained under adversarial traders? This would combine Garrabrant’s computational division of labour with the adversarial robustness of prediction markets. I don’t know of anyone who has done this. 🏗
3.5 Designing against strategic sources
3.5.1 Bayesian persuasion
Kamenica & Gentzkow’s Bayesian persuasion framework (Kamenica and Gentzkow 2011) has the sender designing an information structure to influence a receiver. Our learner is the receiver trying to extract information despite the sender’s strategic design. Understanding the sender’s optimal strategy tells us what the worst-case bias structure looks like.1
3.5.2 Opponent shaping
“Learning with Opponent-Learning Awareness” and descendants (Foerster et al. 2018; Khan et al. 2024; Lowe et al. 2019; Lu et al. 2022; Duque et al. 2025). Instead of best-responding to other agents’ current strategies, model how they’ll adapt to ours, and optimise for the resulting trajectory. If our learner queries/interacts with biased agents repeatedly, it should account for how those agents will respond to being queried. A naive learner that just asks questions gets gamed; an opponent-shaping learner can steer the interaction toward more informative equilibria. The connection holds if we think of the learner as playing a repeated game against information sources with incentives to mislead.
3.5.3 Information design meets mechanism design
S. Chen et al. (2023): proper scoring rules meet principal-agent models. Altman and Tennenholtz (2007): incentive-compatible ranking systems. Y. Chen and Yu (2024): scoring rule design under partial knowledge. The prescriptive question: design a protocol where agents’ self-interested behaviour is the information extraction mechanism. Carey et al. (2025) asks what incentives the mechanism creates for the agents being queried — responsiveness and instrumental control. If the aggregation can be framed as a differentiable function of the inputs, we can learn the mechanism itself (An and Du 2026); Fish et al. (2025) pushes this to generative social choice, using a generative model as the social-choice output (see AI alignment to collective values).
3.5.4 Performative prediction
Predictions that change the distribution they predict. Relevant because: internet content is generated in response to existing discourse. An LLM trained on internet text is predicting a reflexive process. The performative prediction framework gives conditions under which retraining converges to a stable fixed point despite this feedback loop. The limitation: performative prediction is about a single predictor affecting a population, not about extracting signal from a population of strategic reporters.
3.6 Division of labour and the allocation problem
3.6.1 The allocation problem
Smashing these together: agents are bounded in both information (they see different parts of the world) and compute (they can verify different logical consequences). Each agent has comparative advantage in certain inferences. The question: when should I reason for myself and when should I import a peer’s conclusion?
This is structurally an explore-exploit tradeoff over epistemic actions:
- “Explore” = do my own reasoning/observation (costly, first-hand, reliable)
- “Exploit” = adopt a peer’s conclusion (cheap, second-hand, uncertain reliability)
The optimal policy depends on uncertainty about the peer’s reliability, the cost of verification, and the value of getting the right answer. But it’s richer than a standard bandit because:
- Verifying one claim gives information about the peer’s reliability on other claims (correlated types)
- The peer’s conclusions aren’t independent of ours — if we publish our conclusions, the peer updates (reflexivity)
- The verification itself consumes compute that could have been spent on other claims (opportunity cost in a shared budget)
3.6.2 Division of cognitive labour
(Weisberg and Muldoon 2009; Zollman 2010; Thoma 2015). Agent-based models of scientific communities exploring “epistemic landscapes.” Key results: cognitive diversity gives epistemic communities an advantage (Weisberg and Muldoon 2009), but the right goal is transient diversity — explore different hypotheses initially, then converge (Zollman 2010). Too much communication can cause premature convergence; too little wastes information. Network structure matters. Muldoon (2013) surveys the formal models.
Zollman’s transient-diversity result (Zollman 2010) is particularly interesting here: division of cognitive labour can be maintained either by limiting information or by endowing agents with extreme beliefs, but if both are present, agents fail to converge. So we want a mechanism that supports diversity of investigation while still allowing convergence of belief — which is exactly what a well-designed market does.
3.7 Free energy and active inference in multi-agent models
Hyland et al. (2024) proposes modelling interactions between boundedly-rational agents as free-energy equilibria. Walters et al. (2025) applies free-energy risk metrics to multi-agent systems. The active-inference framing says each agent minimises its own free energy (surprise), and the question becomes: under what conditions does a collection of surprise-minimising agents produce an aggregate that is itself low-surprise in a useful sense? Crutchfield and Jurgens (2025) on “agentic information theory” — intrinsic semantics of information processes — bears on the same question. 🏗
3.8 Hard limits
- Speed of learning in network equilibria is bounded by a constant depending only on private signal distributions, independent of network size (Huang, Strack, and Tamuz 2024). As networks grow, almost all private information is lost. Naive information sharing fails even at scale. This is a strong constraint on any “just let agents talk” approach.
- Arrow/Gibbard-Satterthwaite: aggregation mechanisms are either dictatorial or manipulable.
- Information aggregation under ambiguity (Galanis, Ioannou, and Kotronis 2024) — ambiguity aversion distorts aggregation even in designed mechanisms.
- Robust statistics can handle \(\varepsilon\)-corruption if the clean distribution has structure.
- Rational learning converges to equilibrium play (Kalai and Lehrer 1993), but whether learning about source reliability converges fast enough to be useful before the information environment shifts is unclear.
4 What might we actually build?
If I’m thinking like an ML person about what’s buildable, the following seems tractable:
Empirically test the “compression as debiasing” hypothesis. The conjecture: more compression pressure (larger models) recovers cleaner signal from a biased corpus — networks fit compressible structure before noise (Arpit et al. 2017), and language modelling is compression (Delétang et al. 2024). But Krestnikov (2026) runs essentially this experiment and finds the catch: gradient descent favours the most compressible answer cluster, not truth — recovery scales with model size against random corruption (65→85%) but collapses to chance against a coherent alternative rule system. So compression debiases incompressible noise, not a structured adversary — exactly the strategic-corruption case we care about. Concretely: inject varying fractions of random vs. coherent misinformation into a factual QA corpus, measure recovery as a function of model scale and corruption type.
Latent-type inference. Train a model to infer the “type” of a document’s author (or at least a useful projection of it), then study whether conditioning on the inferred type allows better extraction of factual content. This connects to the Dawid-Skene line but with the Bayesian conditioning approach in the mixture-model view.
Robust aggregation with reputation. Apply robust mean estimation (Diakonikolas and Kane 2023) to the internal representations of an LLM processing different documents about the same topic — if biased documents are “corrupted” relative to the factual ones, robust aggregation should recover the factual representation. Wrap it in an online reputation system (bandit-like credibility tracking): each round, query sources, aggregate robustly, update reputations. Study convergence as a function of corruption fraction and reputation learning speed.
SP-inspired probing. For claims in the training data, estimate both the LLM’s “belief” (probability it assigns to the claim) and its “meta-belief” (what it predicts other sources would say). The surprisingly-popular gap between them (see above) might flag where the LLM has extracted minority-but-correct information.
Market / peer-prediction protocol. Design a mechanism where agents submit probabilistic claims and are scored by a rule that doesn’t need ground truth (the peer-prediction family — see Bayesian epistemics — and Miller, Resnick, and Zeckhauser (2005), Witkowski and Parkes (2012)). Does it produce better-calibrated beliefs than unstructured information sharing? One concrete application: replace RLHF’s Borda-ish aggregation with a market where annotators “bet” on which response is better, payoffs set by peer prediction, and compare alignment quality to standard RLHF.
Opponent-shaping query strategies. Implement a learner that models source incentives and adapts its queries to extract maximum information. Compare against naive querying on a simulated population of strategic agents with known bias structures. The LOLA/COLA machinery provides the gradient-based update rule; the question is whether it helps in an information-extraction setting rather than the usual cooperation/competition settings.
Epistemic-allocation simulation. Build the allocation problem as a simulated environment: \(n\) agents, each observing \(k\) facts and verifying \(m\) inferences per round, able to read peers’ published conclusions cheaply. Model each agent’s trust in each peer as a bandit arm (pulling = importing the peer’s conclusion; reward = whether it turns out consistent/useful, a proxy for truth). Study which allocation strategies — observe vs. verify vs. import — and which mechanisms (market prices, reputation scores, peer prediction) best incentivise efficient allocation. Compare against baselines: always verify (too expensive), always import from majority (vulnerable to cascades), import from most-consistent-with-own-observations (cheap consistency check). (Making the traders strategic turns this into the proof-markets question raised above.)
5 Incoming
Conitzer (2013): social networks, social choice and statistical estimation unified — spun out into a notebook on AI Social choice.
Equilibria Network - Designing New Forms Of Collective Intelligence
-
AI-powered tools and services that help people figure out what’s true (“AI for epistemics”) could matter a lot. As R&D is increasingly automated, AI systems will play a larger role in the process of developing such AI-based epistemic tools. This has important implications. Whoever is willing to devote sufficient compute will be able to build strong versions of the tools, quickly. Eventually, the hard part won’t be building useful systems, but making sure people trust the right ones, and making sure that they are truth-tracking even in domains where that’s hard to verify. We can do some things now to prepare. Incumbency effects mean that shaping the early versions for the better could have persistent benefits. Helping build appetite among socially motivated actors with deep pockets could enable the benefits to come online sooner, and in safer hands. And in some cases, we can identify particular things that seem likely to be bottlenecks later, and work on those directly.
6 References
Footnotes
See also “Robust Bayesian Persuasion” and “Algorithmic Persuasion Through Simulation” for the computational side.↩︎