Delegation and coordination using AI
I’ll give utopia when you take it from my cold, dead, invisible hands
2025-10-27 — 2026-06-29
Wherein the Coasean Singularity Is Examined as a Possible Consequence of Deploying AI Agents as Personal Fiduciaries, and the Merging of Economic Negotiation With Democratic Consensus-Building Is Considered.
What if we outsource negotiations at scale? Could Hayek’s dream of distributed information flows through the economy be put into practice in a more humane way using AI agents?
This is the all-in version of civic tech: we lean into massive-scale management by agents on our behalf. The civic-tech tools keep humans as the deliberators and use AI to mediate, bridge, or aggregate; here we contemplate handing the negotiating itself to AI agents acting as personal fiduciaries. Both branches inherit the same downstream problems — adversarial manipulation, alignment of agent to principal, separating belief from preference from affective signal (whatever any of those mean) — and the institutional-design trade-offs surveyed in epistemic communities. Learning from the madness of crowds covers the formal multi-agent question of how a learner can reliably reconstruct what a population of strategic agents actually believes — a problem this notebook’s fiduciary agents must solve internally, against a population of one.
I have many thoughts about the risks and opportunities here. For now, just a placeholder.
1 Coasean Singularity
Seb Krier argues in Coasean Bargaining at Scale that a Coasean Singularity is arriving.
[…] consider AGI deployed as a vast ecology of personalized agents and systems. This emerging ecosystem is what Tomašev et al. (2025) characterize as the “virtual agent economy” a new economic layer where agents transact and coordinate at scales and speeds beyond direct human oversight. While this ecology will contain countless specialized agents, let’s focus on the one that matters most from an individual’s perspective: your personal advocate. Think of it as a fiduciary extension of yourself: a tireless, extremely competent digital representative, closely tied to you, its principal.
What could such an agent do? In principle, it can negotiate, calculate, compare, coordinate, verify, monitor, and much more in a split second. Through many multi-turn conversations, tweaking knobs and sliders, and continuous learning, it could also develop an increasingly sophisticated (though never perfect) model of who you are, your preferences, personal circumstances, values, resources, and more. This should evolve over time—an agent’s alignment should follow the principal’s own evolution. Recent research (Goyal, Chang, and Terry 2024) on negotiation agents finds that “human-agent alignment” is profoundly personal. Users expect agents to not only execute goals but also embody their identity, requiring alignment on everything from preferred negotiation tactics to personal ethical boundaries and the specific public reputation they wanted to project. There are of course important privacy considerations here, but none of these seem fundamentally intractable. For example these systems could be built on technologies like zero-knowledge proofs and differential privacy, ensuring that preferences are communicated and aggregated without revealing sensitive underlying data.
See also Shahidi et al. (2025).
The argument is that a sufficiently advanced economic negotiation might be indistinguishable from democratic consensus-building, and that the economic and political spheres might merge in a Coasean Singularity—or at least that this is a possible future.
Building sufficiently advanced negotiators is left as an exercise.
2 “Meaning economy”
A slightly more galaxy-brained version: price signals and contracts are the medium of communication, and the goal is to coordinate economic activity in such a way that it maximises something richer than the classical revealed preferences of neoclassical economics.
A related concept from Joe Edelman: markets aligned with “deep human values”: Market Intermediaries.
3 Thick models of value
I feel like the thick models of value concept might need to be pried apart from the specific institutional design of market intermediaries, but it’s worth noting that the two are related. See thick models of value.
4 Incoming
- Baumol’s Sawdust - by Oliver Klingefjord
- Design Sketches for a More Sensible AI Future
- Timour Kosters, AI Agents as Coordination Technology
- Gwern’s deep dive into preference-cloning agents— Guardian Angels: LLM Personalization for Productivity and Security (includes the continual learning problem, and the alignment problem in the context of personal agents)
- Comprehensive AI Services - EA Forum
- Updating Drexler’s CAIS model — AI Alignment Forum
- AI • Objectives • Institute