1 Why iterated game theory does not save us

The Tit-for-Tat insight from iterated games — mutual cooperation sustained by the threat of retaliation in repeated play — depends on retaliation being available. The shadow of the future, in Axelrod’s phrase, has to fall on a player who exists. When the players never meet, the shadow falls on no one.

The same goes for the bargaining-set / class-emergence dynamics from Axtell, Epstein, and Young (2001)’s multi-agent bargaining model: those require iterated negotiation between the two parties to the bargain. With future generations, only one party is present in any given negotiation.

2 Overlapping generations

The standard formal device for getting some purchase on the problem is the Overlapping Generations (OLG) model — Samuelson (1958)’s consumption-loan paper introduces the basic structure, and Diamond (1965) adds capital to the model. Generations are not point-events but cohorts with finite lifespans, and at any moment several cohorts coexist. A grandparent, parent, and child can interact directly. They have overlapping interests with neighbouring cohorts, who have overlapping interests with theirs. This gives a transitive chain of reciprocal interactions linking the present to the distant future, without requiring any single agent to bargain with someone unborn.

OLG is enough to recover some intergenerational cooperation under the right preference structures — altruism toward children, social-norm enforcement, public-pension institutions, family-firm continuity. But it does not give a fairness theorem for distant generations. The cooperation chain attenuates with each link: future-future-future agents are weighted only as much as their immediate ancestors cared about their children’s children’s children, which is generally not very much.

3 Axiomatic intergenerational equity

Most of the formal work on intergenerational fairness is not game theory; it is axiomatic social choice over infinite streams of well-being. Each generation gets a well-being level , and the question is which orderings of infinite streams are consistent with two minimal-seeming axioms:

• Strong Pareto (sensitivity): if one generation is strictly better off and none worse, prefer the new stream.
• Finite Anonymity (equal treatment): permuting finitely many generations leaves the ranking unchanged.

There are various impossibility results — Basu–Mitra (Basu and Mitra 2003), Lauwers, Zame, and others — blocking us from jointly satisfying these axioms (Asheim 2010). There is no numerically representable social welfare function that satisfies both (Diamond–Basu–Mitra), and if we drop numerical representability and just ask for an explicitly definable complete ordering, that fails too (Lauwers–Zame). One of the axioms has to weaken or go.

The literature has taken two paths through this.

Keep both axioms, give up completeness. Incomplete orderings like utilitarian or leximin “overtaking” criteria can compare some pairs of infinite streams but not all. Some intergenerational dilemmas come out unrankable, which is awkward for policy advice but at least flagged rather than fudged.

Keep completeness, give up equal treatment. Discounted utilitarianism — the standard policy-economics workhorse — applies a discount factor each period, weighting future generations less. Chichilnisky (1996) names this a Dictatorship of the Present, since beyond some finite time the future stops mattering for the ranking. The Stern–Nordhaus debate over climate policy is largely a debate over how much of this dictatorship to accept.

A third path tries to weaken the axioms in the right direction rather than throwing one out. Asheim’s Hammond Equity for the Future (HEF) is one such weakening. Standard Hammond equity says any transfer from a better-off generation to a worse-off one improves the social ranking, provided the ordering is preserved. HEF restricts this to one specific case — when the present is better off than an infinite number of future generations, and a present sacrifice would benefit all of them while keeping the present better off than the future. My understanding of the intuition here is that a small present sacrifice is worth it if it benefits an infinite number of unambiguously-worse-off future cohorts, while we should not demand maximal sacrifice from a barely-worse-off present to benefit a barely-worse-off future. Asheim, Mitra, and Tungodden show that HEF combined with weakened Pareto and continuity axioms gives a class of sustainable recursive social welfare functions, orderings that allow present sacrifice when the future is worse off but not when the future is already better off (Asheim, Mitra, and Tungodden 2012).

In productive technological environments — Asheim, Buchholz, and Tungodden’s “immediate productivity” condition — Strong Pareto and Finite Anonymity together imply that only non-decreasing well-being streams are undominated (Asheim, Buchholz, and Tungodden 2001). So sustainability is what the minimal-equity axioms imply when the technology can in fact produce things, not an additional normative add-on but a consequence of equality + sensitivity once productivity is in the picture.

The methodological frame across all of this is Rawls’s reflective equilibrium (Rawls 1971). Criteria for intergenerational equity should be judged on the ethical axioms they build on, and also on their consequences in concrete technological environments. If a criterion says we should sacrifice everything for any-positive-gain-by-the-future, or never sacrifice anything no matter how much the future gains, our intuition that both are wrong is itself evidence to revise the axioms. Picking axioms in the abstract and applying them blindly is bad methodology; we iterate between axioms and downstream implications until both feel right.

I started with Asheim’s Annual Review survey (Asheim 2010).

4 Veto-power formalism

A bargaining-flavoured strand of the axiomatic literature models the game as if future generations could refuse. Suppose each generation has a Rubinstein-style alternating-offers position (Rubinstein 1982) with a credible veto over others’ resource decisions (Manzini 2010). Even when future cohorts cannot in fact participate, asking “what allocation would they accept under bargaining parity?” gives a well-defined Social Welfare Relation that excludes the worst expropriations.

Such a veto is more a normative device than a positive one. It tells us what fairness would require if the unborn could speak; it does not predict what self-interested play will produce.

5 Proxy mechanisms in practice

Closer to political economy: institutions that artificially weight the bargaining power of those representing future interests.

Youth quotas in legislative bodies. Younger voters and legislators have longer remaining horizons and internalise more of the long-term consequences; their interests partially overlap with the unborn. The argument is roughly: youths are the next-best proxy for those who cannot yet vote — see the 2015 Intergenerational Justice Review issue for several treatments.

Future-generations tribunals or commissioners, with explicit review or veto powers over policies that bind the long term. Hungary, Wales, Israel, and others have variants; their bite varies enormously by jurisdiction and political will. See Basson et al. (2025) on the human-rights framing.

Constitutional limits on the actions of a present majority — debt ceilings, environmental amendments, supermajority requirements for long-term commitments. The constitutional framing serves as a proxy commitment device — see commitment — that binds future selves of the polity.

These are all imperfect. Youths are not the unborn; commissioners have their own interests; constitutions can be amended. But they are practical institutions trying to do the work the formal models say is needed.

6 Discounting problems

A deep open question is how I should rationally discount future utility when other people will experience it than me, because I will be dead. A positive discount rate — the empirical norm in cost-benefit analysis — heavily reduces the present-day weight on far-future welfare; this is what allows current generations to rationalise expropriation in policy documents. A zero or near-zero rate produces unbounded numbers (any benefit that compounds forever has infinite present value), forcing modellers to introduce extinction probabilities or other regularisations whose values are themselves contestable.

One area where people have though long and hard about this is climate change. The Stern–Nordhaus debate on climate-change policy is largely a debate about discounting: the formal cost-benefit models produce wildly different recommendations depending on a parameter that game theory itself does not pin down. There is no equilibrium-style answer. Famous treatments such as DeCanio’s Game Theory and Climate Change (DeCanio and Fremstad 2013) frame it as multi-objective optimisation under deep uncertainty, with the discount rate as a normative input rather than a derivable feature.

7 Don’t die

One way to ameliorate intergenerational games is to not have any generations. If we can achieve indefinite lifespans, then the future is no longer a different generation; it’s just a later self. OTOH, humans can also suck at helping out their future selves.

8 Connections

Adjacent to: cooperation (mechanisms generally), coalition games (when “the future generation” is itself heterogeneous), commitment (intertemporal commitment as the dual problem), collective action (climate and commons specifically), game complexity (computing fair allocations in OLG with many cohorts is its own complexity story), and economics of growth (the standard growth models — Solow, Ramsey, Dasgupta–Heal–Solow — are doing intergenerational welfare analysis under a particular axiom set, usually discounted utilitarianism, often without flagging the choice).

Longtermism (the EA / x-risk strand) sits inside this axiomatic space — typically a near-zero discount rate plus expected-total-utility aggregation, which is one specific resolution of the SP + FA tension. Whether longtermists engage with the impossibility results explicitly, or just smuggle in their preferred resolution via expected utility, varies a lot.`

AI successionism — that AI systems should or will succeed humanity, and that this is acceptable or even desirable — fits inside the same frame. The problem is sharpened, though, because the “future generations” might have radically different utility functions and need not be biologically continuous with us at all. I am noodling on this elsewhere; the axiomatic frame is one tool for asking which axioms successionism implicitly endorses or rejects.

9 References

1 It works now

The xcode build kinda worked for ages, but it was pretty noisy and slow. Recent Xode updates (I think 26.3) made it way better. LLM no longer needs to parse xcodebuild log spew to figure out what went wrong. It calls a tool, gets back JSON with the file, line, column, and message of each error, and acts. The same shift previously made gopls, pyright, tsc --watch etc. workable inside agent loops. iOS got there later, partly because Apple, and partly because xcodebuild is famously verbose.¹

2 XcodeBuildMCP

An open-source workhorse MCP with ~80 tools. All of this assumes xcode-select and the underlying CLI tools are behaving themselves on the day. They frequently are not; see macos_hacks for the standing repair manual.

Install:

claude mcp add xcodebuildmcp -- npx -y xcodebuildmcp@latest

It covers structured builds, test runs with per-method results, simulator boot/shutdown/install, real-device deployment, LLDB-driven debugging, and SwiftPM in addition to .xcodeproj/.xcworkspace. All returning tidy JSON.

The repo is at cameroncooke/XcodeBuildMCP. The clearest write-up I’ve found is Blake Crosley’s setup post, which is also where I cribbed most of my mental model.

Even if I never open Xcode-the-IDE, this MCP gets me from “Claude wrote some Swift” to “Claude built it, ran it on the simulator, ran the tests, and saw which ones failed”.

3 Apple’s MCP bridge

Ships with Xcode 26 — no separate install, only an enable step:

claude mcp add xcode -- xcrun mcpbridge

Around 20 tools, all interfacing with a running Xcode GUI IDE File ops in the editor, real-time diagnostics from the live build context, Apple documentation search, a Swift REPL, and SwiftUI Preview capture. The agent gets to see the rendered preview of a view, not just the code that produced it, and can iterate visually. Anthropic’s announcement post has more detail.

This one only works with Xcode running, so it is no use for headless CI. For the kind of work I’m doing — sitting at the laptop, iterating on a small app — having Xcode open is not a hardship.

4 Division of labour

XcodeBuildMCP owns the build–test–debug loop. Apple’s MCP owns IDE-layer awareness — diagnostics, docs, previews. Run both. The author of the setup write-up does, I copied that, and it works.

5 Project CLAUDE.md

A short hint at the top of the project’s CLAUDE.md keeps the agent from rediscovering the toolchain on every turn. Mine looks roughly like:

## XcodeBuildMCP integration
- Build: mcp__xcodebuildmcp__build_sim_name_proj
- Test:  mcp__xcodebuildmcp__test_sim_name_proj
- Platform: iOS, Swift 6, SwiftUI, MVVM + @Observable

Enough for the agent to default to the right tool without me re-hinting it on every prompt.

6 In-IDE Claude

Xcode 26.3+ also ships with a Claude Agent panel built into the IDE, using the same Agent SDK and roughly the same ~20 tools as xcrun mcpbridge. Settings → Intelligence, drop in an Anthropic API key, done.²

I do not use this. I live in the terminal, and Claude Code is where my prompts, my git worktree workflow, and my muscle memory live. Also I spent about 15 seconds looking for the agent panel and didn’t find it, so I left, bored. For somebody whose centre of gravity is Xcode itself, might be worthwhile?

7 Loose ends

A scattered list of things I have not yet figured out, or have figured out and disliked.

• Signing and provisioning. Even with the developer account paid up, the rituals around personal devices, free/paid teams, provisioning profiles, capabilities, and the keychain are exactly as user-hostile as everyone says. I have not yet found an MCP that abstracts this away. I suspect there is no abstraction over it because Apple would rather a human was somewhere in the loop to sign off on their nonsense.

• Sideload vs. dev-install. For my own apps on my own phone, the dev-account install with a 1-year cert is fine. For apps I might want to give to one or two friends, the answer is less clear. AltStore? TestFlight? I have not committed. Probably TestFlight.

• Does SwiftUI Preview capture actually close the loop? In principle, the agent sees the preview and iterates. In practice I am not yet sure how well this scales past a single isolated view. View hierarchies, state, mock data, navigation contexts — all of these complicate the preview. Currently I plan to avoid ever finding this out by just not building anything complicated enough to need it.

• Non-trivial system integrations. Core Data, HealthKit, Shortcuts intents, WidgetKit — the bits I actually want to use, since they are what makes the phone useful as an automation target. The simple-Swift-app-with-a-button pattern is well-trodden in LLM-generated examples. The “talk to HealthKit and surface a daily nudge” pattern… maybe? TODO

1 The good regulator in the big world

The internal model principle, AKA the “good regulator theorem”, says: any agent that does well in an environment must contain a model of that environment.¹ So the agent contains a model.

The big-world hypothesis says: the environment is strictly bigger than the agent — there is no fixed point at which the agent has “solved” the world. (A formal version is the Lewandowski et al. setup below — an agent embedded as a finite automaton inside a universal computer, interacting with a POMDP over a countably infinite state space.)

Together, these suggest that we need to think about lossy compression of the world, by construction. What kind of compression is the agent’s internal model? Which features survive? What is it sufficient for? When does the residual look like noise, and when does it look like the world ambushing us?

Somewhere in here there is a bridge between internal models and bounded rationality. The internal model principle says we need a model. Bounded rationality says we cannot afford the naïve one, the territory-as-map. The technical question is what we can afford that still works.

2 Prior art

Short notes on each. Future-me, do not be alarmed by the count — these are mostly variations on a single optimisation problem, viewed from different traditions.

Entry	Operates on	What it does	Where bound enters
Resource-rational analysis	cognitive strategies	picks the optimum	soft cost on cognitive operations
Bounded optimality	programs for architecture	picks the optimum	hard feasibility
Rational metareasoning	computations, online	picks the next one to run	per-step VOC cost
Information-theoretic bounded rationality	policies	picks the KL-constrained optimum	KL budget vs prior
Bounded inductive rationality	logical beliefs over time	characterizes calibrated paths	no-arbitrage in the limit
Computationally tractable choice	decision rules	rules out intractable ones	polynomial-time axiom
POMDP-as-agent	system interpretations	characterizes POMDP-shaped ones	state-space capacity
Computationally-embedded big world	agent automata in computable worlds	picks interactivity-maintaining ones	agent fits inside the world

NARS is somehow philosophically nearby but is not formalised tightly enough to fit a row.

3 Substrate principle, Cantor trap

The thought experiment in the opening — give the demon complete information and unlimited compute and let it simulate the universe — articulates a weird bias in the literature that I had not found words for before. We ignore that the demon needs somewhere to put the simulation.

Wolpert’s stochastic thermodynamics of computation (David H. Wolpert 2019; David H. Wolpert and Korbel 2026) is one attempt to price the substrate. Any logical operation has a thermodynamic cost; any physical computer is itself a dynamical system whose computation we identify by mapping its dynamics onto an abstract machine. A simulator of the entire universe is, at minimum, a physical system as complex as the universe being simulated — the exact lower bound depends on the dynamics, but it is not zero. So a Laplacian demon needs a bigger universe to live in. There is no infinite-compute thought experiment that is internally coherent without dragging in another, larger, world.

Call this the substrate principle. I’m sure there is a better term of art for this. Landauer talked about this kind of thing, although thinking specifically of thermodynamic costs, which might be the right way to frame it, but this does not seem trivial to me. Any agent — including the one we use to motivate the unbounded-compute limit — is a physical system competing for the same resources as the world it models. Bounded-information, bounded compute is not one option of four; it is the only combination that obtains. The other three are interesting idealisations — projections of bounded agents in which we let things grow to infinity. They are not realisable, and moreover it is not clear to me that we ever get close to any of them that they are even a good approximation.

The bias in the embedded agency and broader agent foundations literature is what I like to call a Cantor trap: treating an unrealisable combination of compute or data as a place where one can stand and reason, and then “approximating down” from it. The trap looks like progress because Cantorian objects — countably-infinite hypothesis classes, ω-limits, transfinite hierarchies of agents — are mathematically clean and aesthetically seductive (Hilbert’s paradise from which no one shall expel us!). In practice the infinite limit seems to be a vanishing point. The diagonalisation puzzles flagged in bounded inductive rationality — Löb, 5-and-10, spurious counterfactuals — are symptoms of the same trap: what happens when one tries to make an agent’s reasoning span its own infinite compute. It is worth being aware of the dangers of this trap. Infinities are cool, ngl, but we need to know how informative they are about finite agents before we use ‘em.

A particular annoyance: my embedded agency notebook was my attempt to understand careful, technically polished work on reflective stability, logical counterfactuals over the agent’s own actions, and value-stability under self-modification — exactly the problems the Cantor trap generates. From the bounded agents perspective, the same questions look different: some dissolve, others survive but in milder versions, about capacity and prices on uncertainty rather than impossible self-knowledge. AFAICT a tradition with this much technical care has produced surprisingly little to help us reason about agents that are “like us” in this important way.

Solé et al. (2024) line up adjacent constraints from the biology side — what living systems can and cannot be, given their thermodynamic and informational budgets. Useful as a sanity check that the substrate principle shows up empirically in the systems we already call agents.

Future-me, have you read all these papers yet? One tempting next step is a specific lower bound on a model’s complexity in terms of the agent’s thermodynamic budget. Is that done?

4 Where next

Chatbot suggests:

1. Read Lewandowski et al. (2025) and Baltieri et al. (2025) carefully and write the diff. Both are about embedded agents containing models; one is information-theoretic and asymptotic, the other categorical and structural. There may be a tidy correspondence.
2. Pick one toy problem — a satisficing LQR, or a tree-search agent with anytime stopping — and run the resource-rational, bounded-optimal, and KL-constrained derivations side-by-side. Verify that they tell the same story, or notice where they diverge.
3. Promote the substrate principle (and the Cantor trap with it) to a standalone post; the ingredients above are enough for one.
4. Sketch what a “good regulator theorem under a thermodynamic budget” would look like, even at one page. Surely this must be done? I have not read All the Things yet.

5 Incoming

• Tishby & Polani’s information theory of the perception–action cycle (Tishby and Polani 2011) — close to Ortega-Braun, but starts from the cycle.
• Ho et al. on simplified mental representations for planning (Ho et al. 2022) — empirical evidence that humans plan with a deliberately impoverished model.
• Gigerenzer & Goldstein on fast-and-frugal heuristics (Gigerenzer and Goldstein 1996) — anchor for the descriptive bounded-rationality literature based on humans.

6 References

1 The Australian context

Union density in Australia is about 13.1% of employees as of August 2024 — up from 12.5% in 2022, the first recorded increase since 2011. Over the longer term the trend is down: in 1992, density was 40%. That is to say, the movement has been dying for three decades, and recently it, uh, stopped dying. Something interesting is happening at the margin, driven mostly by young workers, but from a low base.

Inside those averages, professionals are overrepresented (around 20% density) and education-and-training workers more so (27%). So if you are a white-collar knowledge worker reading this, you are not colonising a strange industry — there are plenty of us already.

The Australian union system is unusual in a few ways.

Enterprise bargaining is the central mechanism for negotiating conditions. Pay and conditions for most workplaces are set by an enterprise agreement negotiated between the employer and a bargaining representative — usually the union — at the level of the single workplace or company. Since the Fair Work Act 2009, this is how a modern Australian pay rise gets made. Our union’s main job, most of the time, is to negotiate this instrument.

Protected industrial action is only legal during bargaining for a new agreement, and only after a secret ballot via the Fair Work Commission, and only with three working days’ notice (for most workplaces). Striking in sympathy with another workplace is not protected, which is to say, it is effectively illegal. So, don’t be bringing to mind militant historical or european unionism when you think about the modern Australian version.¹

Most big unions affiliate to the ALP. “Affiliate” means something specific: the union’s President and Secretary sign a pledge to the Labor platform, pay a per-member affiliation fee, and the union’s members are thereby entitled to 50% of delegate positions at Labor state and national conferences. The ACTU itself does not affiliate — it is a peak body — but most of its largest member unions do. In practice, this means some fraction of our dues is helping to elect a particular political party, and that the union’s political stances will be calibrated against not upsetting Labor. We can think this is fine or not fine. IMO it is a narrow kind of politics — not obviously better or worse than the available alternatives, but I am personally not a fan.

2 The flaws

3 Why it’s worth it anyway

Given all the above, why did I sign up as a delegate, and why would I do it again? Because the leverage is huge. Being a delegate got me:

Inside gossip. The delegates’ meetings are where one learns what management is telling the union, which is often a truer signal than what management is telling staff. One sees drafts of redundancy proposals before they become announcements. One hears which divisions are being quietly wound down. Going the other way, the delegate carries back useful information about the workforce that the management chain on its own would not have heard.

The more underrated form of this, IMO, is lateral gossip — what peers know about each other that a hierarchical organisation aggressively filters out. Our formal view of colleagues in other teams is mediated almost entirely by those teams’ managers, who have career incentives to present their units as competent, well-run, and indispensable. The union is one of the few structures in a big organisation where frontline staff from different teams compare notes, and discover: which empire-building manager is winning headcount they don’t have work for, which team is quietly drowning, which senior technical lead is the bottleneck nobody can work around, which department’s nominal stars are coasting, which quiet group does the good work that never shows up in the monthly report. This stuff matters for a lot of mid-career decisions — internal transfers, whether a cross-team dependency is safe to rely on, which up-chain promises are worth trusting — and in an excessively hierarchical organisation almost none of it is available through any other channel. People seriously underestimate how much of their picture of their workplace has been composed for them by interested parties. The union cuts across that.

Strong relationships with colleagues across silos. Research organisations partition into teams that rarely talk to each other, and that compete for resources. The union is one of very few structures where a data scientist talks to a comms officer talks to a facilities manager as peers with a shared interest. This defangs a lot of internal resource competition simply by putting faces on the “other” side of it. The employer benefits from this too, even if they rarely notice: a less siloed workforce wastes less energy on internal turf wars.

Esprit de corps. Not nothing. Knowing that one has a dozen colleagues whose numbers are in one’s phone, and who will pick up, is a psychic resource in a way that is hard to describe until one has had it.

A widened range of sayable things at work. Delegates, by convention and by the adverse-action provisions of the Fair Work Act, have more protection for speaking publicly about workplace issues than ordinary staff do.² We can say things a rank-and-file employee would be nervous to say, because we are saying them in our delegate capacity. This is not absolute — we can still be sacked for many things — but it widens the range of sayable things considerably. I used this a lot.

A direct line to powerful people when it mattered. At bargaining time, at redundancy time, at restructure time, delegates are in the room with the CEO, or the DG, or their direct deputies. One gets to negotiate with the people actually making the decisions. For a mid-career professional this is an unusual amount of face time with senior leadership, and the skills one builds — reading the room, finding the pressure points, knowing when to escalate — are portable.

Friends. I left the delegate role with more friends at CSIRO than I had before I took it. I don’t know a better return on my time.

4 A second centre of power

Most of the above is about what the delegate gets. The less obvious side is what the employer gets from having a functioning union on the premises. IMO a sensible leadership team treats the union as a cheap source of information and a cheap source of accountability that would otherwise be unavailable to it — which is the core of what having a second centre of power inside an organisation buys you.

A back-channel past toxic managers. Every formal complaints mechanism I have encountered routes upward through the chain of command. Which fails at exactly the case that matters most: when the problem is the chain of command. A staff member being ground down by their manager has almost nowhere to go internally — the next rung up is the bully’s mate, or their delegate of authority, or someone institutionally invested in not seeing the bully as a problem. The union is a parallel chain that does not route through the bully. A well-run HR function treats this as a useful signal, because it surfaces problems the formal channel would have swallowed. A badly run one resents it. The ones that resent it tend, in my experience, to be the ones whose formal channel is broken in ways they would rather not admit.

Risk identification. Frontline staff know things that never make it onto a dashboard: safety hazards, project-failure signals, mis-scoped deliverables, customer-abuse patterns, strategies that will not survive contact with reality. A lot of this information is uncomfortable, which is precisely why it gets filtered out as it moves upward through layers of people whose performance reviews depend on appearing to have things in hand. The union is one channel by which those signals can bypass the incentive structures that suppress bad news going up the chain. Same logic as whistleblower protection, operating at lower stakes, more often, on smaller problems — which means the organisation finds out about them while they are still small.

A commitment device. An enterprise agreement, together with an organised workforce that will notice breaches of it, is a way for senior leadership to credibly commit to things. Promises management would like to make but cannot otherwise guarantee — about workload, about career paths, about fair treatment of a category of worker — become more durable when backed by a bargaining process. In principle, the employer gets a more motivated, lower-turnover workforce in exchange for being bound to its own stated intentions.

None of this is free. The union is also, sometimes, the thing that negotiates an inflexible clause into our enterprise agreement that makes something we want to do hard. But on balance — for an organisation that can tolerate being told uncomfortable things by people it cannot easily fire — a functioning union looks closer to an internal audit function than to a hostile external force.

This is not to pretend away structural conflict. Interests do diverge sometimes — about wages, about job security, and above all, who carries the risk when things fail. Conflict, when it arrives, is an important mechanism: the bargaining apparatus is built around exactly those divergences. But most decisions, at least in my workplaces, most of the time, do not need to be zero-sum wars. Some zero-sum fights I was involved in were made to be zero-sum to keep people busy, and those could have been turned into collaborative problem solving. When a bad manager stokes conflict to distract attention from their own failings, this is a power play, not an inevitable state of affairs.

Neither “side” is a monolith. “Workers” and “employers” are each coalitions — junior and senior staff, contractors and indefinite, technical and administrative; shareholders, executives, line managers, finance, operations — with partially overlapping and partially competing interests. Some worker factions are at odds with other worker factions more than with any employer faction. Cross-line alliances — this technical team and the CTO, this cohort of staff and the CFO’s cost-containment agenda — are common and often stable. The two-team framing is convenient shorthand, not an accurate map.

Leaning on the adversarial frame as the default identity of unionism — worker-vs-boss as our core brand — is great for recruiting — people love a fight with the Other. But it has costs: it alienates the moderates on both sides, and leaves win-wins on the table that would otherwise have been obvious. It is important, but it is not the only game.

Most executive teams don’t want union pushback. IMO they should want all the union engagement they can get. A second centre of power inside a dysfunctional organization can be one of the few ways management gets the information and accountability needed to stop making predictable, avoidable mistakes — which is a case of the institutional alignment problem. The union, or at least many unions I have been involved with, can be a flawed instance of that second centre: too procedural, too political, too disengaged, for all the reasons I listed above. But those flaws are not, mostly, irreducible features of unionism. They are habits that unions can fall into, and which we can change by joining the damn union and making it work better.

Members are within their rights to ask that unions be better. Hell, managers are within their rights to want that too. But the union is us. If we want a second centre of power in our organisation that is not captured, not incumbency-biased, and not hoarding communications, the mechanism by which it comes to exist is: we join, and we show up, we pay 0.7% of our salary, we invest our time and sweat equity in making everything else better. The price of admission is unusually low.

5 Advice I wish I had on day one

A few things I had to work out the hard way.

Turn up. A startling fraction of the influence in the delegate body accrues simply to the people who turn up to every meeting. This is unglamorous and correct. Showing up is 80% of the job. If there are literally too many meetings to attend, ramp it back. If the union insists on stupid held-over structures (e.g. place-based organising for an online organisation with distributed teams, looking at you CSIRO Staff Association) we could make it a project to try to modernise.

Ask for the bargaining log of claims early. Every round of enterprise bargaining starts with a list of things the union is asking for. If you are not on the bargaining committee you may not see it until it is near-final. Ask for it. If there are claims we care about, we have to fight for them early in the process, not at the end.

Cultivate union communications the union staff cannot control. Ideally you want to trust your organiser (for the record, I have always trusted my organiser’s intent and competence). But baking too much control into the union staff’s roles sets up bad incentives for them to hoard power just like the bosses. If the union doesn’t run a members’ chat, start a group chat among the delegates you trust. If the union’s email list is one-way, the ground truth of what members want must reach decision-makers through some other channel.

Learn the instrument. So easy these days that there is no excuse. Load the current enterprise agreement into an LLM. The last one too. Also, Part 2-4 of the Fair Work Act. Ask questions. Read specific passages. Learn the stupid terms of art. We become much more useful to our colleagues the moment we can answer basic questions about the instrument instead of referring them upward.

Practise communication skills. Delegate work is the best-calibrated training environment I have found for the workplace-communication skills that white-collar work otherwise hides from us until we are mid-career and already expected to have them: writing a clear update for members, chairing a meeting where people disagree and we need a decision at the end of it, negotiating with a senior executive who has lawyers (we don’t), delivering hard news to a colleague, pushing back on a bad idea without burning the relationship. The stakes are high enough that sloppiness has consequences, and low enough that we get to try again next meeting. Take the opportunity.

Do not let this become your whole life, or even your whole job. Union work is high-leverage precisely because the baseline engagement of everybody else is so low. Which means there is infinite demand for your time. Decide in advance what fraction of your working week you will spend on it, and stick to it. Extension projects are good, but don’t pick too many.

Recruit more delegates. The more delegates we have, the more influence we have, and the less work each delegate has to do. Don’t solve everything yourself; solve one thing and use that as a recruiting pitch for the next delegate. Showing people they can make life better is hugely encouraging.

You get so much love for doing this. People, mostly, value and appreciate the work of delegates. I got given so much respect from my colleagues when I’d shown up for a tough time, but they would make sure the line manager was not around. Then the line manager would also thank me, when their direct reports were not around. Double thanks!

6 If we were starting from scratch

A sceptical reader will have noticed that almost everything in this post is a second-best argument. The union is a patch on a firm shape that has employees and shareholders on opposing sides of an ownership boundary, with managers in between. A lot of what the union does — surfacing risks, piercing silos, keeping managers accountable, sometimes even committing the enterprise to its stated intentions — is only necessary because the firm is designed the way it is.

Other shapes are possible. Let me get galaxy-brained for a second: Worker-owned firms, consumer cooperatives, and various flavours of co-determination redistribute the ownership boundary and fold the “second centre of power” into the first. I have written separately about how one might divide equity in a firm being built along those lines. More broadly, this sits inside the frame of institutional alignment: any institution we build is, in effect, a trained system whose incentives and reporting structures have to be tuned for the work we want out of it, and most organisations have simply never been tuned for the work of collectively caring for the people inside them.

A ground-up redesign is not on the menu for most employees of most organisations. But the frame is useful for explaining why even a flawed union is high-leverage: unions are one of the viable ways to retrofit a power distribution into an organisation that was not designed with one. If we could start over, we might build the distribution in at the foundation. We mostly can’t, so we patch.

7 Resources

Background and statistics

• ABS Trade Union Membership, August 2024 — primary source for density figures.
• RBA RDP 2019-02: Is Declining Union Membership Contributing to Low Wages Growth? — careful empirical work on the wage effects of union decline.

The legal instruments

• Fair Work Ombudsman: Enterprise agreements and bargaining
• Fair Work Commission: Industrial action
• Fair Work Act 2009 (text)

Peak bodies and some relevant unions

• Australian Council of Trade Unions (ACTU)
• CPSU (PSU Group) — federal public sector
• CSIRO Staff Association — a section of the CPSU; my own alma mater
• NTEU — tertiary education
• Professionals Australia — engineers, scientists, managers

8 Things I am less sure about

I don’t know how much of the communications-hoarding phenomenon is specific to CPSU-style professional unions versus a general feature of mid-sized Australian unions. Anecdotes from other delegates suggest the latter; I haven’t confirmed.

1 Urban scaling laws

The urban scaling results are the empirical anchor for all of this. Bettencourt (2013) provides the theoretical derivation: cities are social reactors, and the superlinear scaling emerges from the increasing number of social interactions per capita as density grows. Bettencourt et al. (2007) established the empirical regularities — wealth creation and innovation scale with exponent , infrastructure with .

Methodological critique of the naive approach may be found in Cottineau et al. (2017) — the measured exponent is sensitive to how we define city boundaries, which indicators we use, and which countries we look at. Arcaute et al. (2015) show that for England and Wales, many scaling results vanish or change sign when city boundaries are defined differently. This matters: if the exponent is an artefact of the boundary definition, the whole edifice wobbles.

There is much more (Arcaute et al. 2015; Balland et al. 2020; Bettencourt and Lobo 2016; Cottineau et al. 2017; Kühnert, Helbing, and West 2006; Leishman et al. 2021; Prieto Curiel, Cabrera-Arnau, and Bishop 2022; Rybski, Arcaute, and Batty 2019)

2 Economic scaling laws

The literature on returns to scale is enormous and old, but I want to slice it by the unit of analysis: firms, nations, cities. The scaling exponents turn out to be different in each case, and the reasons why are the interesting part.

3 The cosmic version

Wong and Bartlett (2022) push this question to its most cosmic limit. They take the urban scaling exponent from Bettencourt et al. (Bettencourt et al. 2007) and ask what happens when the entire planetary civilisation becomes, in effect, one city — one densely networked informational superorganism connected through the dataome.¹ The growth equation

predicts singularities — moments when population and energy demand tend to infinity in finite time — which must be averted by innovations that reset the system’s trajectory. But the interval between resets, , shrinks as the population grows. Eventually drops below the timescale on which innovation can actually occur (), and the system faces what they call asymptotic burnout: collapse driven by the superlinear dynamics that previously drove growth.

Their proposed resolution is homeostatic awakening — the civilization consciously reorienting away from unbounded growth toward persistence and well-being. Which is to say, the civilization invents a new objective function. This is presented as a resolution to the Fermi paradox: either civilizations burn out (short-lived), or they pivot to homeostasis and become quiet (long-lived but difficult to detect). Either way, no galaxy-spanning Type III civilizations.

There is an older version of this argument, without the scaling-law formalism. Tainter et al. (2003) (and Tainter’s earlier Collapse of Complex Societies) propose that complex societies face diminishing marginal returns to complexity itself: each additional layer of coordination, bureaucracy, or infrastructure costs more and delivers less, until the society can no longer sustain its own overhead and simplifies — voluntarily or otherwise. Wong and Bartlett’s burnout is the superlinear-dynamics version of Tainter’s complexity trap: the system does not merely stagnate (as in Tainter) but actively accelerates into crisis. Both end in simplification; the mechanism differs.

I find this idea interesting as a question organizer even if the quantitative model is schematic. The key variable is : whether the scaling exponent for a given system sits above or below 1 determines whether it exhibits increasing returns (and eventual crisis) or diminishing returns (and eventual stagnation). And the value of is not the same for all systems.

4 AI and the Coasean boundary

If the Coasean boundary of the firm is set by the cost of internal coordination relative to market transaction costs, and if AI agents radically reduce both, what happens to the equilibrium firm size?

There are at least two competing effects. Korinek and Vipra (2025) argue that AI itself exhibits enormous economies of scale — training frontier models requires billions of dollars in compute, and the resulting capabilities can be deployed at near-zero marginal cost. This pushes toward concentration: a few firms with the best models capture most of the value.

But there is a countervailing force. If AI agents can handle complex contracting, negotiation, and monitoring at scale — Coasean bargaining at scale, as it were — then transaction costs on the market side also collapse. The “Headless Firm” model (Harré and Ormerod 2025) proposes that agentic AI changes how coordination costs scale: from in human-managed organisations to in protocol-mediated agent systems. If so, we might see a simultaneous expansion of what large organisations can coordinate and shrinkage of what needs to be inside an organisation at all.² Which effect dominates?

Wolpert and Harper (2025) model societies as computers, where the computational power available depends on how agents are organised. If AI lets us reorganise the topology of economic interaction — not just doing the same coordination faster, but enabling coordination structures that were previously impossible — then the scaling exponent itself might change. We might move the boundary between the firm-like regime () and the city-like regime ().

I don’t know which effect wins. Possibly both at once, in different domains. But the stakes are those of Wong and Bartlett (2022): if we push high enough, we push down, and we accelerate the approach toward burnout — or toward the moment we have to decide whether homeostatic awakening is something we can actually do.

5 Where does this leave the Singleton?

The scaling laws give us a tentative answer to the opening question. The world probably does not converge on one system, because the scaling exponents differ across levels of organisation. Cities generate superlinear returns, but the containers that govern cities — firms and states — face diminishing returns to scale. The Leviathan is bounded.

But that conclusion rests on current coordination technology. If AI changes the exponents — if it makes large-scale coordination cheap enough that firms or states start exhibiting city-like superlinear scaling — then the attractor shifts. The singleton becomes more plausible, or at least the threshold for “too big” moves upward.

There is a flavour of technological determinism to excavate here — the idea that coordination technology determines the viable scale of organisation, and therefore the shape of political and economic life. If the printing press enabled the nation state, and if the internet enabled global supply chains and platform monopolies, then AI-mediated coordination might enable — or force? — another jump in the viable scale of collective action. The scaling exponents are not constants of nature; they are artefacts of the coordination technology available. Change the technology, change , change the equilibrium size of everything.

This is not quite hard technological determinism — we are not claiming the technology uniquely determines outcomes. Alesina and Spolaore (2003) already argues that the equilibrium depends on the international security environment, trade openness, and democratic norms, not just coordination costs. But the scaling exponents set the envelope of what is possible, and the technology moves the envelope. Soft determinism, maybe. The tools constrain the menu; the polity still orders from it.

Then we are back to Wong and Bartlett’s question: a civilization with a sufficiently high will face accelerating singularities. Is homeostatic awakening something that can be engineered? Or is it the kind of thing a civilization only stumbles into after a sufficiently frightening brush with burnout? And — the multi-level agency angle — who does the awakening? The civilization, the nation, the firm? A single God-like uploaded CEO? Roco’s basilisk? If the scaling exponents differ across levels, the incentive to keep growing differs across levels too. The firm-level agents face sublinear returns and might happily plateau; the city-level dynamics keep accelerating regardless of what any individual agent wants. The homeostatic awakening, if it happens, has to happen at the level where — which may not be the level at which anyone has decision-making authority.

TODO: dig into Wolpert and Harper (2025) — their “society as computer” model feels relevant.

TODO: the biological scaling literature deserves its own subsection. Kleiber’s law ( for metabolic rate) and the West-Brown-Enquist model are the template that Wong and Bartlett are generalising.

TODO: chase Anderson’s Imagined Communities — there’s a history-of-coordination-technology story here that connects the printing press → nation state → telegraph → empire → internet → platform monopoly → AI → ??? sequence. Each technology changes something — the coordination exponent? — for a different level of organisation.

6 References

1 People and things from Czechia I knew before

Experimental conditition: I primed myself before I arrived, writing down everyone and everything I have heard of that is Czech.

At time of writing I wonder: What will I miss? Which prejudices will I be demonstrating thereby? I am greatly curious.

I am attempting to triangulate the country from its more famous children. Such a method has obvious shortcomings— Children rarely resemble their parents, and famous children least of all.

Read down below this section for a post-arrival addendum of whom I missed.

2 People and things from Czechia I missed

2.6 Jena Codex

The Jena Codex, also called the Jenský kodex or Antithesis Christi et Antichristi, is badass Figure 2. It is, I am told by Czech wikipedia an illuminated Hussite manuscript from around 1500, commissioned by the Utraquist (i.e. Hussite-flavoured) Bohuslav of Čechtice. I cannot read it, but apparently it is full of theological and polemical texts that contrast the early Christian Church with the Church of Jan Hus’s time. Here are three online versions of it:

• Manuscriptorium
• eSbirky.cz - Antithesis Christi et Antichristi
• Category:Jenský kodex (whole pages) – Wikimedia Commons

3 When was Prague the capital of the Holy Roman Empire?

Turns out that five rulers of the Holy Roman Empire — or its elected German kingship — had, or nearly had, their court in Prague between 1346 and 1612. Sometimes Holy Roman Emperor or court in Prague is a matter of considerable freedom of interpretation.

Six years after Matthias left, the Second Defenestration of Prague (1618) launched the Thirty Years’ War; the Habsburgs won the Battle of White Mountain in 1620 and Catholicised Bohemia by force. Czech politics (at least as national politics) then went into hibernation until 1918.

4 Language

5 Is Czechia in Central Europe or Eastern Europe?

I vote Central.

Reasoning:

• It was in the Holy Roman Empire, the most central European institution there ever was
• OK it was in the Soviet bloc, but like Poland and Hungary were as vassal states, not like Russia was
• Speaks a Slavic language, which tends to code easternness (but then cf Romania, which speaks a non-Slavic langauge and is coded way more eastern). Also their script has been Latin since the Middle Ages.
• More geographically west and less religiously Orthodox than Greece which is not eastern because they invented democracy and philosophy and stuff and the Mediterranean gets you a pass for some reason?
• Godless, which is to say, has not joined the eastern trend to re-embrace religion as a marker of identity post communism
• Successfully invaded by Hitler in 1938, but not successfully invaded by Stalin in 1948 (unlike Hungary and Poland)
• They sauna naked (unlike the Russians, but like the Scandinavians)
• Invented the protestant reformation 80 years before Luther and fought a war about it. What could be more central European than a war of protestant peasants against an oppressive church?
• To paraphrase my Romanian colleague Teodora, “pfft you call this eastern? you should see Romania.”

Conclusion: No more eastern than Slovenia, and more central than Hungary and Poland. Therefore: Central.

Now, a more interesting case is former partner state Slovakia. Are they central or eastern Europe? This question is left as an exercise for the reader.

6 Prague

OK, it is actually pretty. Decayed imperial grandeur, baroque churches, gothic cathedrals, Art Nouveau facades, and a river running through it. Overlaid soviet utilitarianism, low-budget 90s lowest-bidder capitalist renovation. Not quite as pretty as Venice, say, but way more lived-in and less museum-y. By which I mean, it is less museum-like.

It is probably more museum-dense. Wow, so many museums! Some really fancy-sounding ones, and a long tail of very random shit; Madame Tussaud’s franchise, chocolate shops with “chocolate museums” attached to them, and some really suspect Ripley-axis stuff, e.g. “Museum of Sex Machines”.

More tourists than I was expecting, but like, even in the old town, also some normal humans doing non-tourist lives in between the tourists. The mix of tourists, Habsburg cosplayers and philosophy students on their way to band practice seems fine?

6.1 Bikes

So far, Prague’s bike infrastructure is meh. I would describe it as being at the level of “90s in Australia but with more cobblestones”. Also the riding directions are aggressively bad on Apple Maps, like “Please ignore the bitumen road and take this much longer detour via some cobbled pedestrian walking street which is both more crowded and more difficult to ride on” Figure 3.

To be fair, Apple Maps generally sucks here. Google Maps is the main game.

7 Incoming

• TODO: get corrected on all of the above by someone who lives there.

8 References

1 A phenomenon of note

A mind modelling another mind is an agent embedded in an environment that contains agents with comparable representational capacity to itself. If the only faithful model of Alice is Alice, then Bob cannot fit one in his head. A practical Bob must carry a compressed Alice: fewer parameters, coarser predictions, maybe with a cartoon-level ontology. Call this a reduced-rank other-model.¹

Bob must also act, and acting well might require that Bob predict his own future behaviour. If the only faithful model of Bob is Bob, he cannot fit one of those in his head either. So Bob carries a “reduced-rank” self-model. This self-model is what Metzinger calls the phenomenal self-model (Metzinger 2003), what Graziano’s Attention Schema Theory (Graziano 2013) makes into a neural control-theoretic object, and what Schmidhuber-flavoured AI calls a world-model-containing-self (Ha and Schmidhuber 2018).

The bicameral-mind literature (Jaynes 1976) sounds like it’s somewhat related — the sense that “I” am addressed by a voice that is also “I” — but it doesn’t seem formal enough to build on, so I will basically ignore it here.

I can think of three axes along which theories might vary:

1. Other-modelling. How do formalisms represent nested belief (“I think that you think that I think…”)?
2. Self-modelling. How do formalisms represent an agent that contains a compressed simulacrum of itself?
3. Reduced rank. By rank I mean the computational fidelity of a sub-model — the bits or parameters devoted to it, equivalently the resolution at which it can discriminate the situations it cares about. How is this reduction made rigorous — rate-distortion, PAC bounds, etc.?

2 Other-models

3 Self-models: the formal landscape

3.1 World models containing self

A direct ML instantiation is Ha & Schmidhuber’s World Models (Ha and Schmidhuber 2018), where a recurrent latent model predicts both environment dynamics and the consequences of the agent’s own actions. The agent’s policy is trained inside this compressed dreamscape. The self here is a reduced-rank conditional — “what would my controller do, given this latent” — rather than an introspectable entity, but the “compression” part sounds well-posed.

The lineage continues through Dreamer (Hafner et al. 2024), MuZero (Schrittwieser et al. 2020), and the larger world-model-RL programme.

See also world models.

4 Reducing fidelity of representation

Several toolkits formalise “reduced rank”:

• Rate-distortion theory applied to cognition (Sims, Jacobs, and Knill 2012; Zénon, Solopchuk, and Pezzulo 2019; Lai and Gershman 2021): the cost of mental representation is an information-theoretic rate, the benefit is task performance, and optimal bounded agents sit on the rate-distortion frontier.
• Information bottleneck (Tishby, Pereira, and Bialek 2000; Alemi et al. 2019): compress inputs to a latent that is maximally informative about a downstream variable. When the downstream variable is “the other agent’s next action”, the bottleneck induces a reduced-rank other-model.
• Resource-rational analysis (Lieder and Griffiths 2020): agents are optimal given bounded compute; the bound is the reduction.
• Successor representations / features (Dayan 1993; Barreto et al. 2017): compressed future-prediction models that generalise well across reward functions. A kind of reduced-rank self-model of one’s own policy.
• Bounded rationality as a research programme (Simon 1955; S. J. Russell and Subramanian 1995; S. Russell 2016).
• Epsilon-machines / computational mechanics (Crutchfield and Young 1989; Shalizi and Crutchfield 2000): minimal-sufficient-statistic models of a process. The causal states are the minimum-rank predictor.

My research agent further recommends we look at theory of mind as mutual information (Jara-Ettinger 2019) and the recent graph-theoretic accounts of social abstraction (Stolk, Verhagen, and Toni 2016).

5 Multi-agent self

The bicameral intuition — “the mind is many minds talking” — turns up across Minsky’s Society of Mind (Minsky 1986), Global Workspace Theory (Baars 1993; Dehaene 2014) and its neural-network and RL formalisations (VanRullen and Kanai 2021; Goyal et al. 2021), mixture of experts (Jacobs et al. 1991; Shazeer et al. 2017), and Dennett’s multiple drafts (Dennett 1993). They share an architectural claim: the “self” is what falls out of specialist modules competing for a low-capacity shared bottleneck, and that bottleneck is where the reduced-rank self- and other-models have to sit. Attention Schema Theory (above) fits here too. Fuller treatment at multi-agent self.

6 Self-referential agents and the proof-theoretic frontier

This is where reflectivity — the pressure for an agent’s beliefs about its own future beliefs to be consistent — becomes a first-class concern. If we want to prove things about minds that model themselves, we hit self-reference, and self-reference hits Löb’s theorem and friends if we’re not careful. As presaged, I am not super excited about the parts of this line of work that edge into unbounded compute.

Reflectivity is operationalised differently by each of the constructions below, and each buys self-consistency in a different coin — fixed-point iteration, market mixing, proof search — so its “cost” lives on a different axis depending on who you ask. I’ll treat its cost as real but construction-dependent.

• The self-consistency cluster. Reflective oracles (Fallenstein, Taylor, and Christiano 2015) (a fixed-point construction of probability distributions closed under self-reference), logical induction (Garrabrant et al. 2020) (a market-based learner whose beliefs about its own future beliefs converge), and the Löbian obstacle (Yudkowsky and Herreshoff 2013) (the negative result that a self-modifying agent unwilling to endorse a weaker-or-equal successor runs into Löb’s theorem): three constructions attacking the same self-consistency problem in three different coins.
• Modal combat agents and program equilibrium (Bárász et al. 2014; Critch 2019): agents that condition on source-code-level models of each other; admits bona fide equilibria in the one-shot prisoner’s dilemma.
• AIXI and approximations (Hutter 2005; Leike et al. 2016): a formally optimal agent whose self-model is implicit in the universal prior; computable approximations (e.g., AIXI-tl, MC-AIXI) buy tractability at the cost of reducing the rank of the prior.

MIRI’s Agent Foundations programme is the main hub for this line of work.

7 Let’s attempt synthesis!

The three axes above — other-modelling, self-modelling, reduced-rank — are not independent knobs an architect sets arbitrarily. I’d argue they are three demands on the compute embedded in a single joint representation carried by a social agent — depth of recursion, rank of each sub-model, and reflectivity — competing for the same finite resources. This is of a piece with my general heuristic that we should always think about where to spend our compute to make sense of the AI landscape as it exists.

“Compute” here is a cover term: it lumps together bits of representation at rest, operations per decision, and data to fit the representation in the first place. These don’t fully interchange, and nobody defines a fungible unit that resolves the trade-offs cleanly AFAIK. For the argument that follows the weaker claim is enough — that the three axes all draw on a shared pool, and that each spends it through its own, largely unknown, return on investment. Rate-distortion theory gives a concave for rank (Sims, Jacobs, and Knill 2012); the analogous curves for recursion depth and reflectivity are open problems.

On this reading the existing literature is a tour of corner solutions: each framework spends its compute on one or two axes and lets the rest go to zero.

• I-POMDPs and level- / cognitive hierarchy: spend on depth; each level is a cartoon of the next. Humans clustering at is consistent with a small budget.
• Rate-distortion cognition, information bottleneck, successor features: spend on rank at fixed depth (); reflectivity ignored.
• Reflective oracles, logical induction, Gödel machines: spend on reflectivity; other-structure kept simple enough that the construction survives.
• Active inference: allocation across self- and world-model; depth and reflectivity implicit.
• LOLA / opponent shaping: ; rank inherited from the opponent’s parameter count.
• World models (Ha/Schmidhuber, Dreamer, MuZero): storage in the latent dimension; ; reflectivity absent.
• ToMnet and I-POMDP-Net (Han and Gmytrasiewicz 2019): learned reduced-rank other-models; ; reflectivity absent.
• SOM (Raileanu et al. 2018): with self-model and other-model sharing bits; reflectivity absent. A case of the axes interacting rather than competing.
• Reflexion / LATS (Shinn et al. 2023; Zhou et al. 2024): inference-time reflectivity over an in-context self-model; no learnt other-model; rank bounded by the context window.

8 Angles of attack

I got an LLM to ideate below ideas for me. Lightly edited for baseline sanity. The budget view suggests several angles of attack on what we actually care about — understanding how real cognition, human or machine, allocates its compute across world, self, and other. Two concrete ones, plus a brief aside on scaling the same calculus to collectives.

9 Related

• Agent foundations
• Causality in multi-agent systems
• Predictive coding
• Dunning-Kruger theory of mind — the failure modes of human other-models
• Iterated social games

10 Incoming

• Dennett and Hofstadter’s The Mind’s I as a pre-formal reading list.
• Hofstadter’s strange loops (Hofstadter 2008): evocative, not formal, but a useful bridge.
• Tononi’s IIT as a measure on self-models, if one is feeling brave.
• Whether the “observer self” of contemplative traditions corresponds to an attention-schema-style reduced-rank self-model — this seems to be Metzinger’s read.

11 References

1 Neural MMO

A large, open-world game environment for multi-agent research (Suárez et al. 2019; Suarez 2024; Suárez et al. 2023). Claim to fame: leveraging gaming-industry technology to provide a persistent, large-scale world for agent interaction, going beyond the typical matrix game or grid world.

2 AgentScope

See Pan et al. (2024).

modelscope/agentscope / documentation

A multi-agent platform oriented toward LLM-powered agents, with actor-based distribution and fault tolerance.

3 PettingZoo

PettingZoo is the multi-agent extension of Gymnasium (formerly OpenAI Gym), providing a standard API for multi-agent environments. Covers classic games (chess, Go, connect four), Atari multiplayer, and MPE (multi-particle environments commonly used in cooperative/competitive MARL papers).

4 Melting Pot

DeepMind’s Melting Pot is a suite of social dilemma environments for evaluating multi-agent cooperation and competition. Environments are designed to test generalisation: agents must cooperate with novel partners in scenarios inspired by the collective action literature.

5 References

1 Build or buy?

The first decision is whether to customize an existing platform or build from scratch.

2 Path 1: Cloud hosting

This is a sensible starting point. Low upfront cost, no hardware to maintain, easy to iterate.

┌─────────────────────────────────────────────┐
│              Reverse proxy (Caddy)           │
│         TLS termination, static files        │
├─────────────────────────────────────────────┤
│            App server (Django/Gunicorn       │
│               or Phoenix)                    │
├──────────────────┬──────────────────────────┤
│   PostgreSQL     │     Redis                │
│   + PostGIS      │   (cache, sessions,      │
│                  │    WebSocket broker)      │
└──────────────────┴──────────────────────────┘
│              Object storage (S3/R2)          │
│          (user uploads, listing images)      │
└─────────────────────────────────────────────┘

# docker-compose.yml (simplified)
services:
  web:
    build: .
    ports: ["8000:8000"]
    environment:
      DATABASE_URL: postgres://db:5432/localplatform
      REDIS_URL: redis://redis:6379
    depends_on: [db, redis]

  db:
    image: postgis/postgis:16-3.4
    volumes: ["pgdata:/var/lib/postgresql/data"]

  redis:
    image: redis:7-alpine

  caddy:
    image: caddy:2
    ports: ["80:80", "443:443"]
    volumes: ["./Caddyfile:/etc/caddy/Caddyfile"]

volumes:
  pgdata:

3 Path 2: Closet servers

This is the sovereign option. The community’s data lives on hardware the community physically owns, in someone’s house or a shared space. It costs more upfront, requires more operational skill, and is less reliable than cloud hosting—but it means no one can pull the plug on our community because of a terms-of-service change or a foreign government’s policy shift.

4 The migration path

Start on cloud hosting (Path 1). It’s cheaper, faster to set up, and lets the community focus on the social problem (do people actually want this?) rather than the operational one (is the server up?).

If the platform takes off and the community decides it wants data sovereignty, the migration to local hardware (Path 2) is straightforward because we’ve containerized everything from the start. The procedure is:

1. Set up the closet servers and run the same docker compose stack
2. Take a database dump from the cloud and restore it locally
3. Switch DNS to point at the local server
4. Keep the cloud instance as a fallback for a month
5. Decommission the cloud instance

This should take an afternoon and the community will experience maybe 30 minutes of planned downtime.

5 Features roadmap

What do we actually build, and in what order? This is a suggestion, not a prescription—the community should decide.

6 AI-assisted development

Since I argued in the companion post that LLMs lower the development cost, let me be concrete about what that looks like.

A competent developer working with a coding assistant (Claude, Cursor, Copilot, or inference from the community’s own compute) can realistically:

• Scaffold the entire v0.1 in 2–3 weekends (Django project structure, models, views, templates, basic CSS)
• Generate test suites that would otherwise take days to write by hand
• Draft moderation tooling (content classifiers, spam detection) using local models rather than shipping community data to a commercial API
• Produce documentation and runbooks for operational procedures, reducing the bus factor

This doesn’t eliminate the need for human judgement—architecture decisions, UX design, security review—but it compresses the grunt work enough that a small volunteer team can make it go.

The sovereign compute connection is particularly neat here: if the community owns an LLM inference box, the same machine that serves the platform’s AI features (search, moderation, translation) can also be the coding assistant that helps maintain the platform itself. The tools build the tools.

7 Security considerations

A neighbourhood platform holds personal information (names, addresses, phone numbers, transaction history, private messages) for people who literally live near each other. A breach would be personally harmful in a way that’s different from a breach of a global platform—the attacker knows where we live.

Non-negotiable security measures:

• TLS everywhere. Caddy handles this automatically with Let’s Encrypt certificates.
• Passwords: don’t. Use passkeys where possible, email magic links as fallback. No password database to breach.
• Input sanitization. Django’s ORM and template system handle the common injection vectors, but we should still run a security-focused code review before launch.
• Rate limiting. Prevent brute-force attacks on auth and abuse of messaging. Django-ratelimit or equivalent.
• Encryption at rest. Full-disk encryption on the server (LUKS on the closet laptops; most cloud providers offer this as a checkbox).
• Minimal data collection. Don’t store data we don’t need. If we don’t need to know someone’s exact address (we probably don’t—postcode is enough for locality), don’t ask for it.
• Penetration testing. Before launch, ask a security-minded community member (or hire someone for a day) to try to break in. Fix what they find.

8 Legal structure for the platform

The platform needs a legal home—someone has to own the domain, pay the hosting bills, and be accountable for the data.

The simplest option is an incorporated association under Victorian (or relevant state) law. This gives the platform a legal identity, limits members’ personal liability, and costs ~$200 to set up and ~$57/year to maintain. The association’s committee manages the platform on behalf of its members.

If the platform grows and starts handling money (marketplace transaction fees, membership dues), a co-operative under the Co-operatives National Law might be more appropriate—it’s the legal form designed for member-owned economic infrastructure. See the friendly society legal analysis for a deeper discussion of the options, which overlap substantially.

Privacy obligations: under the Australian Privacy Act, organisations with annual turnover under $3 million are generally exempt from the Australian Privacy Principles. A neighbourhood platform is unlikely to hit this threshold. That said, the community should adopt privacy-respecting practices regardless of legal obligation—it’s the right thing to do, and it builds trust.

9 What this costs, total

10 Open questions

• Is Django + HTMX the right default stack, or should we optimize for the specific skills of whoever shows up to build it? (If the founding devs are Rails people, build it in Rails. If they’re Elixir people, use Phoenix. The technology matters less than the people.)
• How do we handle identity verification without becoming creepy? The postcard-with-code idea is fun but has edge cases (apartments, PO boxes, renters who move). Vouching by existing members is socially robust but creates gatekeeping risks.
• What’s the right moderation model for neighbourhood scale? Professional community managers are too expensive; pure volunteer moderation burns people out; algorithmic moderation misses context. Some hybrid is needed.
• Should the platform support financial transactions (escrow for marketplace purchases, membership fee collection), or keep money external (bank transfers, cash on pickup)? Handling money adds regulatory complexity but removes friction.
• How do we handle the “missing stair” problem—a community member who everyone quietly knows is problematic, but who hasn’t technically violated any rules?

As with everything in this series: if any of us have opinions, experience, or want to help build this, get in touch.

1 Why now?

Two things have changed that make this more tractable than it was five years ago.

2 Start with an itch

The graveyard of alternative social platforms is vast, and nearly all of them died of the same cause: they asked people to join a new social network for its own sake, on the promise that it would be better. It turns out that “better” is not a compelling reason to abandon the place where our friends already are.

So don’t start with “a social network”. Start with something concretely useful—something that scratches an itch that the incumbents have stopped scratching.

3 The hard problems are actually social

I am a person who writes about DIY social networks and federated protocols and decentralized infrastructure, so I am constitutionally inclined to think about social problems in terms of technology.

The social angle is the real hard bit, though. The technology for running a local social platform is essentially solved—there are mature, open-source options that would work out of the box.

The hard problems are:

Cold start. How do we get the first hundred users? Network effects mean a platform is useless until it isn’t, and the transition happens fast—but getting to that transition requires sustained effort by a small group of very motivated people. For a neighbourhood platform, the bootstrap group is probably 10-20 households who know each other and commit to using the platform for three months regardless of whether it feels “worth it” yet. That’s the price of admission.

Governance. Who decides the rules? Who moderates disputes? What happens when someone is creepy? Platform democracy is a real field of research, and the work on mini-publics and deliberative processes is directly relevant here. But at neighbourhood scale, governance can be simpler: a small elected committee, rotating terms, transparent decisions, published minutes. The friendly society governance model maps cleanly onto this.

Attention and norms. A neighbourhood platform lives or dies on whether it feels good to use. Not “engaging” in the way that Facebook means (maximum time-on-site, maximum emotional arousal, maximum ad impressions), but good—useful, warm, not a time-sink. The Center for Humane Technology, the Strother School of Radical Attention, and a growing body of work on the attention economy all remind us that the design choices in a social platform shape the social behaviour that emerges on it. A platform that doesn’t have an engagement-maximizing algorithm doesn’t produce engagement-maximized behaviour. That’s a feature.

Sustainability. What happens when the founding enthusiasts burn out? Community projects die when they depend on a small number of unpaid heroes. The membership-fee model helps—it means there’s a budget for someone to do administration, even if “someone” is a member who gets a fee waiver in exchange for 10 hours a month of maintenance.

4 What about the attention economy?

The platforms we’re trying to escape are attention-extraction machines. Their business model is to capture as much of our time and cognitive bandwidth as possible, because that’s what they sell to advertisers. This produces the behaviours we’ve all learned to recognize: infinite scroll, notification bombardment, algorithmic rage-bait, the manufactured sense of urgency about things that don’t matter.

A neighbourhood platform funded by membership fees has no reason to do any of this. There’s no advertiser to sell attention to. The incentive is reversed: the platform should help us get what we need and then go do something else. Check the marketplace, find the bookshelf, arrange pickup, done. Check the calendar, see there’s a park clean-up on Saturday, RSVP, done.

This is a radical proposition in the current landscape, and it’s worth being explicit about: we are building a platform that wants us to spend as little time as possible on it. The value isn’t in engagement; it’s in the social infrastructure that exists because the platform exists.

5 Technical sketch

I don’t want to go deep on technology here—that’s for a companion post—but a rough sketch helps ground the costs.

The simplest version of this is a web app (mobile-friendly, not a native app—app store review is a bottleneck we don’t need) running on a modest VPS. Off-the-shelf options include Bonfire (a modular, federatable social platform built for community governance), Discourse (which despite being “forum software” is increasingly used as community infrastructure), or even a custom build on top of something like Elixir/Phoenix or Django with real-time features.

Ballpark costs for a neighbourhood of ~1,000 active users:

• Hosting: $20–50/month for a VPS (Hetzner, DigitalOcean, or a local Australian provider like VentraIP)
• Domain and email: $50–100/year
• Development: This is the wild card. An initial build using AI coding tools and volunteer labour could be near-zero in cash terms, or a few thousand dollars for a contractor to set up and customize a Bonfire or Discourse instance.
• Ongoing maintenance: 5–10 hours/month of volunteer or modestly-compensated time

At $5/month per household and 200 participating households, that’s $1,000/month—more than enough to cover hosting and a small maintenance stipend, with surplus for improvements.

The federation question is interesting: platforms like Bonfire speak ActivityPub, which means a neighbourhood instance could connect to the wider Fediverse (Mastodon, etc.) for broader discovery while keeping local governance local. Whether this is desirable depends on whether the community wants to be a walled garden or a node in a network. I’d bet on starting closed and opening up deliberately, rather than starting open and trying to close down later.

6 How this connects to the other projects

This platform is the social layer that the other mutual-aid projects need.

The neo-friendly society needs a communication channel that isn’t Facebook, isn’t email, and isn’t yet another WhatsApp group. It needs a place where members can discuss proposals, vote on decisions, and coordinate activities. A local social platform provides this.

The compute collective needs a community of users. The local platform’s members are the natural user base, and the AI infrastructure can in turn power features of the platform—content moderation assistance, search, translation for multilingual neighbourhoods, smart matching for the marketplace.

Together, these three projects form something like a stack: financial resilience (the friendly society), computational sovereignty (the compute collective), and social infrastructure (the local platform). Each is useful on its own; together, they’re the skeleton of a genuinely self-reliant local community.

7 Prior art

8 What’s still missing from the literature

Geek time! I would also like to research how and why people join these things, and learn how to make local platform development easier. How do users actually migrate from an enshittified platform to a community alternative when network effects penalize first movers? This is a coordination game with heterogeneous switching costs, and there’s almost no formal modelling of it. The RadicalxChange Plurality book gestures at this but doesn’t model it. If someone wants a research problem that matters, hit me up.

9 Open questions

• What’s the right first wedge for a specific neighbourhood? Marketplace? Events? Something else? This probably varies by community and the only way to find out is to ask.
• How do we verify “local” without being creepy about it? Postcode checks? Vouching? Physical meetups?
• What’s the right governance structure for a platform that’s too big for consensus but too small for representative democracy?
• How do we handle the transition when someone abuses the platform—doxxing, harassment, commercial spam? The norms for a 500-person neighbourhood platform are different from the norms for Twitter.
• Is federation (ActivityPub, etc.) an asset or a liability at this scale? Does connecting to the global Fediverse dilute the local trust that makes the platform valuable?
• Can this platform model be replicated—published as a template, the way the friendly society template is meant to be?

As with the other posts in this series: if you know about any of this, if you’ve tried something like this, or if you’re in Melbourne and want to help build it, I’d love to hear from you.

The technology is the easy part. The institutional design is the interesting part. The actual community is the hard part. Let’s start.

1 Compute trajectories

I think we should distinguish between the rate at which compute is performed and the cumulative “stock” of computation that has been done.

Let denote the compute rate — the FLOP/s of AI-related computation happening in the world at time . This is exogenous capacity: hardware, data centres, and investment. We treat it as non-decreasing and right-continuous. We make no distinction between training and inference — is unstructured compute, all of it. The more of it that is running, the more things are happening; we don’t need to model the internal structure of what kinds of workloads are running for this granularity of analysis.

The cumulative compute — total FLOPs performed by time — is

A policy lever — the safety fraction — splits the compute rate into two streams:

• Capability rate : compute aimed at expanding what AI systems can do.
• Safety rate : compute focused on ensuring AI systems do what we want.

The cumulative stocks in each pool are then

with at all times.

The function is the policy lever — the thing a civilisation chooses.

2 Two competing events

We model doom and deliverance as the first arrivals of inhomogeneous point processes — random events whose chance of firing at any moment depends on how much compute has accumulated so far:

Doom ()

An X-risk catastrophe happens. This is an irreversible absorbing state.

Deliverance ()

We achieve guaranteed alignment — a state after which X-risk from AI is effectively zero.

Each event has a latent arrival time — and are the times at which doom and deliverance would fire if nothing else intervened. But the race ends at : we observe whichever fires first, and from that moment both hazard rates cease to apply. Doom either happens first or not at all; deliverance either happens first or not at all; or neither fires within any relevant timeframe. When we write “” below, we always mean the outcome “doom fires first” — i.e.  — not the marginal probability that the doom process would eventually fire in isolation.

Each event has a hazard rate (instantaneous arrival intensity given that neither event has yet occurred — i.e. while still being in limbo) that depends on the cumulative stock of compute in its respective pool:

where are monotonically non-decreasing functions, with and . Monotonicity captures the assumption that the more capability compute we’ve done overall, the higher the catastrophe hazard per unit time; likewise, the more safety compute we’ve done overall, the higher the alignment-breakthrough hazard per unit time.

Note that these hazard rates depend on time via the cumulative compute stocks. The compute rate does not appear directly—it enters only by determining how fast and grow. So the compute growth rate matters: doubling does not double the hazard at time , but it does make reach any given threshold sooner.

x = np.linspace(0, 10, 300)
scenarios = [
    ("Linear",              g_lin(x),  h_lin(x)),
    ("Pessimistic (g convex, h concave)", g_pess(x), h_pess(x)),
    ("Optimistic (g concave, h convex)",  g_opt(x),  h_opt(x)),
    ("Sigmoidal overhang",  g_sig(x),  h_sig(x)),
]

fig = go.Figure()
for name, gv, hv in scenarios:
    vis = name.startswith("Pessimistic")
    fig.add_trace(go.Scatter(x=x, y=gv, name="g(x)  doom", line=dict(color=C_DOOM, width=3),
                             visible=vis, hovertemplate="%{y:.2f}"))
    fig.add_trace(go.Scatter(x=x, y=hv, name="h(x)  safety", line=dict(color=C_SALV, width=3),
                             visible=vis, hovertemplate="%{y:.2f}"))

buttons = []
for i, (name, _, _) in enumerate(scenarios):
    vis = [False] * (2 * len(scenarios))
    vis[2*i] = True; vis[2*i+1] = True
    buttons.append(dict(label=name, method="update", args=[{"visible": vis}]))

fig.update_layout(
    **LAYOUT,
    updatemenus=[dict(type="buttons", direction="down", x=1.0, xanchor="left", y=1.0,
                      buttons=buttons, bgcolor="white")],
    xaxis_title="Cumulative compute 𝒞",
    yaxis_title="Hazard rate",
    height=420,
)
fig.show()

3 Three outcomes of the race

The probability that the race is still in limbo at time — neither event has fired — is what survival analysis calls the joint survival function:

In words: the probability of still being in limbo decays exponentially as the hazard accumulates. Here and are cumulative hazard functions. Note the nested structure: each cumulative hazard is an integral over time of a function that itself contains an integral over time.

The race has three mutually exclusive outcomes:

where:

• — the doom event fires first,
• — the deliverance event fires first,
• — neither event ever fires.

In our model, is monotone and stocks keep growing, so : the race always resolves eventually. But “eventually” can be a very long time. Over any finite horizon, there is residual probability mass on “neither yet” — the green curve in the plots below — and this residual is a practically relevant quantity. A world where stays large at human-relevant timescales is one where we just muddle through indefinitely, which is arguably closer to most people’s baseline expectation than either doom or deliverance.

At each instant , while we’re still in limbo, the probability that some event fires within is , and the conditional probability that the firing event is doom rather than deliverance is

So doom probability can also be written

which decomposes neatly into: “probability we’re still waiting at time ” × “probability something happens right now” × “probability that the something is doom rather than deliverance.” This is the key integral of the post — everything below is about what determines its value.

alpha_race = 0.5
t = np.linspace(0, 35, 800)
c_rate = np.exp(0.1 * t)                          # compute rate c(t) in FLOP/s
Cc = cumulative_trapezoid((1 - alpha_race) * c_rate, t, initial=0)  # 𝒞_c(t)
Cs = cumulative_trapezoid(alpha_race * c_rate, t, initial=0)        # 𝒞_s(t)

# Response functions applied to cumulative stocks
ld = 0.005 * Cc**2       # superlinear doom
ls = 0.015 * Cs           # linear safety

cumhaz = cumulative_trapezoid(ld + ls, t, initial=0)
S = np.exp(-cumhaz)
doom_density = ld * S
deliv_density = ls * S
P_doom = np.trapezoid(doom_density, t)
P_deliv = np.trapezoid(deliv_density, t)
P_limbo = 1 - P_doom - P_deliv

fig = make_subplots(specs=[[{"secondary_y": True}]])
fig.add_trace(go.Scatter(x=t, y=deliv_density, name="λ_s S(t)  deliverance density",
                         fill='tozeroy', line=dict(color=C_SALV, width=1),
                         fillcolor='rgba(36,113,163,0.35)',
                         hovertemplate="t=%{x:.1f}  deliv=%{y:.4f}"),
              secondary_y=False)
fig.add_trace(go.Scatter(x=t, y=doom_density, name="λ_d S(t)  doom density",
                         fill='tozeroy', line=dict(color=C_DOOM, width=1),
                         fillcolor='rgba(192,57,43,0.35)',
                         hovertemplate="t=%{x:.1f}  doom=%{y:.4f}"),
              secondary_y=False)
fig.add_trace(go.Scatter(x=t, y=S, name="S(t)  limbo",
                         line=dict(color=C_SURV, width=3),
                         hovertemplate="t=%{x:.1f}  S=%{y:.3f}"),
              secondary_y=True)

# Annotate all three outcomes
fig.add_annotation(x=t[np.argmax(doom_density)], y=max(doom_density) * 1.15,
                   text=f"P(doom) = {P_doom:.2f}", showarrow=False,
                   font=dict(size=14, color=C_DOOM))
fig.add_annotation(x=t[np.argmax(deliv_density)], y=max(deliv_density) * 1.15,
                   text=f"P(deliverance) = {P_deliv:.2f}", showarrow=False,
                   font=dict(size=14, color=C_SALV))
fig.add_annotation(x=t[-1] * 0.85, y=0.05, yref="y2",
                   text=f"P(limbo) = {P_limbo:.2f}", showarrow=False,
                   font=dict(size=14, color=C_SURV))

fig.update_xaxes(title_text="Time t")
fig.update_yaxes(title_text="Event density", secondary_y=False)
fig.update_yaxes(title_text="S(t) — limbo", secondary_y=True, range=[0, 1.05])
fig.update_layout(**LAYOUT, height=440)
fig.show()

4 The constant-hazard-ratio case

Consider an ultra-simple sanity-check case. If the ratio is constant over time, then holds regardless of the compute trajectory — this is a standard competing risks identity. stays constant when and are linear and is constant. If and , then and . [TODO clarify]

and the cancels. The probability of a doom outcome depends only on and the ratio , not on the compute rate or how fast it grows. This is the regime in which the speed of progress doesn’t matter — only the split between capability and safety compute determines your fate. If you’ve ever heard someone say “it doesn’t matter how fast AI progresses, only whether we invest enough in safety” — this is the (very specific) model in which that’s true.

It’s also not very plausible. Let’s get complicated.

5 Interesting response curves

With non-linear response functions, this invariance breaks down. now depends on the full trajectory , because the time spent at each cumulative compute level determines how much hazard accumulates at that level. In the plot below, we compute by integration over a trajectory with exponential compute growth , for two different growth rates.

alphas = np.linspace(0.01, 0.99, 80)
t_sim = np.linspace(0, 60, 2000)

def compute_pdoom(g_fn, h_fn, alpha_val, r):
    """Compute P(doom) by integration over a trajectory."""
    c_rate = np.exp(r * t_sim)
    Cc = cumulative_trapezoid((1 - alpha_val) * c_rate, t_sim, initial=0)
    Cs = cumulative_trapezoid(alpha_val * c_rate, t_sim, initial=0)
    ld = g_fn(Cc)
    ls = h_fn(Cs)
    cumhaz = cumulative_trapezoid(ld + ls, t_sim, initial=0)
    S = np.exp(-cumhaz)
    doom_density = ld * S
    return np.trapezoid(doom_density, t_sim)

curves = [
    ("Linear",      g_lin,  h_lin,  C_NEUT),
    ("Pessimistic",  g_pess, h_pess, C_DOOM),
    ("Optimistic",   g_opt,  h_opt,  C_SALV),
    ("Sigmoidal",    g_sig,  h_sig,  C_RUIN),
]

fig = go.Figure()
for name, gf, hf, col in curves:
    for r, dash, suffix in [(0.3, "solid", "fast r=0.3"), (0.05, "dash", "slow r=0.05")]:
        pr = np.array([compute_pdoom(gf, hf, a, r) for a in alphas])
        fig.add_trace(go.Scatter(x=alphas, y=pr, name=f"{name} ({suffix})",
                                 line=dict(color=col, width=2.5, dash=dash),
                                 hovertemplate="α=%{x:.2f}  P(doom)=%{y:.3f}"))

fig.add_hline(y=0.5, line_dash="dot", line_color="#bbb", annotation_text="P(doom) = ½",
              annotation_position="bottom right")

fig.update_layout(
    **LAYOUT,
    xaxis_title="Safety fraction α",
    yaxis_title="P(doom)",
    height=480,
)
fig.show()

Interesting cases arise when and have different shapes.

Some scenarios to consider:

Convex , concave — “fast takeoff, hard alignment.” The doom hazard rate is superlinear in cumulative capability compute; the deliverance hazard rate has diminishing returns in cumulative safety compute. This is the pessimistic scenario: increasing helps at first, but as grows, the doom hazard rate eventually dominates regardless of allocation, so , and the doom outcome becomes near-certain. In this regime, slowing down itself — reducing the rate at which compute accumulates — is the only robust strategy. This is roughly the worldview behind calls for a compute pause: if the response curves are this shape, allocation alone cannot save us.

Concave , convex — “slow takeoff, tractable alignment.” The doom hazard rate saturates; the deliverance hazard rate is superlinear once we invest enough. This is the optimistic scenario: there exists a threshold of cumulative safety compute above which the deliverance outcome is almost certain. This is the implicit model behind “we just need to invest enough in alignment” — the alignment tax is finite and worth paying.

Sigmoidal and with different inflection points — “the capability overhang.” Both processes have thresholds, but they may not be in the same place. If the safety threshold is much larger than the doom threshold , there is a dangerous window where cumulative capability compute is in the steep part of while cumulative safety compute is still in the flat part of . This is arguably the scenario most alignment researchers are worried about: a period in which capabilities are advancing rapidly but safety hasn’t yet reached its own inflection point.

depends on too — “capabilities are fungible.” Safety research itself requires capable AI systems. If contributes to both and (because safety compute also advances capabilities as a side effect), the model needs modification. One could write — making doom risk a function of total cumulative compute — while . This makes the allocation problem strictly harder, because safety investment has a capability externality.

Let’s plot the first of those, then return to the last one.

alpha = 0.3
t = np.linspace(0, 25, 800)
c_rate = np.exp(0.3 * t)                               # compute rate
Cc = cumulative_trapezoid((1 - alpha) * c_rate, t, initial=0)  # 𝒞_c(t)
Cs = cumulative_trapezoid(alpha * c_rate, t, initial=0)        # 𝒞_s(t)

# Sigmoidal response to cumulative stocks
ld = sigmoid(Cc, 1.5, 4)
ls = sigmoid(Cs, 1.5, 10)

pi_d = ld / (ld + ls + 1e-12)

# Window boundaries (where doom is >0.5 of max but safety is <0.5 of max)
t_doom_on = t[np.searchsorted(ld, 0.5 * 5)]
t_salv_on = t[np.searchsorted(ls, 0.5 * 5)]

fig = make_subplots(rows=2, cols=1, shared_xaxes=True,
                    subplot_titles=["Hazard rates", "Conditional doom probability π_d(t)"],
                    vertical_spacing=0.12)

fig.add_trace(go.Scatter(x=t, y=ld, name="λ_d (doom)", line=dict(color=C_DOOM, width=2.5),
                          hovertemplate="%{y:.2f}"), row=1, col=1)
fig.add_trace(go.Scatter(x=t, y=ls, name="λ_s (safety)", line=dict(color=C_SALV, width=2.5),
                          hovertemplate="%{y:.2f}"), row=1, col=1)
fig.add_trace(go.Scatter(x=t, y=pi_d, name="π_d(t)", line=dict(color=C_RUIN, width=2.5),
                          hovertemplate="%{y:.3f}"), row=2, col=1)

fig.add_vrect(x0=t_doom_on, x1=t_salv_on, fillcolor="rgba(192,57,43,0.10)",
              line=dict(color=C_DOOM, width=1, dash="dash"), row=1, col=1)
fig.add_vrect(x0=t_doom_on, x1=t_salv_on, fillcolor="rgba(192,57,43,0.10)",
              line=dict(color=C_DOOM, width=1, dash="dash"), row=2, col=1)

fig.add_annotation(x=(t_doom_on + t_salv_on)/2, y=2.5, text="Dangerous<br>window",
                   showarrow=False, font=dict(size=13, color=C_DOOM), row=1, col=1)

fig.update_xaxes(title_text="Time t", row=2, col=1)
fig.update_yaxes(title_text="Hazard rate", row=1, col=1)
fig.update_yaxes(title_text="π_d(t)", range=[0, 1.05], row=2, col=1)
fig.update_layout(**LAYOUT, height=560)
fig.show()

6 What if safety compute is also capabilities compute?

In our base model, the doom hazard depends only on capabilities compute , and the deliverance hazard depends only on safety compute . But that separation is optimistic. Safety research often requires running large models, probing their behaviour, red-teaming, and training oversight systems — all of which also advance capabilities as a side effect. At the limit, maybe all compute advances capabilities, regardless of intent — we cannot do useful safety research without also pushing the capability frontier forward.

The externality model modifies the hazard rates to reflect this:

where is the total cumulative compute. The doom hazard rate now depends on everything — capability compute and safety compute alike — while the deliverance hazard rate still depends only on the safety fraction.

In this model, the allocation cannot reduce the doom hazard rate at all. Since holds regardless of how the compute is split, the doom hazard is set entirely by the exogenous trajectory . The only thing can do is push the deliverance hazard rate up fast enough to compete with it.

The conditional doom probability becomes

In the linear case (, , with constant ), this simplifies to

Let’s compare this with the separable model’s . The externality model is always worse: the numerator is rather than , because diverting compute to safety no longer reduces the doom hazard — it only increases the deliverance hazard. With , the separable model gives at ; the externality model gives at — we need all our compute on safety just to get even odds.

Even at , the doom hazard hasn’t gone away. It’s — because all that safety compute is also capability compute. We’re in a race where every step toward deliverance also drags doom closer.

alphas_ext = np.linspace(0.01, 0.99, 80)
t_ext = np.linspace(0, 60, 2000)

def compute_pdoom_externality(g_fn, h_fn, alpha_val, r):
    """P(doom) under the externality model: g depends on total compute."""
    c_rate = np.exp(r * t_ext)
    C_total = cumulative_trapezoid(c_rate, t_ext, initial=0)
    Cs = cumulative_trapezoid(alpha_val * c_rate, t_ext, initial=0)
    ld = g_fn(C_total)       # doom depends on TOTAL compute
    ls = h_fn(Cs)             # deliverance depends on safety compute only
    cumhaz = cumulative_trapezoid(ld + ls, t_ext, initial=0)
    S = np.exp(-cumhaz)
    return np.trapezoid(ld * S, t_ext)

def compute_pdoom_separable(g_fn, h_fn, alpha_val, r):
    """P(doom) under the separable model: g depends on capability compute only."""
    c_rate = np.exp(r * t_ext)
    Cc = cumulative_trapezoid((1 - alpha_val) * c_rate, t_ext, initial=0)
    Cs = cumulative_trapezoid(alpha_val * c_rate, t_ext, initial=0)
    ld = g_fn(Cc)
    ls = h_fn(Cs)
    cumhaz = cumulative_trapezoid(ld + ls, t_ext, initial=0)
    S = np.exp(-cumhaz)
    return np.trapezoid(ld * S, t_ext)

fig = go.Figure()
for g_fn, h_fn, name, col in [
    (g_lin, h_lin, "Linear", C_NEUT),
    (g_pess, h_pess, "Pessimistic", C_DOOM),
    (g_opt, h_opt, "Optimistic", C_SALV),
]:
    r = 0.1
    pd_sep = np.array([compute_pdoom_separable(g_fn, h_fn, a, r) for a in alphas_ext])
    pd_ext = np.array([compute_pdoom_externality(g_fn, h_fn, a, r) for a in alphas_ext])
    fig.add_trace(go.Scatter(x=alphas_ext, y=pd_sep, name=f"{name} (separable)",
                             line=dict(color=col, width=2, dash="dash"),
                             hovertemplate="α=%{x:.2f}  P(doom)=%{y:.3f}"))
    fig.add_trace(go.Scatter(x=alphas_ext, y=pd_ext, name=f"{name} (externality)",
                             line=dict(color=col, width=3),
                             hovertemplate="α=%{x:.2f}  P(doom)=%{y:.3f}"))

fig.add_hline(y=0.5, line_dash="dot", line_color="#bbb", annotation_text="P(doom) = ½",
              annotation_position="bottom right")
fig.update_layout(
    **LAYOUT,
    xaxis_title="Safety fraction α",
    yaxis_title="P(doom)",
    height=480,
)
fig.show()

The gap between the dashed (separable) and solid (externality) curves is the cost of the capability externality — the additional doom probability we bear because safety research also advances capabilities. In the pessimistic (convex ) regime, the externality model is especially punishing: the superlinear doom hazard is driven by total compute, which cannot touch.

7 What pauses do

This formulation makes the effect of a “compute pause” precise. Suppose we’re in the separable model. At time the compute rate drops to near zero: for . During the pause, the cumulative stocks and are frozen — no new capabilities, no new safety progress. But the hazard rates and are also frozen at their current values, and the cumulative hazard continues to grow at that frozen rate. Doom can still arrive during a pause — we’ve stopped accumulating new risk, but we haven’t reduced the hazard we’ve already built up.

A pause buys time in a specific sense: it extends the interval over which stays at its current value, rather than letting cumulative compute push us into a worse regime. If we’re in the flat part of (low hazard), pausing wastes the opportunity to accumulate safety compute. If we’re in the steep part of (high hazard), pausing prevents things from getting worse while the current hazard ticks away — useful only if we spend the pause changing or directly (via policy, regulation, or new alignment techniques).

In the linear case ( constant), pausing is useless: the conditional doom probability is the same before, during, and after the pause. In the convex- case, pausing in the steep region is the only way to avoid the superlinear runaway.

The “externality” variant also changes the calculus of pauses. In the separable model, a pause freezes both hazard rates. In the externality model, a pause still freezes both — but the difference is what happens after the pause. Resuming compute at any feeds the doom hazard at the same rate, because all compute is capability compute. The only lever is .

8 What this doesn’t model

This framework is minimal and stylized. Some important things it ignores:

1. Discrete actors. There is no game theory here — just a single planner choosing . In practice, the allocation is the outcome of many actors with misaligned incentives.

2. We achieve alignment but it is expensive, so we don’t use it. The deliverance event is modelled as a single point arrival — a moment at which we get guaranteed alignment. But in practice, we might have a breakthrough that gives us the option of guaranteed alignment, but it is so expensive to implement that we don’t actually deploy it.

3. Partial doom, incremental deliverance. Both events are modelled as discrete point arrivals — single moments at which the state transitions irreversibly. This is reasonable for doom (a single catastrophe), but strange for deliverance. Real alignment progress is incremental: better interpretability, verified properties, scalable oversight, each partially reducing risk. A more plausible model would replace the deliverance point process with one whose arrivals down-modulate the doom hazard rate — each safety milestone reduces rather than ending the race outright. We could also model incremental doom — each catastrophe raises the baseline risk, although that feels less natural. If we’re worried about bad-but-not-doom events, we’d probably move to some continuous badness index, like “dollar value of harm” or “number of lives lost”, rather than a binary doom/deliverance outcome.

4. The doom hazard might not be monotone in compute. If AI systems can be made robustly safe at high capability levels, the hazard might eventually decrease. This would require to be non-monotone, which is a qualitatively different model.

5. Correlation. Doom and deliverance might not be conditionally independent given the compute trajectory. Alignment breakthroughs might come from the same capability advances that increase risk. Or the opposite: compute used for safety might be the same compute used for capabilities, so and aren’t really separate stocks.

6. Optimal control. We haven’t solved for the optimal . This is a dynamic optimal control problem, because affects cumulative stocks and at all future times . That sounds fun, but it’s not worth investigating because even if the model were true we wouldn’t know the response functions well enough to solve it, and even if we knew how to solve it, I cannot imagine us coordinating to implement that solution.

7. Granular allocation of compute to many different teams or ideas with different safety/capability profiles, rather than a single aggregate . People have made the case to me that this matters. I think we might be able to produce a more granular model by allocating compute to buckets via some kind of stick-breaking process, then taking the max hazard? Definitely out of scope for this post, but maybe worth exploring in the future.

8. A world with aligned AI could still suck.

9 Connections

• The economics of how compute is allocated: Operationalizing the bitter lessons
• The survival analysis / hazard rate formalism: Survival analysis and reliability
• Point process foundations: Point processes

10 Incoming

• Decisive strategic advantage - EA Forum
• Soft takeoff can still lead to decisive strategic advantage
• On “first critical tries” in AI alignment

11 References

1 The problem with renting our cognition

If we use Claude, or ChatGPT, or one of the rapidly improving Chinese models like Qwen or DeepSeek, we are renting inference from someone else’s data centre. This is fine, in the way that renting our apartment is fine: it works until it doesn’t, and when it stops working, the failure mode is abrupt. Except worse, because an apartment is a physical thing we build here. AI compute is piped in over fragile optical fibres, through zones of geopolitical strife, from providers subject to pressure from multiple governments.

2 What “sovereign compute” looks like at a small scale

When governments talk about sovereign compute, they mean billion-dollar data centres and national AI strategies. I want to talk about something much smaller: what does it look like for a group of 25–50 people to own and operate enough compute to run a frontier-class AI model?

It turns out this is newly, nearly feasible, because of a convergence of two trends: open-weight models have gotten very good, and the hardware to run them has gotten (relatively) affordable.

There are a lot of free variables here (which model? which hardware? what trade-offs?), so I’m going to pick some baseline examples for what follows. These are slightly arbitrary, but I don’t want to bore us with a combinatorial list of options, and these are all solid choices that would work well for a small collective.

3 Removing CCP guardrails

There’s a catch with using Chinese open-weight models: they come with censorship built in. Research from Shisa.AI has documented the specific patterns: Qwen models will hard-refuse certain prompts (anything touching Taiwan sovereignty, Tiananmen Square, various stuff in Xinjiang), and increasingly, newer versions have shifted from outright refusal to controlled compliance—they’ll answer our question, but steer the framing toward CCP-aligned positions. The behaviour is language-dependent: the Shisa.AI analysis found significantly fewer refusals in Chinese than in English on the same questions (>80% fewer), suggesting the censorship is calibrated for different audiences.

For an Australian collective, this is a solvable problem. The open-source community has developed several techniques for removing these guardrails, ranging from essentially free to moderately expensive:

Abliteration (cost: ~$100–$200 AUD for models of our size) is a technique from representation engineering where we identify the “refusal direction” in the model’s internal representations and remove it through a linear algebra operation on the weights. No training required—it’s a post-processing step that takes hours, not days. Over 4,000 community-modified models have been published using this method on HuggingFace alone. It’s effective at removing hard refusals, though it doesn’t fully address the subtler framing biases.

Preference fine-tuning via Direct Preference Optimization (cost: ~$1,500–$4,000 AUD including dataset creation) goes deeper. We create a dataset of question-answer pairs where the “preferred” answer is neutral/factual and the “dispreferred” answer is CCP-aligned, then train the model to prefer the neutral framing. This addresses both hard refusals and soft steering. The training runs on rented cloud GPUs—a few hundred to a couple of thousand dollars’ worth—and the resulting model can be deployed on our own hardware permanently.

Either way, the total cost of removing CCP guardrails is a rounding error compared to the hardware. And once it’s done, it’s done—the modified model weights live on our machine, and no one can remotely re-censor them.

4 Audition before we buy

We don’t have to commit $160k on faith. Cloud GPU rental is now cheap enough that a collective can test-drive the full setup before buying hardware.

Rent a couple of H100 GPUs from a provider like Lambda Labs or RunPod for $2–$3 USD/hour per GPU. Deploy the model with the same inference software (vLLM or SGLang) that we’d use on the DGX Station. Run our actual workloads for a week or two. See if the throughput, latency, and model quality meet our needs.

Total cost for a two-week test: $500–$1,000 AUD. If the collective decides to proceed, that’s money well spent on due diligence. If it decides not to, we’ve lost the cost of a nice dinner, not a house deposit.

NVIDIA also offers DGX Cloud, which runs the exact same software stack as the physical DGX Station—same NIM inference microservices, same NGC containers, same management tooling. If workflow portability matters, this is the most seamless way to audition.

5 The NBN still sucks but not so badly that we can’t work around it

There’s a mundane infrastructure challenge that’s easy to overlook: Australian residential internet is not great. If the machine lives in someone’s house on an NBN connection—particularly one of the older HFC or FTTN links—we’re looking at multiple brief outages per day, asymmetric upload speeds that make remote access sluggish, and no SLA to speak of. For a token server, this is mostly an annoyance (our request fails, we retry), but for longer agentic workflows that run over minutes or hours, connection drops mid-session are annoying.

There’s a spectrum of options here. At one end: someone’s spare room, cheap, community-feeling, unreliable. At the other: a quarter-rack at a local colo facility with redundant fibre, expensive, corporate-feeling, rock-solid. In between: a business-grade NBN plan with a static IP and better SLA, or a 5G failover link for redundancy. The right answer probably depends on how many members are remote versus local, and how tolerant the collective is of occasional downtime.

The point to remember is that it might be nice to be as reliable as Anthropic’s US data centres during peace time, but what we actually want to beat here is the speed of access when the undersea cables are cut and cyberattacks are flying, and I think even the NBN can probably do OK against that baseline.

6 Other community infrastructure

In the previous post, I argued that neo-friendly societies should invest in counter-cyclical assets—things that retain value precisely when the state and conventional institutions are under stress.

Sovereign compute infrastructure fits this criterion pretty well:

• It becomes more valuable if geopolitical tensions restrict access to foreign AI services
• It becomes more valuable if commercial API providers raise prices or restrict capabilities
• It becomes more valuable if regulatory changes create compliance barriers to using foreign-hosted AI
• Unlike financial assets, it has direct use value—it does useful work for members every day

It’s also a natural complement to the other things a friendly society might do. The same AI infrastructure that serves our members’ professional needs can also help run the society itself—automating compliance paperwork, generating regulatory filings, processing claims, managing communications. This is the AI-assisted administration angle from the previous post, but with infrastructure we own rather than rent.

7 Replicating this model

As with the friendly society model itself, the most valuable output from a societal perspective isn’t a single collective with a single machine, but the documented, replicable process that other groups can fork.

The hardware purchase process, the model selection and decensoring procedure, the inference server configuration, the access management for members, the cost-sharing model—all of this can be packaged as a how-to guide. Publish the playbook, let others spin up their own nodes.

A network of small collectives, each with their own sovereign compute, each running their own models, would be meaningfully more resilient than any individual group relying on a single commercial provider. And unlike a data centre, a DGX Station fits under a desk and plugs into a standard power circuit (a standard Australian 10A outlet can do that, and a 20A circuit is a routine job for an electrician). The barrier to entry could be financial, not technical.

8 Legal structure

A collective that owns a $160k asset needs a legal entity. The technical companion explores the options in detail, but the short version: an incorporated association is the simplest starting point (~$200 to set up, ~$57/year), and a cooperative under the Co-operatives National Law is the better long-term fit if the model proves viable—it’s the legal form designed for groups of people who jointly own infrastructure they all use. ACNC registration as a charity is probably not applicable unless the collective has an explicit community education or digital inclusion mission. Either way, budget $2,000–$5,000 for a solicitor to review the constitution before committing members’ money—cheap insurance on a six-figure purchase.

9 Open questions

• How do we handle the operational side—who has physical access to the machine, who administers it, what happens if it breaks?
• Where does it physically live? Someone’s house? A shared office? A colo? (See broadband discussion above.)
• Is there appetite for a network of these collectives, sharing infrastructure knowledge and potentially load-balancing across nodes?
• What’s the right model governance framework? Who decides which models to run, what guardrails to keep or remove, what use policies to enforce?
• How do we handle the ongoing model updates? New versions of Qwen and other open models are released regularly; someone needs to evaluate, decensor, quantize, and deploy them.

As with the friendly society post: if you know about any of this, or if you’re interested in being part of a pilot group, I’d love to hear from you.

The hardware is available for now. The models are available. The software stack seems mature enough. The missing piece is the institutional form—the small, trust-based collective that can actually make it go.

That’s the part we need to build.

10 Related projects

• Project NOMAD - Offline Knowledge & AI Server / Crosstalk-Solutions/project-nomad “Project N.O.M.A.D, is a self-contained, offline survival computer packed with critical tools, knowledge, and AI to keep you informed and empowered—anytime, anywhere.”

1 Hardware: DGX Station GB300

2 Model selection and quantization

3 KV cache budget and concurrency

With the model in AWQ 4-bit occupying ~124 GB of HBM, we have ~128 GB remaining for KV cache (conversation context).

4 Throughput estimates

Autoregressive decode (token generation) is memory-bandwidth-bound. Each token requires loading the active expert weights from HBM:

With realistic overheads (attention computation, KV cache reads/writes, memory controller efficiency):

For agentic workloads specifically, these numbers are better than they look. Agentic patterns are bursty: the model generates a short tool call (50–200 tokens), waits for tool execution, then processes the tool result as a new prompt. The system spends much of its time in prefill (fast) rather than sustained decode (slower), and the gaps between bursts let other users’ requests interleave.

5 Inference software stack

6 De-censoring: removing CCP guardrails

This is covered at a high level in the companion post. Here are the technical details.

7 Alternative models and the agentic landscape

Qwen3–235B-A22B is the current sweet spot for the DGX Station, but the landscape is moving fast. Here are other models worth considering, especially for agentic use:

8 Cloud audition pathway

Before committing to hardware, test the full stack on rented compute. Here’s the recommended sequence:

9 Workflow portability: NIM + NGC as the abstraction layer

The single most important architectural decision for long-term flexibility: build our workflows against NIM APIs and NGC containers, not against any specific cloud provider’s tooling.

NIM (NVIDIA Inference Microservices) provides an OpenAI-compatible API that works identically on DGX Cloud, any major hyperscaler, and our own DGX Station. NGC (NVIDIA GPU Cloud) containers are the deployment units.

If we standardize on this layer, we can:

• Develop and test on rented cloud GPUs
• Deploy to our own hardware with no code changes
• Fall back to cloud if our hardware fails or demand spikes
• Migrate between cloud providers without rewriting anything

The DGX ecosystem’s portability story is genuinely good here—it’s one of the few areas where vendor lock-in actually works in our favour, because the vendor (NVIDIA) has a strong incentive to make its software run everywhere NVIDIA hardware exists.

10 Cost summary

The one-time cost amortized over three years adds roughly $55–$75/month per member in a 50-person collective. Combined with operating costs, total cost of ownership is $155–$193/month per member depending mainly on whether we host at home or in a colo facility. This is competitive with commercial API access for serious users—and with the advantages of no rate limits, no per-query metering, and full data sovereignty.

11 Risks and mitigations

12 Legal structure

A compute collective needs a legal entity that can own a $160k asset, collect monthly contributions from members, enter contracts (hosting, internet, support), and limit personal liability. Australian law offers several options. None is perfect; here’s how they compare for this specific use case.

13 Next steps

If we’re interested in forming or joining a compute collective, the immediate actions are:

1. Gauge interest: find 10–20 people who would commit to ~$150/month for sovereign AI access
2. Run a cloud pilot: $200 buys a weekend of testing on rented GPUs
3. Choose a legal structure: start as an incorporated association, convert to a cooperative if the model works (see legal structure section above)
4. Order hardware: current lead times for DGX Station are “months, not weeks”—starting the order process early is important
5. Document everything: the playbook we write becomes the replication kit for the next collective

The companion post makes the case for why. This post, I hope, makes the case that it’s how.

1 What are we in Australian law?

This is the hard part. There are several possible legal structures, each with different trade-offs. I’ll try to sketch the landscape; this is emphatically not legal advice.

2 Wargaming maximum ambition: the APRA path

OK. Let’s stop being sensible for a moment and ask: what if we went all the way? Not “pick the lightest-touch legal structure and hope nobody notices”, but “actually register as a proper APRA-regulated friendly society and use AI to make the compliance affordable.” This is the maximally ambitious version. Let’s wargame it.

3 What I still don’t know

• Whether the Broodfonds gift-based model survives contact with Australian financial services law and ATO scrutiny.
• The practical costs of running an unregistered MIS with ≤20 members—I have estimates above but no real-world data.
• Whether any FIAA-qualified actuary would actually take on a tiny fund at the rates I’m projecting, or whether professional indemnity makes it uneconomic.
• Whether APRA would even entertain a licence application from a micro-entity with this profile.
• How to structure the investment function in a way that’s both legally sound and actually useful as a crisis hedge.
• What existing Australian co-ops or mutuals are doing that’s close to this vision—and what we can learn from them.

If we know about any of this, I’d love to hear from you.

1 Cults and agency

The Antidote to Trickery is an interview with Diane Benscoter, a de-radicalisation expert, who argues that there is a distinction between being programmed by a cult and being de-programmed by a non-cult.

Is sublimating my goals to a larger group really a loss of agency?

2 Is domestication a loss of agency?

See Human domestication.

3 Coercive control

Researchers seem to agree that coercive control is a thing, and that it’s bad (Lohmann et al. 2024). So intuitively some of the things that agency should include are things that should be diminished by coercive control.

cf What the research evidence tells us about coercive control victimisation.

4 Addiction

Is addiction a loss of agency?

5 “High-agency” vs NPC

Jasmine Sun decodes the Bay Area use of the term

You can be a top-ranked competitive coder or know every world leader’s birthday by heart, but the only real metric of success is whether you can build a life you’re happy with. Cold-emailing your way into a dream job is pretty high-agency; quitting it to become a strawberry farmer is even more so. Agency is initiative, resourcefulness, a high internal locus of control. Not stressing about roadblocks and assuming you’ll figure it out along the way.

6 Attention hacking

If my attention is being hacked, do I lose agency?

7 Reward tampering

If my preferences are altered to be what some other entity wishes them to be, do I lose agency?

8 Empowerment

If I am empowered, do I gain agency?

9 Incoming

• dcsoft-yyf/CAAM

  The Carbon-based AI Analysis Method (CAAM) redefines human freedom as a “Non-Lock-in State” within a five-dimensional phase space—characterized by diffused goals, controllable leverage, native pain interpretation, accessible exit, and weak coupling. If CAAM withstands rigorous mathematical proof in the future, human rights, within this framework, cease to be abstract moral halos and become computable dynamical constraints for maximizing the accessible set in phase space.

10 References

1 A running example

We’re reusing the running example from the ES-for-beginners post. Let us consider the challenge of training a tiny neural net without backprop.

We have a simple classifier (say, a two-layer MLP). The standard supervised objective is:

Backpropagation gives us . ES assumes we won’t use that, and instead treats “run the model forward, then compute the loss” as a black box we can call:

• Input: the parameters
• Output: a scalar score (loss, reward, accuracy, etc.)

To match the usual ES maximization setup, we define a fitness function to maximize, for example:

The ES loop looks like this:

1. Sample random perturbations .
2. Evaluate fitness on the perturbed parameters .
3. Combine the results to update .

Here is the “noise scale” (how far we probe). A standard choice is Gaussian perturbations (why is it always Gaussian? It makes it feel like we’re doing SGLD):

Define a smoothed objective. This is what ES targets.

Backprop optimizes directly (when it can). ES typically optimizes a Gaussian-smoothed version .

2 Large data variants

So far this seems reasonable, but it also seems to assume full-batch losses rather than stochastic mini-batches, which, as SGD argues, are the secret to NN scaling. In the basic version, it sounds like we need the full-data likelihood, which is a non-starter.

What does ES do instead to scale to large data? What are the analogues of mini-batches and SGD updates?

I asked an LLM to soothe my qualms, and it did, citing (Lenc et al. 2019; Salimans et al. 2017)

In practice you do the same thing you do with SGD: replace the true objective (an expectation over the data distribution) with a stochastic estimate from a minibatch, and treat the resulting extra noise as part of your gradient-estimation noise.

Write the population objective as an expectation over data (example ):

(For supervised learning you can think .)

Given a minibatch , define the minibatch fitness

Now you just plug into the standard ES estimator. With antithetic pairs:

and update .

Key practical detail: use the same minibatch for every member of the population within an iteration, and for both and . This “common random numbers” idea at the data sampling level makes the difference cancel a lot of minibatch noise and makes antithetic sampling actually do its job. If each perturbation sees a different minibatch, we inject extra variance that antithetic pairing cannot cancel.

A minimal “SGD-like” recipe:

1. Sample a minibatch .
2. Sample perturbations (and use antithetic pairs).
3. Evaluate in parallel.
4. Form , optionally normalize/center fitness within the population.
5. Update .

This looks like SGD, but the gradient is estimated by perturb-and-evaluate rather than backprop. The minibatch adds stochasticity exactly the way it does in SGD: it doesn’t “break” the algorithm, but it affects the variance and therefore the batch size / population size you need for stable progress.

3 ES at very large scale

Evolution Strategies at the Hyperscale (Sarkar et al. 2025) pushes this really far, and I’m pretty fascinated by what the authors pulled off. More soon.

4 Incoming

• ESHyperscale/HyperscaleES: JAX Codebase for Evolutionary Strategies at the Hyperscale

• google/evojax

• evotorch

  • nnaisense/evotorch: Advanced evolutionary computation library built directly on top of PyTorch, created at NNAISENSE.
  • EvoTorch

5 References

1 String diagrams

Why would we bother with this? I saw Martin Biehl present work (Biehl and Virgo 2023; Virgo, Biehl, and McGregor 2021) that used the “string diagrams” formalism to run computations over Bayesian networks, and he argued it might be simpler to work with complex systems in this style.

Are patchers a special case of string diagrams? A sloppy generalization?

2 Tools

• DisCoPy, “DisCoPy is a Python toolkit for computing with string diagrams.”
• TikZiT looks like a reasonably good alternative GUI for tikz with string diagram support.

3 References

1 References

1 Reinforcement learning

The simplest and AFAIK earlilest analysis of learning to act in a causal setting applies to bandit problems (Lattimore 2017). Since then, there have been many more developments:

Oberst and Sontag (2019):

We introduce an off-policy evaluation procedure for highlighting episodes where applying a reinforcement learned (RL) policy is likely to have produced a substantially different outcome than the observed policy. In particular, we introduce a class of structural causal models (SCMs) for generating counterfactual trajectories in finite partially observable Markov Decision Processes (POMDPs).

Schulte and Poupart (2024):

Reinforcement learning (RL) and causal modelling naturally complement each other. The goal of causal modelling is to predict the effects of interventions in an environment, while the goal of reinforcement learning is to select interventions that maximize the rewards the agent receives from the environment. Reinforcement learning includes the two most powerful sources of information for estimating causal relationships: temporal ordering and the ability to act on an environment. This paper examines which reinforcement learning settings we can expect to benefit from causal modelling, and how. In online learning, the agent has the ability to interact directly with their environment, and learn from exploring it. Our main argument is that in online learning, conditional probabilities are causal, and therefore offline RL is the setting where causal learning has the most potential to make a difference. Essentially, the reason is that when an agent learns from their own experience, there are no unobserved confounders that influence both the agent’s own exploratory actions and the rewards they receive. Our paper formalizes this argument. For offline RL, where an agent may and typically does learn from the experience of others, we describe previous and new methods for leveraging a causal model, including support for counterfactual queries.

2 Predictive coding

TBD

3 References

1 Breaking the (Claude code)-(Google Chrome) tether

Claude Code has browser integration but only for Google Chrome, and they do not intend to fix that. I am not a fan of Google Chrome because it is creepy and gross. Firefox is not at all well supported, but many Chromium browsers can be forced into working.

Whether it is wise to let your browser traffic flow through the Anthropic servers I’ll leave to you; they already have your intimate thoughts though from all that Claude usage, so why not get some extra value, I guess?

Anyway, there are community-supported hacks to get non-Google-Chrome browsers working at stolot0mt0m/claude-chromium-native-messaging). I like Vivaldi.

macOS setup:

git clone https://github.com/stolot0mt0m/claude-chromium-native-messaging.git
cd claude-chromium-native-messaging
/opt/homebrew/bin/bash setup.sh

Take care before running this; for all I know it sends your API keys to Macedonian script kiddies and plays the sad trombone sound.

2 ollama tokenization versus huggingface

There are several convenient ways to invoke LLMs locally. I am learning to understand their affordances by breaking them.

First exercise: mxbai-embed-large-v1 from Hugging Face and mxbai-embed-large from Ollama are nominally the same model. The trained weights derive from the same upstream artefact. The stacks around those weights are not the same, and on real inputs the two stacks produce different outputs from the same text.

The HF path: pip install transformers, then AutoModel.from_pretrained(“mixedbread-ai/mxbai-embed-large-v1”) in our own Python process. The first call downloads the weights, the tokenizer config (tokenizer.json), and the model config from huggingface.co into ~/.cache/huggingface/hub/. Inference runs in our process, via PyTorch (or sentence-transformers, or whatever library is wrapping it). Tokenization runs in the same process, performed by HF’s Rust tokenizers library reading that same tokenizer.json. One process, one library, one set of files on disk.

The Ollama path: install the ollama daemon, then ollama pull mxbai-embed-large. The daemon downloads a .gguf file from Ollama’s registry, not from huggingface.co. That .gguf is a single binary bundling the weights (usually quantised to ~4 bits), a re-implementation of the tokenizer, and the model config¹. Inference runs inside the long-lived ollama serve process. We talk to it over HTTP on localhost:11434. Tokenization runs server-side, in C++, against the tokenizer rules baked into the .gguf.

So: same upstream weights, two stacks, two tokenizer implementations. The two ought to agree, but in practice the agreement is approximate at best. HF’s Rust tokenizers and llama.cpp’s C++ tokenizer are independent implementations. I assume the conversion that produces the GGUF tokenizer rules from the original tokenizer.json is not bit-perfect: pre-tokeniser steps, Unicode normalisation, whitespace handling, and the post-processor rules ([CLS]/[SEP] insertion) don’t all round-trip exactly. On clean prose they agree. On markdown — code fences, tables, weird whitespace — they diverge by 5% on the same input.

That divergence only matters when we tokenize with one stack and embed with the other — which is what we do if we use HF’s tokenizer to pre-truncate before sending to Ollama. The HF tokenizer says “510 tokens, fits comfortably under the model’s 512 limit.” Ollama, looking at the same text with its converted tokenizer, says “534 tokens, doesn’t fit,” and 400s with the input length exceeds the context length. There is no /api/tokenize on Ollama, so we cannot ask the server how it tokenizes anything; the server-side truncate=True flag is unreliable in 0.20.x for this model. The two paths cannot be made consistent from the outside.

So: do not mix and match. I switched to an all-transformers stack for embedding for this blog. sentence-transformers is a thin wrapper around HF transformers that loads the same weights into our Python process and tokenizes with the same tokenizer it embeds with. One model, one tokenizer, one process, no IPC, no version skew.

from sentence_transformers import SentenceTransformer
model = SentenceTransformer("mixedbread-ai/mxbai-embed-large-v1")
embs = model.encode(texts, normalize_embeddings=True, convert_to_numpy=True)

Dropping the Ollama hop also lets us set dtype=torch.float16 on MPS — a 15× speedup over fp32 with indistinguishable quality. The fp16 path only matters on GPU or Apple Silicon; CPU stays in fp32 because half-precision on CPU is slow and pointless.

import torch
gpu = torch.cuda.is_available() or torch.backends.mps.is_available()
kwargs = {"model_kwargs": {"dtype": torch.float16}} if gpu else {}
model = SentenceTransformer("mixedbread-ai/mxbai-embed-large-v1", **kwargs)

3 Reading the internet

MCPs are handy for reading the internet.

Currently my favourite is Jina, which is affordable.

• Reader API
• jina-ai/MCP: Official Jina AI Remote MCP Server

Their competitor Firecrawl is too expensive for my use case at marginal value add.

4 Google is also a bit shit at copy-pasteable content

Google broke its Markdown output in an interesting way: links don’t work in the output docs. This is particularly infuriating in my favourite Google product, Deep Research. It’s supposed to be about citations and links, and yet it can’t turn URLs into linked text (try copy-pasting the output to get inline citations if you don’t believe it).

For that reason—and because Gemini subscriptions are annoying and have shitty bundle pricing—as soon as Google released an API version of Google Deep Research, I wrote a Google Deep Research client that does all kinds of clever stuff to bypass their horrible output formatting and get the links and math and tables and code blocks out in a format I can actually use. Unlike Google’s horrible subscription options, it is pay-per-use at standard token pricing. My last research report was about USD1.70.

See danmackinlay/gemini_deep_research_client. It does quite a lot to make sure the output works, including links, mathematics, and the other things researchers actually want included in their work. Pull requests welcome.

5 OpenAI UX sucks increasingly and they do not really care about that

The ChatGPT client used to generate lovely markdown that I could use where I wanted structure (math, code, tables) and HTML for text-y unstructured things.

That was a good time. A few months ago they totally fucked it up, and they’ve shown no interest in fixing it. We can no longer copy-paste tidy, structured Markdown from ChatGPT — just the slobbery mess of HTML.

Read on if you want to fix that.

But actually, I have just decided to quit OpenAI and use a different provider, so I don’t care about this any more. They used to have the best model for advanced mathematics, but Anthropic’s Claude is now better at that for my use cases, and it has beautiful Markdown output. Anyway, OpenAI is emitting worrying signs of being unethical.

I’m leaving the following bit here for posterity. I suspect OpenAI’s interest isn’t in chat clients — that’s a legacy product they keep around while they plan on brain-computer interfaces or utility fog or something. The fix won’t come from them.

Solutions:

First, we could move to a good client like Jan that preserves Markdown on copy-paste and, as a bonus, lets us use diverse backends.

Alternatively, if we’re being lazy, open the chat we want to copy in a web browser and then use a browser extension to convert the HTML. Here are two that were recommended to me:

• Chrome: Markdown Capturer - BibCit
• Firefox: ChatGPT LaTeX Copy Fix

Partial fix: Use a macOS script to convert clipboard HTML to Markdown. Here is a fish script

function chat2md
    if not type -q pandoc; echo "Install pandoc: brew install pandoc" >&2; return 1; end

    pbpaste -Prefer public.html 2>/dev/null | \
    sed 's/class="Apple-converted-space"//g; s/<span[^>]*>//g; s/<\/span>//g' | \
    pandoc -f html+raw_html -t markdown+raw_tex+tex_math_dollars+fenced_code_blocks+hard_line_breaks --wrap=preserve -s |
    pbcopy
    echo "HTML → Markdown complete (LaTeX, code, newlines preserved)"
end

TBH, this still messes with Markdown math, but it preserves code blocks and newlines, which is a win.

6 But actually just ditch OpenAI

Claude has beautiful Markdown output. Also, OpenAI seems to be on some kind of slide into being generally evil, and I’ve noticed that my friends who work for OpenAI soon quit or stop getting invited to fun parties. Various actors, e.g. QuitGPT have been arguing that OpenAI is a bad actor. I am not qualified to assess all their claim

7 Incoming

• Granola — The AI notepad for people in back-to-back meetings

1 Historical Context

In the 1960s, amid Cold War computing advances and behavioural psychology’s shift from behaviorism, Simon positioned the artificial sciences as complementary to the natural ones, drawing on his work in bounded rationality and organization theory. He responded to debates on whether engineering and design qualified as “science,” arguing synthetic systems—like economies or AI—warrant rigorous study despite their goal-oriented, adaptive nature. This sounds like some kind of pushing the edges of the scientific method, which often includes a universality that he explicitly rejects here.

2 Methodological Approach

Simon defines artificial systems by their interface with their environments: inner states adapt to outer constraints via functions and goals, enabling prediction through simulation. This yields an information-processing view of cognition, rejecting perfect rationality in favour of empirical models testable via computers, as in his “nearly decomposable hierarchies” for complexity.

3 Philosophy of Science

Philosophically, Simon elevates design as a science—devising actions to change situations—bridging “what is” (natural laws) and “what ought to be” (goals), thus unifying the social sciences, economics, and engineering under empirical rigour. Maybe.

He critiques reductionism, favouring hierarchical structures where wholes exceed their parts.

4 Post hoc relevance

In hindsight, a lot of things look like Herbert Simon call-outs. AI Evals and ML Benchmarks are about evaluating designed systems. institution design, movement design, and incentive mechanisms are about designing socio-technical systems. Epistemic systems are about designing knowledge systems. All of this feels very relevant.

5 Incoming

• Herbert Simon and the Sciences of the Artificial, Through the Lens of a New Paradigm

6 References

1 Williams & Beer’s original PID

Williams & Beer consider a target and sources , and aim to decompose into atoms corresponding to:

• Redundant information (shared by multiple sources about )

• Unique information (available only from one source)

• Synergistic information (available only from sources in combination).

They formalize this by:

• Introducing a redundancy function defined on sets of sources.

• Requiring this redundancy measure to satisfy three axioms: symmetry, self‑redundancy, and monotonicity.

Using the lattice of “information antichains” over source subsets, they show that, given such a redundancy measure, the mutual information can be decomposed by Moebius inversion over this lattice, yielding guaranteed non‑negative information atoms.

As a concrete proposal, they define redundancy via the “minimum information” measure , which roughly takes the minimal specific information any source provides about each outcome of , averaged over outcomes.

2 Subsequent theoretical developments

While the framework (axioms + lattice) was widely accepted, the specific Williams–Beer redundancy quickly attracted criticism.

Much subsequent work has tried to preserve the Williams–Beer axioms and lattice structure while replacing with better‑behaved redundancy or intersection‑information measures.

3 References

1 OK London, whatcha got to entertain me?

I have a couple of days up my sleeve to do tourist stuff. What is the optimal tourist stuff for a guy like me to see in this town?

To clarify, I think I can best summarise what I’m interested in going to as “peak London” activities. This is an idiosyncratic definition that I hope encompasses the idea of things that I am unlikely to find elsewhere, and which would appeal to me, being specific to London and high on intimate, visceral, and personal dimensions of experience. Going to look at Westminster doesn’t count because it’s just a building—one clogged with ten thousand people taking selfies and then getting bored. Likewise, London Bridge. It’s a nice bridge, but in what sense is that experientially “peak”? How would that earn its place in my schedule?

No, for my day off I must answer the question: What’s some INTENSE LONDON BUSINESS?

I have been told that London is strong on history and Things of Interest to the Progeny of Oligarchs, so let’s lean into those for a starting point.

• Robo Victoria and Albert (not the canonical name): In east London there’s a museum which is actually the storehouse for the more famous V&A museum. It has the most polarised TripAdvisor reviews I have read. Next to it, there’s the world’s tallest and longest tunnel slide.

• Kew Gardens, hub of the ‘empire of botany’ (not the canonical name) that helped Britain dominate the world. They’ve got all kinds of weird collections, all gussied up nicely (Bonsai, Cacti…)

• The Grant Museum at UCL has a JAR OF MOLES.

• The Viktor Wynd Absinthe and Monsters schtick looks kitsch enough that I probably wouldn’t go if I lived in London, but as a tourist I feel like I’ve got a licence.

• Newspeak House

  Newspeak House, The London College of Political Technology, is an independent residential college founded in 2015. Our mission is to study, nurture and inspire emerging communities of practice across civil society and the public sector in the UK.

  Five stars! I spent all evening arguing with a German cypherpunk on her way back from the CCC, and an Italian Civic Tech developer about the technical barriers to citizen participation.

• Tube Strike

• Village Underground seems like a cute venue where community meets raves.

• Dennis Severs & His House sounds like a peak mix of fabulous history, style, and queerness.

  Dennis Severs came to Spitalfields in 1979 and bought a derelict house saved by the Spitalfields Trust. He reconfigured it to tell the story of an imaginary Huguenot family who had lived there since it was built in 1724.

  Dennis moved into 18 Folgate Street with a candle, a chamber pot and a bedroll – and the house became his life’s work.

  Just your standard 1720s/1990s pastiche.

2 London governance is cooked

London is a cabinet of governance curiosities. After more than a millennium of never-fully-resolved power struggles, it’s accumulated an incredible collection of bizarre case studies in not really redesigning anything—just adding more layers and carve-outs on top of the old ones.

Greater London looks like a single world city. Do not be fooled. It’s more like an unstable regional government that’s been created, abolished, and recreated, plus a set of corporate endowments that do things I’d normally expect a state to do.

A basic fact that points to the rest of the weirdness: “London” contains a legally distinct city inside it: the City of London Corporation governs the Square Mile. The remaining 600 square miles of the capital are governed through the Greater London Authority (Mayor + Assembly) and the 32 London boroughs set up under the London Government Act 1963 (Q: Why does the governing act mention “the Temples” so much?).

The full story of how it got this way is beyond me, but I rustled up some highlights.

3 Transport

4 Class

I’m baffled by the UK class system, as is traditional for outsiders. Being baffled by the UK class system is very much part of the recommended outsider experience. We all enjoy, I think, being confused by the cultural hijinks of a weird island nation that hasn’t had a thorough socioeconomic reset since the last time it was thoroughly invaded a millennium ago, or at least not as much as its neighbours.

So! Some notes for my own understanding.

I’m not going to talk about accents. That is a whole can of worms.

5 Attitudes to heating

My God. People here are obsessed with the high cost of their heating. I knew this before I arrived. There are adverts at every Tube station for cheaper energy suppliers. I was ready to be sympathetic to their plight. My sympathy has, er, cooled now that I’ve seen what people around me are trying to heat, and how.

At least in the places I’ve been, buildings have ancient single-pane windows and leaky doors, and these barn-like structures are heated by a constant roar of gas heaters, usually with dumb design choices like putting the radiator just below the window on an exterior wall, which minimizes how much heat the building retains. When the space gets too hot, do they turn down the thermostat? No, they do not. It seems to be completely culturally normal to open windows in winter while the heating is blasting rather than figure out how to use the thermostat controls. Some people will not be happy until the very laws of thermodynamics are repealed.

Update: Apparently I am misinterpreting this. In fact, the windows are open to air places out, not to cool them down. So the cultural weirdness is that people don’t turn the heating off when airing out the space? And they also leave the windows open for long periods of time because their buildings are simultaneously too leaky to heat but not leaky enough to breathe?

ALSO, what is it with the prevalence of gas heating? Even if you think that climate change isn’t a thing, why introduce an extra supply-chain dependency on imports? And why not get electric heat pumps so we can have heating in winter and cooling in summer?

IDK, I think they just aren’t very good at living in the climate they have, no matter how I try to rationalize the specific ways this manifests.

6 Bargelyf

So many houseboats and bargehomes on the canals!

In contemporary London, living on a canal boat is less a heritage reenactment and more a housing market adaptation. The London Assembly’s “Moor or Less” report notes that rising rents and house prices have pushed more people onto the water, estimating up to ~10,000 people living across London’s waterways (it cites roughly 100 miles of canals plus 42 miles of the Thames). Fun thing to note: the number of boats rose faster than the supply of moorings and essentials like water points and waste disposal.

The institutional setup is characteristically baroque English legacy weirdness. Much of the network is run by the Canal & River Trust, a charity created in 2012 from British Waterways’ assets. To live aboard you need a licence; if you don’t have a lawful “home mooring”, the fallback is “continuous cruising” under British Waterways Act 1995, section 17: the Trust says you must be on “bona fide navigation” and (generally) not stay more than 14 days in a given neighbourhood (CRT guidance in a given neighbourhood).

“Permanent” London moorings are scarce and often priced like the kind of scarce urban land rights that boats are supposed to supplement; the Guardian’s 2019 account describes CRT leasing many moorings via auctions and reports mooring costs that can reach eye-watering levels in prime spots. If you’re a continuous cruiser, the bureaucracy treats “needing to stay within commuting distance of work or school” as an unacceptable reason to linger, so “moving house every fortnight” sounds onerous. Even though you’re afloat, council tax can apply to houseboats as domestic property, by the logic that houseboat crowding still puts pressure on the council.

They look charming though.

7 London is biggggg

Elsewhere discussed: Urban scaling laws, which predict the distribution of city sizes—for a given nation-state, how many people are in the biggest city versus the second biggest, etc.

London breaks those laws; it makes up too damn big a chunk of the UK’s population and economic output to fit the usual patterns. The term of art for this is “London behaves like a primate city”: it is so dominant in population and economic output that it sits off the top end of the usual scaling laws that relate city size to indicators like GDP or innovation. When you plot European cities on the standard superlinear urban scaling curves, London (and IIRC its French counterpart, Paris) tends to appear as an outlier “dragon‑king” megacity rather than just the next step up in a smooth hierarchy (Arcaute et al. 2015; Bettencourt and Lobo 2016). Something about being the former capital of a global empire, perhaps?

8 Cartography

A good city to map, based on how many maps have been made.

Everyone in London speaks in postcodes. “I live in NW3—can we get coffee there?” is a thing we could say. Notably, the print shops I mentioned above have postcodes listed. There are many guides to decoding the postcodes, but here is the one I used: Decoding London Postcodes, The Easy Way. tl;dr: NW9 is northwest of the middle and farther away from that middle than NW1. We can guess most of the rest.

There is of course a cool history; the modern postcode grew out of an older postal district system. Of course, it totally clashes with other administrative units.

• Iris Murdoch’s London (i.e. from her books) has been mapped. Much Soho wow.
• Folded maps by Urban Good Paper maps! of urban nature and footways and such! Made by a lovely chap I met over haggis, Charlie Peel.
• Of course I should mention Beck’s iconic Tube map
• Lauren Leek’s map of underrated eateries
• London Inequality Explorer
• Relatedly, Lauren Leek’s Latte Line map, with Political economy back story
• londonmaxxxing, the london tech “heat” map

9 Sexy, queer London

• Of course I should have mentioned Damien Frost’s exquisite documentation of the Night Flowers (Frost 2016)
• WAX feels very London: a sex‑positive events empire that built its own app, launched it with a big multi‑brand party at Heaven in 2023, and iterated it into WAX/WeAreX— a hybrid dating‑and‑events platform for kink‑ and queer‑friendly, sexy-time communities that treats London’s nightlife as its home turf while steadily scaling outward. Includes adorably local features such as integrating with the National Health Service STI tests. Looks like excellent community design for, like, bangs. As such, I’m calling this a distinctive local sexy institution.
• The Institute of Sexology is a literal sexual institution, at the Wellcome Collection. Damn, I should go.

10 London fiction

• Fallen London “Forty years ago, London was stolen by bats.” —Local interactive gothic horror comedy

11 Material goods

• Freecycle: Museum Freecycle UK Group.
• UK fashion market Vinted is the best I have seen so far. Not “good” in the sense of “this website is well-designed and easy to use”. No, it’s a total retrograde piece of shit. But the market is good despite that, because 80 million British party animals are flogging their glad rags on there.

12 Coffee

Survivable, albeit in a grim joyless kind of way where we are either drinking burned coffee pod sludge, or eternally questing for the 1 in 20 cafés that can hire baristas worthy of the name.

13 References

1 References

1 Statistical methods for evaluating AI systems

Historically, we’ve done this remarkably poorly. Many published works notoriously ignore construct validity, randomized controlled trials, statistical power, multiple comparisons, and other basic tenets of experimental design and analysis. Many have written on this theme; see e.g. (Biderman et al. 2024; Eriksson et al. 2025; Gevers et al. 2025; Salaudeen et al. 2025).

• BetterBench | Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices (Reuel et al. 2024)
• mo-arvan/chatbot-arena-analysis

2 Inspect

A neat software framework from the UK AISI for evaluating LLMs and other AI systems.

• Inspect
• UKGovernmentBEIS/inspect_ai: Inspect: A framework for large language model evaluations

Inspect can be used for a broad range of evaluations that measure coding, agentic tasks, reasoning, knowledge, behavior, and multi-modal understanding. Core features of Inspect include:

3 Incoming

Exploring LLM evaluations and benchmarking | genai-research – Weights & Biases

4 References

1 Maximin updates over set priors

As far as I know, this is the classic approach. Informally: we have a set of prior distributions representing our beliefs. When we see data, we update each prior using Bayes’ rule to get a set of posteriors. When making decisions, we consider the worst-case expected utility across all posteriors in this set and choose the action that maximizes this worst-case utility.

Easy to say, but I haven’t really used this myself and suspect it’s annoying in practice (Camerer and Weber 1992; De Bock 2020; Giustinelli, Manski, and Molinari 2021; Hayashi 2021; Walley 1991).

2 PAC-Bayes methods

There’s a large body of work on this; I’m not an expert (Catoni 2007; Haddouche and Guedj 2022; Rivasplata et al. 2020; Rodríguez-Gálvez, Thobaben, and Skoglund 2024; Sucker and Ochs 2023; Thiemann et al. 2017).

PAC-Bayes (Probably Approximately Correct Bayesian) methods offer a theoretically grounded way to aggregate misspecified models in the M-open setting, giving high-probability generalization bounds without assuming realizability. They originate from (McAllester 1998, 1999) and were sharpened by Catoni (2007). The bounds control the expected risk of a posterior over hypotheses via the KL divergence to a prior, so we can, in principle, do robust model selection or ensembling even when the true data-generating process lies outside the model class .

I’m unclear whether all stacking bounds are in fact of PAC-Bayes type.

PAC-Bayes seems to justify techniques like stacking by quantifying how well a data-dependent posterior generalizes: for concreteness, Catoni’s bound states that for bounded losses , with probability over data of size , prior , and posterior . This sidesteps M-closed assumptions in some sense, but don’t ask me for details.

Tools like paccube or pacbayes.py implement optimization of these bounds for neural nets and beyond.

3 Infrabayesianism

Infrabayesianism — I presume it’s different from standard set-prior approaches, but I don’t really know it well enough to say how.

It starts with the same assumption as M-open Bayes: the true state of the world is likely outside an agent’s hypothesis space. They argue this to be especially critical for embedded agents—agents that are part of an environment vastly more complex than they are. An AI can’t model every atom in its server room, let alone the universe, so its world model is necessarily incomplete.

I confess I don’t follow that emphasis myself — I also can’t model every atom in anything I study, and I get by without infrabayesian reasoning. I should re-listen to Vanessa Kosoy on this theme. Infrabayesianism is nonetheless motivated as a framework for the future of AI systems that must navigate a world of deep and unavoidable uncertainty.

Where Bayesianism uses a single probability distribution to represent belief, Infrabayesianism uses infradistributions—which are convex sets of probability distributions.

Instead of saying, “The probability of rain is 40%,” an infrabayesian agent might say, “The probability of rain is somewhere between 30% and 60%.” I don’t know how this differs from the above definitions of imprecise probability, which directly capture the agent’s uncertainty and acknowledge the limitations of its model. This allows for more robust reasoning by considering a range of plausible worlds rather than committing to a single, likely-wrong one.

The framework apparently provides update rules and decision-making procedures (like minimax or upper-expectation reasoning) that are philosophically robust, ensuring the agent doesn’t discard useful information and can handle deep uncertainty. I haven’t used any of these in practice yet, so I’ll refrain from offering too much opinion.

4 References