Optionality as an end in itself

1 Formal versions

2 Intuitive versions

3 Why might optionality be a moral good?

A few candidate arguments, in increasing order of how much weight they bear:

1. Instrumental. Optionality is a low-regret proxy for whatever the good turns out to be. If we cannot identify the target, preserving the ability to aim is the next best thing. Optionality lets us defer the hard moral philosophy.
2. Aggregative. There are many possible goods, they are not commensurable, and we cannot pick one. The union of futures realising different goods is therefore better than any single future, in something like the Dixit-Pindyck sense of option value (Dixit1994Investment?), applied to ethics rather than capital budgeting. This crops up in environmental economics of natural resources: the preservation value of an ecosystem includes the option value of being able to use it later under future preferences and information we do not yet have (Weisbrod 1964; Krutilla 1967; Arrow and Fisher 1974). The optionality-as-moral-good intuition is older than the long-now-vintage rebranding suggests.
3. Constitutive. The open-endedness of possibility is itself the thing we value. A frozen optimum, even an optimal one, is dead in some morally load-bearing sense. This is the open-ended intelligence intuition translated into ethics: a process worth running for a billion years is one that keeps generating novelty, not one that converges.

Each of these is more stringent than the last. Most arguments I see in this neighbourhood — including the Long Now piece — are vague about which ones they buy.

4 Compared with utilitarianism

An obvious competitor is utilitarianism, which already comes with a moral arithmetic. Both are consequentialist ethics — actions are evaluated by their downstream effects on the world; we just disagree about which downstream effects matter. Deontological and virtue-ethics objections to consequentialism apply equally to both, and that conversation is not the one I am having here. The family argument is internal. Four observations on it.

First, optionality can be made to look like a particular utility function. Define \(U(s) = \log |\text{reachable futures from } s|\), or some entropy-like proxy thereof, and a maximiser of \(U\) is a maximiser of optionality. So we could call this just a special utilitarianism, with a specific if weird utility function. The empowerment literature does flag this — empowerment is sometimes called a pseudo-utility for exactly this reason. If optionality is utility-with-extra-steps, what is the relabelling buying us?

One answer is: it is utility with a structural commitment, namely non-substitutability of futures. Standard utilitarianism lets us trade distinct possible futures for a higher aggregate score; an optionality-style utility refuses, because losing a future means losing access to whatever value lives in it, and that loss is not compensated by extra value somewhere else. This is closer in shape to a maximin or a Kelly criterion than to a standard expected-utility maximisation. Hansson (1997) argues for maximin (rather than minimax-regret) as the formalisation of the precautionary principle, on grounds that look structurally identical to the ergodicity argument in the next paragraph. I have grumbled about the wider problem in utility if we must. It resembles, for example, KL-regularisation in a reinforcement-learning objective, which is a kind of “stay close to the prior” optionality preference, if we interpret the KL term not just as a regulariser but as the utility function.

Second, ergodicity economics argues that classical expected-utility reasoning is the wrong frame for systems where the gambles are non-ergodic — i.e., where ensemble averages systematically diverge from time averages. Most things we care about, including civilisations, have this property. Under non-ergodicity, the time-average of a multiplicative process has a built-in pull toward ruin avoidance — do not bet so much that there is a chance of busting out of the support — which is structurally identical to the optionality preference. Some of the disagreement between utilitarianism and optionality may be a disagreement about which dynamical regime we are in, not about ultimate values.

TODO: write something about how ergodicity still tends to have something like stationarity baked in, and we do not ahve even that if we a “learning the world” as we go.

Third, aggregation across agents. Utilitarianism’s default aggregation is linear: \(U_{\text{total}} = \sum_i u_i\). The arithmetic is additive, the agents are commensurable, my utility trades for yours one-for-one. Information-theoretic optionality does not aggregate that way by default. Joint empowerment \(I(A_1, \ldots, A_n;\, S’)\) has interaction terms — synergy when agents together can reach futures none could alone, redundancy when their action sets overlap on the same parts of the future. In general \(I(A_1, \ldots, A_n;\, S’) \neq \sum_i I(A_i;\, S’)\), and which side of that inequality is bigger depends on the structure of the world, not on a moral choice. We can force linear aggregation by definition — assign each agent a scalar empowerment and add them up — but at that point we are doing utilitarianism with a particular intermediate quantity, not optionality at the collective level. Whether the natural sub- or super-additivity of joint information is closer to the moral truth than utilitarianism’s linearity is, AFAICT, undertheorised. The non-additivity is also why the multi-agent empowerment problem stays unsolved: there is no canonical “we” to aggregate over without first making a structural choice about whether collective optionality is a sum, a joint, or something else.

Fourth, optionality dodges some of utilitarianism’s classical headaches. It does not produce a Parfit-style repugnant conclusion in the standard form: it has no built-in preference for vast populations of barely-distinct futures over small populations of richly-distinct ones, because both directions can be folded into the entropy-over-futures measure depending on how the distance metric is chosen. It does not endorse extinction by negative-utilitarian logic, because extinction has measure zero in reachable-futures space.

It also dodges the standard form of Pascal’s mugging — the longtermist worry that tiny probabilities of astronomically large future utilities should dominate present moral reasoning. Pascal-mugging gets its leverage from the multiplicative structure of expected utility (\(EV = P \times U\), with \(U\) unbounded above). Optionality measures are typically bounded — channel capacity by \(\log |S’|\), archive entropy by archive size — and the ergodicity-economics move explicitly rejects ensemble-average reasoning that lets tail outcomes dominate. Tiny-probability astronomical futures simply do not get the same purchase on the maximand. Tarsney (2025) makes a parallel move within utilitarianism itself, capping expected-value reasoning under sufficiently high background uncertainty; the structural worry is similar but the bound is imposed as a side constraint rather than built into the framework.

OTOH the longtermist worry has a dual that does seem to apply. A sufficiently committed optionality-maximiser will trade present welfare for future option-value, which is its own form of fanaticism even if structurally distinct from utilitarian Pascal-mugging. The Wong & Bartlett asymptotic-burnout argument (Wong and Bartlett 2022) can be seen as a longtermist claim cast in optionality terms: pull back from \(\beta > 1\) scaling now in order to keep cosmic-timescale option-value alive later. The asymmetry between present sacrifice and future possibility is the same asymmetry that makes utilitarian longtermism uncomfortable; the framework just bounds how bad it can get.

5 Optionality catastrophes

Empowerment is also exactly the convergent instrumental goal that the AI-safety literature worries about — Omohundro’s basic AI drives (Omohundro 2018), Turner et al. on optimal policies seeking power (Turner2021Optimal?). An agent that preserves its own future optionality can be, by the same arithmetic, foreclosing ours. “Keep options open” is symmetric across agents only when there is no resource competition; there is, so it is not. The information-theoretic version of “live and let live” ends up looking a lot like imperialism if we give it enough time and compute. Even if optionalities are not summable, we can still argue about the weighting.

That is the awkward thing about the Long Now framing too: civilisational optionality is supposed to be ours collectively, but in practice the operationalisation would need to cash out as some specific institutional configuration that — surprise — happens to coincide with the favoured policy of whoever is doing the operationalising. I do not have a clean way out of this. The empowerment literature has not solved the multi-agent version in a way I find morally satisfying.

There is also a more boring objection. Every option preserved is a commitment foregone. Hamlet does not get to be a tragic hero by maximising his option value. The civilisational analogue of “preserve optionality” can presumably quietly slide into “do nothing decisive,” which is itself a foreclosure of futures. The dual to the asymptotic-burnout case in returns to leviathan is not perpetual deferral but homeostasis at a deliberately chosen state — and choosing requires closure. Optionality without any closure might be its own dead end.

The Hamlet objection above is general. There is a sharper objection in the EA-adjacent literature that targets a particular narrow use of the optionality argument — a use that this post has not been advancing, but which it is worth disambiguating from.

A common longtermist deployment runs: “do not foreclose options now — if civilisation turns out to be going badly, future agents can always choose to wind it down later.” This treats optionality narrowly, as the option to perform some specific corrective action (“end civilisation,” “halt AI research,” similar) at a future date. The bite — laid out at ‘The option value argument doesn’t work when it’s most needed’ on the EA Forum, with s-risks (suffering-risks) as the load-bearing example — is that in the bad futures where this corrective option would matter most, the option is not in fact exercisable. Those future agents are inhabitants of the bad future. They will have whatever distorted values, weakened coordination capacity, and corrupted institutions that bad future has produced. They are not, generically, going to be the sort of people who calmly assess that civilisation has gone wrong and collectively elect to stop. The capacity to reflect on civilisational direction, coordinate across populations, and execute a large-scale “we have decided to wind down” decision is the kind of thing that severe failures destroy on their way to becoming severe.

The framework I have been working with throughout this post is broader than the deployment the bite targets. Optionality here is the volume of reachable futures, not the option to perform a specific corrective action later. Under the broad reading, dystopia-bound trajectories are foreclosures — they reduce the reachable-future-volume — and an optionality-maximiser should argue against the trajectory long before the dystopia arrives. So the EA Forum bite mostly does not land against the broad framework: the work it would need to do against narrow “preserve the option to wind down” deployments is the work the broad framework would already have done, earlier, on its own terms.

There is residual force, though. The broad framework only opposes dystopia-bound trajectories if dystopia in fact has low reachable-future-volume. That is plausibly true for some dystopias and less obviously true for others — a stable totalitarian regime with lots of internal variation might preserve a respectable amount of state-space volume, and the framework would not flag it as problematic on its own terms, even where other moral intuitions would object. So a gap survives even on the broad reading: bad futures with high option-value, where the optionality measure misses what we (somehow, on independent grounds) consider wrong.

I do not have a clean answer to this residual gap. The shape of the answer I would want involves distinguishing options-that-preserve-the-capacity-to-choose from options-that-do-not, and counting only the former. But that distinction smuggles in all the moral commitments the framework was supposed to dispense with: it requires us to know what kinds of futures contain exercisable optionality, which is a much harder question than counting the futures.

Mind you, classic utilitarians get to dispense with assuming this kind of thing away all the time, so maybe this is at least normal for a moral framework.

6 Threads to chase

The Wong & Bartlett asymptotic-burnout argument (Wong and Bartlett 2022) is an interesting worked example of the civilisational version of the optionality claim available — it gives a quantitative reason to pull back from \(\beta > 1\) scaling, and “homeostatic awakening” might be approximately what the Long Now piece is getting at. I covered that one in returns to leviathan.

Other threads:

• What the multi-agent generalisation of empowerment looks like formally when the agents have to share an environment. The technical literature is thin and the moral literature seems unaware that the technical literature exists.
• Robust Decision Making (Lempert, Popper, and Bankes 2003) in policy analysis under deep uncertainty operationalises optionality preservation in policy. There is presumably useful cross-pollination between RDM and the formalisations above that I have not done.
• Ecosystem robustness and biodiversity as the biological cousin of the civilisational claim. The same maths probably applies; ecologists have been there longer.
• Does the moral worth of chickens uneaten scale linearly in number of chickens? — i.e., does optionality over animal-welfare futures aggregate the same way utility supposedly does, or does it have its own pathologies of aggregation?
• The connection back to antifragility (Taleb; ACX): is antifragility a strict superset of optionality, a strict subset, or a sibling concept? My current guess is sibling, but I have not done the work.
• How do individual optionality (empowerment) and collective optionality come apart, and what the moral arithmetic looks like in those cases. I suspect this is where the substance of an “optionality ethics” resides.
• The notion of generalised Kelly betting seems like a cool natural result to arise from optionality. I suspect it also arises prefigurative politics.

7 References