Agency wat

Dan MacKinlay

2026-07-13

A talk for Human-aligned AI Summer School 2026.

Terminology note

• “agent” is a terminology tarpit.

• other tarpits we leave out of scope for now

  • alignment
  • empowerment
  • flourishing
  • rationality

Aligning what to what?

\[ \underbrace{\operatorname{align}}_{\text{not covered}}( \mathbf{A}, \mathbf{B} ) \]

The most basic alignment claim has the form “A is aligned to B”. What is an admissible entity in this relation? Note the type of the relation: alignment is entity–entity — the event title says “human-aligned”, not “mission-aligned”. Leaving aside what align means, intuitively this is a relation between “agents”.

In practice align(A, B) in AIs is implemented via some proxy target, e.g. manufacture a target out of B’s — extracted preferences etc — and point A at it. Maybe talk about this?

Could discuss boundaries here? In AI agents, is A the weights, the model, the model plus harness plus memory plus tools, the deploying company, or the whole socio-technical ecosystem? Or what?

Reasons to formalize agency

1. We do it anyway, implicitly, when we talk about aligning
2. In practice, modernism solves for something, so maybe agency is better than the alternatives (?)
3. It seems to be important in our notion of moral patienthood

tl;dr agency ends up playing implicit normative and descriptive roles regardless, but it is not clear exactly what it is

Agents seem to be the entities in alignment problems

• \(\operatorname{align}(🧑, 👶)\)
• \(\operatorname{align}(🧑, 🧑)\)
• \(\operatorname{align}(🧑, 🤖)\)
• \(\operatorname{align}(🤖, 🤖)\)
• \(\operatorname{align}(🧑, 🏟️)\)
• \(\operatorname{align}(🪨, 🪨)\)

Implicit theories of agency

Many modes of AI risk have an implicit theory of agency, e.g.

• instrumental convergence (Omohundro 2008) — Expected Utility Maximizer
• power-seeking (Turner et al. 2021) — POMDP solvers
• simulators (janus 2022)— unsupervised predictors
• Roleplayers (Shanahan, McDonell, and Reynolds 2023) — unsupervised predictors
• …

Whether they transfer to the systems that actually exist (LLM plus harness plus RLHF weirdness plus …) is an open question.

There are other concepts we’d love to be crisper — “intelligence”, “good”, “flourishing”. This is less interrogated relative to its importance, e.g. the orthogonality thesis is about intelligence and value; value-specification is a named subfield. Agency is systematically overlooked — everyone imports a tacit model from their home discipline, and it feels too obvious to theorize.

Implicit theories of human agency

Pre-AI fields care a lot about agency

• welfare economics (A. K. Sen 1977) — the revealed-preference chooser
• bioethics & law (Beauchamp and Childress 2019) — the autonomous, competent chooser
• psychology (Frankfurt 1971) — the authentic self
• capability approach (A. Sen 1985; Nussbaum 2011) — agency as effective freedom

Welfare economics: the agent is the revealed-preference chooser reconstructed from choices, and revealed preference tracks welfare — “Rational Fools” (A. K. Sen 1977) is one objection.

Bioethics and law: the agent is the autonomous, competent chooser presupposed by informed consent (Beauchamp and Childress 2019).

Competence exclusions — minors, psychological incapacity — are operationalizations of the forensic notion of agency: (Locke 1689).

Psychology: therapeutic and folk talk of the authentic or true self presupposes that a person’s real preferences can be told apart from distortions — addiction, coercion, manipulation; higher-order desires (Frankfurt 1971) are a formalization. This gives us a reference for “manipulated preferences”.

Capability approach: (Nussbaum 2011; A. Sen 1985) Sen uses agency and Nussbaum capabilities for agency aspect of a person, distinct from the well-being aspect.

Many harms we care about seem to implicate agency — coercion, addiction, attention capture; not all models can capture all these.

Models of agency to be confused about

• decision theory — VNM / EV maximization · game theory
• machine learning — RL / POMDP · AIXI · intelligence-is-optimization · agents-vs-devices · Discovering Agents · LLM + harness · simulators / role-play
• information theory & cybernetics — cybernetic control · empowerment (Klyubin–Polani) · info-theoretic individuality
• physics & biology — dissipative / autopoietic · active inference · enactivist / biogenic
• psychology — empirical human psychology
• law & polity — forensic (fit to be punished) · group / institutional
• folk — “high agency” (Bay Area)
• … a fog of others

Hopefully the audience comes from several different disciplines, and the notion that there are many models will be less than obvious because it might be less contentious from a given discipline. (“obviously, agents are … X”) Fun observation

• agent is a count noun, the object of detection (Discovering Agents, agents-vs-devices);
• agency is a mass noun, a graded quantity that invites measurement and then maximization (empowerment, “high agency”)

Models doing live safety work: VNM (the risk theorems), RL/POMDP (the systems we train), LLM plus harness (the systems that exist), empowerment (both roles at once), Discovering Agents (the detection procedure) (Kenton et al. 2023), collective/institutional agents (gradual disempowerment) (List and Pettit 2011; Kulveit et al. 2025), simulators (janus 2022), human psychology (the thing we align to).

Theoretical anchors: AIXI (anchors unbounded, solipsistic, boundary-primitive) (Hutter 2005b), active inference (anchors the boundary-derived claim, contested as it is), agents-vs-devices (anchors observer-relativity) (Orseau, McGill, and Legg 2018), cybernetic control (some kind of floor case: the thermostat), information-theoretic individuality à la Krakauer–Flack (anchors boundary-derivation from data) (Krakauer et al. 2020).

Enactivist/biogenic : Barandiaran, Di Paolo, and Rohde (2009) (individuality, interactional asymmetry, normativity) and Levin (2022) (the cognitive light cone).

Intelligence-as-optimization would be nice to go deep on; Is VNM a subset? Seems so.

Cosider Yudkowsky’s cross-domain optimization power (“squeezing the future into narrow high-value regions”) (Yudkowsky 2008), Legg–Hutter universal intelligence (Legg and Hutter 2007), Flint’s “ground of optimization” (Flint 2020). Note the measurement subtlety that makes it observer-relative: optimization power is only defined relative to a preference ordering someone supplies.

“high agency” as the Bay Area folk virtue. Maybe cashes out in self-efficacy (Bandura 1977) or internal locus of control (Rotter 1966).

Forensic agency — Locke argues “person” is a forensic term (Locke 1689) — an agent is whatever is fit to be held answerable and punished; cf Hart’s capacity-responsibility (Hart 1968) and Strawson’s reactive attitudes (Strawson 1962), and legal personhood (rights-bearing, corporations included) is its civil cousin.

Forensic deterrence seems to coincide with the Discovering Agents definition (Kenton et al. 2023) — punishment works on a system whose policy adapts when consequences change. Criminal law is a causal interventionist model - exempting children and the insane precisely because their policies do not “adapt” the right way.

The fog: mesa-optimization (Hubinger et al. 2021), shard theory, thermodynamic accounts (Still et al. 2012; Rosas et al. 2024), the phenomenology of the sense of agency, agent-foundations constructions like QACI. Definitely other stuff.

Agency as a cluster concept

For a competent, adult human, these all coincide.

For machines, not so much.

Nor for superorganisms.

Why might agency feel too obvious to theorize? Because on the paradigm case — the healthy adult human — almost criterion coincides: the metabolic individual is the forensic person is the decision-theoretic chooser is the boundary is the empowerment channel is the psychological self.

Agency is a cluster concept — a homeostatic property cluster (Boyd 1991): a bundle of properties held together not by definition but by a common cause. Here the common cause is being a singular, evolved, metabolic, strategic organism. This correlates metabolic individuality, unit-of-selection, locus-of-behaviour, and having-a-perspective.

Extant examples of these factors failing to coincide: forensic agency exempts children and the insane (biogenic agents, yet not answerable); split-brain patients, committees and mobs and other group agents (List and Pettit 2011), the extended mind…

Collective organisms are a test also. A colony metabolizes as a unit and the ants individually, so individuality is some kind of graded quantity (Krakauer et al. 2020).

New AI “agents” (e.g. coding agents) look adopt some of the features of agency, but far from all. Gradient descent selects the weights (so they optimize) but they do not metabolize; the running agent acts but does not persist; the firm metabolizes and persists but does not (?) phenomenally act. They notably exclude any of the forensic ones at the individual level. Canonical answer doesn’t exist for those; just disciplinary perspectives. The human case let us conflate the criteria and so they (we) seem obvious.

Axes of variation

• boundary — agent/environment cut: primitive ↔︎ derived
• multi-agent — solipsistic ↔︎ multi-agent-first
• observer-relativity — agency intrinsic ↔︎ ascribed
• boundedness — unbounded ideal ↔︎ substrate-bound
• warrant — descriptive (is) ↔︎ normative (ought)

First draft taxonomy. Kinda bullshit, but I think it at least maps the space. And we can plot ’em! Next best thing to unifying them is showing that they are not obviously the same at all.

The first plot is boundary × multi-agent, (bonus! coloured by boundedness) — can the theory find agents in the world as it actually is: embedded, plural, resource-bound? The second plot is warrant × observer, coloured by normative/descriptive usage of the model — could the theory say what “good” agency is, in a form that survives being written into a specification? Multi-agent and observer-relativity are suspiciously correlated. Candidate for merging? Warrant × observer-relativity is the nearest-to-orthogonal pair. Is that for a good reason? Is agency a property of the system, or a fact about system-plus-describer?.

Descriptive model analysis: can the account find agents in the world as it is — embedded, plural, resource-bound? Where we cut the world of candidate agents into categories decides what we align, eval, and regulate. Not that nothing is top-right (boundary-derived × multi-agent-first) and bright (substrate-bound). So nothing

• gives us agent/world boundaries, and
• is multi-agent-first, and
• on a finite substrate (substrate-bound).

Research direction?

Near-misses: Discovering Agents reaches the corner but is pale (unbounded); autopoietic is bright but solipsistic; simulators are close near-tenant; Polani is the nearest occupied cell. NB, we might have a group-level agency model which does fit in that corner: transaction-cost economics (Coase 1937; Williamson 1985) — firm boundaries derived from cost, and AI crashing those costs reshuffles them.

Markov-blanket makes a naked boundary-derivation claim, albeit contested (Bruineberg et al. 2022).

Normative work analysis: x is warrant (is/ought), y is observer-relativity (intrinsic/ascribed), colour is the job it does. A good value target would be in bottom-right — normative and intrinsic enough to specify without observer-subjectivity.

VNM, AIXI, “high agency”. The formal occupants of that corner are unbounded idealizations we suspect to be fragile optimization targets.

Forensic sits alone top-right — not sure if that is right.

Is⇄ought crossings

We implicitly move between normative and descriptive all the time.

We move betwwen normative and descriptive uses of models all the time.

Ought→is: VNM coherence is a theorem about consistent preferences, deployed as a prediction that advanced systems maximize EU. This might be true when Dutch book arguments actually penalise inconsistent agents (Shah 2018; Ngo 2019; Thornley 2025; Petersen 2023; Gustafsson 2022). Instrumental convergence (Omohundro 2008) seems to be both Is→ought: empowerment could be measure, deployed as a target (“empower the humans”). The premise is that optionality is “good”.

Agency as target

• It seems like we generally think of agency as a good
• High modernism works best if you can measure a good
• Can agency be made objective enough to support this role?
• Maybe agency is even a natural kind?

Here the two jobs collide: is-work gives stance-relative, observer-indexed agency; ought-work wants an optimizable good. If agency is an ascription, what does “protect the humans’ agency” even target? Real-patterns helps — the compression facts are objective — but a strong optimizer can hit the measure of the pattern, not the pattern. Best shape of an answer we know: List’s scale-relative realism (List 2019) — real relative to a coarse-graining, objective once the level is fixed. Not an answer yet. Good place to open the discussion.

Fragility of value, agency edition

Specify “human agency” slightly wrong → agency-theatre

Find a “placebo button” image? Gradual disempowerment erodes our agency, so agency isn’t only the threat model — it’s a candidate value to specify. Then fragility of value bites (Yudkowsky 2011): specify it slightly wrong — say, as raw empowerment — and a strong optimizer ships agency-theatre. A human in a control room of buttons they don’t understand; preferences quietly reshaped to be easier to satisfy (Carroll et al. 2024); option-count maxed, meaning gutted. The asymmetry: an ought-work theory has to survive being optimized against. Is-work never faces that. align(A, humans) becomes optimize(A, target-extracted-from-humans), and the extraction is only as good as the agency model behind it.

Existing challenges

Existing challenges

• coercive control
• addiction
• attention capture

These all push the boundaries of already-existing intuitive notions of agency.

These are naturally-occurring agency-“erosion” attacks. Any “agency-as-value” that scores the cult member, the slot-machine addict, or the doomscroller as highly empowered is probably not doing what we want. Mostly the formalisms can’t classify them right yet.

Historical case: rational addiction (Becker and Murphy 1988) modelled the addict as a forward-looking maximizer optimally choosing to be addicted — and made correct predictions (addicts respond to anticipated future prices). So this is descriptively successful, but normatively unsatisfying.

Under rational addiction, cigarette taxes hurt smokers; under a behavioural rebuild (Gruber and Kőszegi 2001) the same taxes help them. Same data, different agency model but the opposite verdict. So whether an AI reshaping our preferences is help or attack is decided by the agency model, not the data alone. Being explicit let the model be attacked in print (Rogeberg 2004).

Existing challenges

• What even is life?
• In what senses are collectives agents?
• Better metrics to track some good we care about

Future applications

• Gradual disempowerment
• Moral worth of machine intelligences
• …

References

Arrow. 1951. Social Choice and Individual Values.

Arrow, and Debreu. 1954. “Existence of an Equilibrium for a Competitive Economy.” Econometrica.

Bandura. 1977. “Self-Efficacy: Toward a Unifying Theory of Behavioral Change.” Psychological Review.

Barandiaran, Di Paolo, and Rohde. 2009. “Defining Agency: Individuality, Normativity, Asymmetry, and Spatio-Temporality in Action.” Adaptive Behavior.

Beauchamp, and Childress. 2019. Principles of Biomedical Ethics.

Becker, and Murphy. 1988. “A Theory of Rational Addiction.” The Journal of Political Economy.

Boyd. 1991. “Realism, Anti-Foundationalism and the Enthusiasm for Natural Kinds.” Philosophical Studies.

Bruineberg, Dołęga, Dewhurst, et al. 2022. “The Emperor’s New Markov Blankets.” Behavioral and Brain Sciences.

Camara. 2021. “Computationally Tractable Choice.”

Carroll, Foote, Siththaranjan, et al. 2024. “AI Alignment with Changing and Influenceable Reward Functions.”

Coase. 1937. “The Nature of the Firm.” Economica.

Debreu. 1974. “Excess Demand Functions.” Journal of Mathematical Economics.

Dennett, Daniel C. 1991. “Real Patterns.” The Journal of Philosophy.

Dennett, D. C. 1998. The Intentional Stance. A Bradford Book.

Du, Tiomkin, Kiciman, et al. 2020. “AvE: Assistance via Empowerment.” In Advances in Neural Information Processing Systems.

England. 2013. “Statistical Physics of Self-Replication.” The Journal of Chemical Physics.

Flint. 2020. “The Ground of Optimization.”

Frankfurt. 1971. “Freedom of the Will and the Concept of a Person.” The Journal of Philosophy.

Gruber, and Kőszegi. 2001. “Is Addiction ``Rational’’? Theory and Evidence.” The Quarterly Journal of Economics.

Gustafsson. 2022. Money-Pump Arguments. Elements in Decision Theory and Philosophy.

Hadfield-Menell, and Hadfield. 2018. “Incomplete Contracting and AI Alignment.”

Harsanyi. 1955. “Cardinal Welfare, Individualistic Ethics, and Interpersonal Comparisons of Utility.” Journal of Political Economy.

Hart. 1968. Punishment and Responsibility: Essays in the Philosophy of Law.

Hayek. 1945. “The Use of Knowledge in Society.” The American Economic Review.

Hedden. 2015. “Time-Slice Rationality.” Mind.

Hubinger, Merwijk, Mikulik, et al. 2021. “Risks from Learned Optimization in Advanced Machine Learning Systems.”

Hutter. 2005a. “The Universal Algorithmic Agent AIXI.” In.

———. 2005b. Universal Artificial Intelligence: Sequential Decisions Based on Algorithmic Probability. Texts in Theoretical Computer Science.

Hutter, Quarel, and Catt. 2024. An Introduction to Universal Artificial Intelligence.

janus. 2022. “Simulators.”

Jensen, and Meckling. 1976. “Theory of the Firm: Managerial Behavior, Agency Costs and Ownership Structure.” Journal of Financial Economics.

Kasirzadeh, and Gabriel. 2023. “In Conversation with Artificial Intelligence: Aligning Language Models with Human Values.” Philosophy & Technology.

Kenton, Kumar, Farquhar, et al. 2023. “Discovering Agents.” Artificial Intelligence.

Klyubin, Alexander S., Polani, and Nehaniv. 2005. “All Else Being Equal Be Empowered.” In Advances in Artificial Life. Lecture Notes in Computer Science.

Klyubin, A.S., Polani, and Nehaniv. 2005. “Empowerment: A Universal Agent-Centric Measure of Control.” In 2005 IEEE Congress on Evolutionary Computation.

Korsgaard. 2009. Self-Constitution: Agency, Identity, and Integrity.

Krakauer, Bertschinger, Olbrich, et al. 2020. “The Information Theory of Individuality.” Theory in Biosciences.

Kulveit, Douglas, Ammann, et al. 2025. “Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development.”

Legg, and Hutter. 2007. “Universal Intelligence: A Definition of Machine Intelligence.” Minds and Machines.

Levin. 2022. “Technological Approach to Mind Everywhere: An Experimentally-Grounded Framework for Understanding Diverse Bodies and Minds.” Frontiers in Systems Neuroscience.

List. 2019. “Levels: Descriptive, Explanatory, and Ontological.” Noûs.

List, and Pettit. 2011. Group Agency: The Possibility, Design, and Status of Corporate Agents.

Locke. 1689. An Essay Concerning Human Understanding.

Maturana, and Varela. 1980. Autopoiesis and Cognition: The Realization of the Living. Boston Studies in the Philosophy of Science.

Mohamed, and Rezende. 2015. “Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning.” In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2. NIPS’15.

Ngo. 2019. “Coherent Behaviour in the Real World Is an Incoherent Concept.”

Nussbaum. 2011. Creating Capabilities: The Human Development Approach.

Omohundro. 2008. “The Basic AI Drives.” In Proceedings of the 2008 Conference on Artificial General Intelligence 2008: Proceedings of the First AGI Conference.

Orseau, McGill, and Legg. 2018. “Agents and Devices: A Relative Definition of Agency.”

Petersen. 2023. “Invulnerable Incomplete Preferences: A Formal Statement.”

Prigogine, and Stengers. 1984. Order Out of Chaos: Man’s New Dialogue with Nature.

Ray. 2007. A Game-Theoretic Perspective on Coalition Formation. The Lipsey Lectures.

Ray, and Vohra. 2015. “Coalition Formation.” In Handbook of Game Theory with Economic Applications.

Riker. 1962. The Theory of Political Coalitions.

Rogeberg. 2004. “Taking Absurd Theories Seriously: Economics and the Case of Rational Addiction Theories.” Philosophy of Science.

Rosas, Geiger, Luppi, et al. 2024. “Software in the Natural World: A Computational Approach to Hierarchical Emergence.”

Rotter. 1966. “Generalized Expectancies for Internal Versus External Control of Reinforcement.” Psychological Monographs: General and Applied.

Ryle. 1949. The Concept of Mind.

Salge, and Polani. 2017. “Empowerment as Replacement for the Three Laws of Robotics.” Frontiers in Robotics and AI.

Sen, Amartya K. 1977. “Rational Fools: A Critique of the Behavioral Foundations of Economic Theory.” Philosophy and Public Affairs.

Sen, Amartya. 1985. “Well-Being, Agency and Freedom: The Dewey Lectures 1984.” The Journal of Philosophy.

Shah. 2018. “Coherence Arguments Do Not Entail Goal-Directed Behavior.”

Shanahan, McDonell, and Reynolds. 2023. “Role Play with Large Language Models.” Nature.

Still, Sivak, Bell, et al. 2012. “Thermodynamics of Prediction.” Physical Review Letters.

Strawson. 1962. “Freedom and Resentment.” Proceedings of the British Academy.

Thompson. 2007. Mind in Life: Biology, Phenomenology, and the Sciences of Mind.

Thornley. 2025. “The Shutdown Problem: An AI Engineering Puzzle for Decision Theorists.” Philosophical Studies.

Turner, Smith, Shah, et al. 2021. “Optimal Policies Tend To Seek Power.” In Advances in Neural Information Processing Systems.

von Neumann, and Morgenstern. 1944. Theory of Games and Economic Behavior.

Williamson. 1985. The Economic Institutions of Capitalism: Firms, Markets, Relational Contracting.

Yudkowsky. 2008. “Measuring Optimization Power.”

———. 2011. “Complex Value Systems in Friendly AI.” In Artificial General Intelligence (AGI 2011). Lecture Notes in Computer Science.

Zhi-Xuan, Carroll, Franklin, et al. 2025. “Beyond Preferences in AI Alignment.” Philosophical Studies.

Parking lot

You should stop here. Everything elow is Claude muttering to himself while I give him research tasks.

The observer ladder

Laplace’s demon ascribes no agency.

First, the working definition of the axis, because the slide only names it. The distinction is the arity of the agency predicate — how many arguments it takes. Intrinsic (one-place): the theory treats “S is an agent” as a complete claim, a property S has or lacks on its own, the way “S has mass” is; the physicist, the layperson and Laplace’s demon all return the same verdict, because the verdict is a fact about S. Ascribed (two-place): the real claim is “S is an agent relative to describer D”, a relation between the system and a describer, and its truth can flip when D changes. The one-move test to place a model: try to delete the “…relative to D”. If nothing is lost, it is one-place / intrinsic (autopoiesis: “the cell maintains its own boundary” needs no observer). If the sentence goes incomplete — “an agent… to whom?” — it is two-place / ascribed (Dennett: “S is an intentional system” means the intentional stance toward S earns predictive traction, which is relative to the stance-taker). The thermostat makes it concrete: a one-place theory says there is a fact of the matter (no — no self-production); a two-place theory says yes relative to a coarse describer for whom “it wants the room at 20°” predicts well, and no relative to an engineer with the wiring diagram — and both are correct, because the predicate has a second slot. So the test in one line: does “both verdicts can be right” sound acceptable (two-place) or like one side is simply mistaken (one-place)? Two warnings. The two “places” are (system, describer), not two agents — this axis is orthogonal to multi-agent, which is why solipsistic-yet-ascribed agents-vs-devices keeps them distinct. And two-place is not subjective: given the describer the fact is objective (the Real-Patterns point below), so the axis measures how observer-relative, not how made-up. Now the walk up the axis — stops along the continuum, not four separate dimensions. Stop 0, intrinsic — accounts in the physics idiom, agency-facts with no ascriber anywhere; the dissipative/autopoietic cluster (Prigogine and Stengers 1984; England 2013; Maturana and Varela 1980) makes the strongest boundary-derivation claim in the zoo — the system produces and maintains its own boundary — and insists that boundary is an observer-independent physical fact. Stop 1, an unmodelled describer — Dennett’s intentional stance (D. C. Dennett 1998); not eye-of-the-beholder subjectivism: “Real Patterns” (Daniel C. Dennett 1991) — whether the agent-description compresses and predicts better than the device-description is an objective fact; Orseau, McGill and Legg quantify it (Orseau, McGill, and Legg 2018), and Discovering Agents upgrades it from prediction to intervention (Kenton et al. 2023) — stance, made experimentally testable. The tempting third stop — “the observer’s resources enter the theory: agency-as-compression is only non-vacuous for a bounded observer” — is not a further step up this axis; it is the boundedness axis pointed at the theorist. That is exactly why Laplace’s demon reads two ways — as no observer (stop 0) and as an unbounded observer (max boundedness): the device-description is free for it, so the agent-model never wins, and agency-talk exists because describers are bounded. Fold it back onto boundedness and the double-placement dissolves. The top stop, observer as participant (the dashed band on the plot) — game theory (my model of my opponent contains their model of me), mechanism design (the designer is in the game), embedded agency (self-ascription), and the forensic model: agency ascribed from inside a community of mutual accountability; Strawson drew this boundary avant la lettre (Strawson 1962).

How to evaluate a formalism-builder

The taxonomy is a job specification, not a demolition; its criticisms aim at implicit models. Polani is maximally explicit, so by the talk’s own standard he is the hero, not the target — our criticisms of empowerment are possible because it is explicit; implicit models cannot even be wrong. The rubric: (1) are the assumptions inspectable? (2) does it convert a confusion into a parameter — something we can vary and compute? (3) is the programme fruitful — generating new uses, not just defences? (4) does it know which side of the is/ought line it is on? (5) are its failure modes legible from inside? Empowerment on (2): it does not solve the boundary problem, but it turns boundary-dependence into a visible input — carve the sensors and actuators differently, compute again, compare. On (3): variational empowerment as intrinsic motivation in RL (Mohamed and Rezende 2015); Turner’s POWER (Turner et al. 2021) is a close cousin of expected optionality, so the threat canon’s is-work already runs on Polani-adjacent machinery; assistance-via-empowerment uses human empowerment as the ought-work target (Du et al. 2020). On (5), the deficiencies are named within the programme itself: computability (approximations are loose), Goodhart under naive maximization (the control-room of buttons is roughly an empowerment-maximizing configuration), capacity-not-commitment (options measured, goals not — the cult member might score fine). Named, publishable deficiencies inside a programme that can metabolize them: the difference between a research programme and a vibe. The two bullets are the questions to carry into Polani’s talk. Empowerment is one of the very few frameworks explicitly hired for both jobs: descriptive measure of an agent’s grip on the world (A. S. Klyubin, Polani, and Nehaniv 2005; Alexander S. Klyubin, Polani, and Nehaniv 2005), and proposed normative target for what to preserve in humans (Salge and Polani 2017). Same information-theoretic quantity, read in two directions: the AI’s empowerment — its channel capacity onto the world — is what power-seeking arguments fear; human empowerment is a proposed alignment target. Same formalism, opposite sign, depending on whose actuation channel it is. And the boundary confusion resurfaces inside the most formal model on offer: empowerment needs an actuation channel defined, which means someone already drew the agent’s edge. The audience should carry these as instruments, not verdicts.

The type confusion

“AI agents in practice” runs on a category mistake (Ryle 1949).

• The LLM is not an agent as given. The weights are a conditional distribution — a generator of agents, not the sort of entity the align relation takes. The agent is generated: harness, tool loop, memory, and above all the policy for filling the context window are what turn the generator into something that acts. Simulators and role-play (janus 2022; Shanahan, McDonell, and Reynolds 2023) already make this claim at the prompt level; the 2026 product harness is the industrialised version — it pins one simulacrum, equips it with actuators, and ships it as “an agent”. This is not the boundary axis again: that axis asks where to draw the edge; this asks whether what the edge encloses is of agent type at all — a generator is not merely a small agent. (It also explains why LLM + harness is the hardest point to place on the plots: it is not one model of agency but a machine for minting them, so its coordinates smear.)
• ”Is the LLM aligned?” is therefore type-ambiguous. Read of the instantiated system, it is an eval claim about one agent in one harness; read of the generator, it is a dispositional claim — the agents it instantiates tend to come out aligned — with entirely different success conditions. Alignment discourse slides between the readings: model cards make generator claims; incidents involve instantiated agents. Mesa-optimization (Hubinger et al. 2021) is the training-time twin of the same type split (base optimizer vs the optimizer it grows); harness-instantiation is the deployment-time version.
• The larger system acts; it still is not a forensic agent. Deterrence needs a policy that adapts when consequences change (the Discovering-Agents test the courts have run for centuries) and a persistent identity for sanctions to reach. An instantiated agent dies at the end of its context window — sanctions do not propagate to the next instantiation except through the training loop or the deploying firm. So the forensic address resolves upward: the company (legal personhood, the civil cousin from the zoo notes) or the training pipeline, never the agent that acted. Connects to the temporal-boundary bullet below (context window as time-slice person, harness memory as prosthetic diachronic glue).
• If promoted to the main body: the two-place opening already carries the short version in its notes; the full version is a slide of its own after “Implicit theories of agency” — the 2026-topical instance of “everyone imports a tacit model”.

From Beyond Preferences

Zhi-Xuan et al. (2025)

Angles the taxonomy forgot.

• A third job — blueprint. Agency models are not just hired for is-work on A and ought-work about B; for the AI itself the model is a design choice. Coherence is not rationally required (Thornley 2025; Petersen 2023), coherent EU maximisation is intractable (Camara 2021), and deliberately incomplete preferences buy corrigibility — preferential gaps across contexts remove the incentive to manipulate which context the agent is in. The blueprint job wants different virtues again: coherent enough to be useful, incomplete enough to shut down.
• Motivational currency axis (thin ↔︎ thick). Every model silently answers “what is the agent moved by”: thin preferences (bare betterness), scalar reward, thick evaluative concepts (honest, novel), reasons, role-norms. Their Evaluate–Commensurate–Decide model makes preferences constructed outputs, not primitives. Would separate models our current axes lump together (VNM and RL are both thin; forensic and role-constituted are thick).
• The temporal boundary. Our boundary axis is synchronic; preference change (drift, volition, manipulation (Carroll et al. 2024)) and time-slice vs person-unity debates (Hedden 2015; Korsgaard 2009) raise where the agent ends in time. Live for LLM agents: is each context window a fresh time-slice person, with harness memory as prosthetic diachronic glue?
• Role-constituted agency (in the dataframe at fog tier; parked alongside the blueprint role it exemplifies). Align to the normative ideal of a social role — the good assistant — not to preferences (Kasirzadeh and Gabriel 2023). Drift bonus: RLHF already functions as role-norm alignment (annotators judge goodness-of-a-kind) while being described as preference matching.
• Their four preferentist theses factor along our geometry — theses 1–2 are the is/ought sides of agency-talk about A, theses 3–4 the single/multi-principal question about the B side. The paper is a book-length audit of one named point on our plot; citable validation of the two-place framing.

Economics fought these battles first

The senior discipline’s failure catalogue, mapped onto our structure.

• Arrow–Debreu is the reassurance theorem of economics (Arrow and Debreu 1954). The welfare theorems are proved for idealised preference-maximizers and get quoted as if they described actually-existing agents — the same ought→is drift as the power-seeking theorems, opposite emotional valence. The century of battles over when equilibrium breaks (externalities, incomplete markets, information asymmetries) is the empirical record of model #1 meeting substrate.
• Sonnenschein–Mantel–Debreu, companion to Arrow in the B-side file (Debreu 1974). Aggregation destroys structure: aggregate excess demand of perfectly rational individuals can be essentially anything. Arrow says the collective preference may not exist; SMD says even when the aggregate behaves, the theory implies nothing about it. Both are bad news for “align to humanity”.
• Theory of the firm as boundary derivation (Coase 1937; Williamson 1985). Coase, then Williamson: the firm’s boundary is derived from transaction costs, not primitive. Challenger to the empty-quadrant claim (see the boundary × multi-agent plot note). The Coasean singularity (delegated_agent_governance) is the AI-era sequel — agent boundaries reshuffle when AI crashes transaction costs.
• Externalities are the original alignment failures. Every agent locally aligned to its principal, aggregate outcome misaligned with everyone; gradual disempowerment is an externality story before it is anything else. Pigou vs Coase is a mechanism-design fork alignment will re-run: tax the misalignment, or assign rights and let agents bargain.
• Coalition formation makes the partition itself strategic (coalition_games). The action question — manipulate \(v(\pi, S)\) so a preferred partition is stable — is a disempowerment mechanism: divide-and-rule played against the B side. “Humanity” is not just hard to aggregate (Arrow, SMD); it is a coalition an adversary can actively prevent from forming.
• Meta-lesson for the talk. Alignment is speed-running economics’ arc with the same implicit agency model, and the battle reports are published.

Aligned to whom, carved how

Alignment is indexed; the verdict moves with both indices.

• The alignment matrix (candidate slide object). Rows: dismemberments of the system (weights / product / firm / ecosystem). Columns: candidate principals (user, shareholders, society). Worked example everyone has intuitions about: Facebook’s recommender is a success story in the (product, shareholders) cell and the canonical horror story in the (product, user) and (product, society) cells — nothing about the artefact changes, we rotate the evaluation. Same exercise for Claude: the weights in-context vs the product vs the firm may each align best to a different principal. This makes the boundary axis practical: choosing where to cut is choosing which alignment claims come out true.
• Surplus under misalignment is the norm (Jensen and Meckling 1976). Delegation persists whenever surplus exceeds monitoring costs plus bonding costs plus residual loss — nobody expects a lawyer to be aligned, only worth it net of agency costs. Reframes the AI question from purity to surplus-sharing: who captures the residual in the user–lab–model triangle? Incomplete contracting is the mature formal home (Hadfield-Menell and Hadfield 2018) — the “contract” with an AI cannot specify every contingency, which is Goodhart in contract-theory clothing. Connects to alignment_problems and delegated_agent_governance.
• “Agent” is a homonym; the deck currently covers one sense. Agency-1: the capacity to act (the zoo’s sense). Agency-2: acting on behalf of — agency law, agency costs, fiduciary duty. Alignment discourse slides between them: “more agentic = more dangerous” is an agency-1 claim; “a good assistant is loyal” is an agency-2 claim; “AI agents” as a product category equivocates on both. Monument to the pun: the HTTP User-Agent header — the browser was named as the user’s delegate, and the ad-blocker and DRM wars are a fight over whose agent the user agent is. The role-constituted row is the agency-2 member of the zoo (a fiduciary is a role whose norms are agency-2 norms); worth saying aloud when it comes up.

Auditioning for relevance

Keep these prominent until they win or lose a slide. (Omohundro’s basic drives and the money-pump arguments won — absorbed into the is⇄ought-crossings notes.)

• Social choice theory (Arrow 1951) — if collective agents are subject to Arrow impossibility, to what can we align? The B side again: “align to humanity” presupposes an aggregate preference that Arrow says may not exist; connects the group/institutional row to day 4. Companions from Beyond Preferences: Harsanyi’s aggregation theorem (Harsanyi 1955) is the positive case for aggregation and rests on exactly the EUT-plus-comparability assumptions the rest of the deck undermines; aggregate-preference optimisation refounds the socialist calculation debate (Hayek 1945) and is politically infeasible besides; their contractualist answer is negotiated, role-scoped norms rather than aggregated preferences.