Informational empowerment
A particular mathematization of intrinsic motivation
2022-11-27 — 2026-06-22
Wherein Empowerment Is Formalised as the Mutual Information Between an Agent’s Actions and Its Future States, and Pressed into Service as an Intrinsic Motivation.
The drive to empowerment is a hypothesized, generic, internal pressure on an agent to move itself into states from which it has lots of influence (or many options) going forward. If an agent “seeks empowerment”, it ‘aims’ to maximize its ability to affect the future. We could weaponise this concept in the 21st century AI agency age, where it helps us ask questions like: is power-seeking a generic property of intelligent agents?
Maybe? The technical definition here looks a little like that, but it also looks a little like a formalization for robots and abstract algorithms of the notion of having fun, or some other intrinsic motivation
Informational empowerment is tightly defined in a formal sense: it’s a precise, information-theoretic quantity from optimal control theory, specifically the mutual information between actions and future states. Borrowing the POMDP notation — sans-serif for random variables, \(\mathsf{a}_t \in \mathcal{A}\) for actions and \(\mathsf{s}_t \in \mathcal{S}\) for states — the \(n\)-step empowerment at state \(s\) is the channel capacity from an action sequence \(\mathsf{a}_t^n = (\mathsf{a}_t, \mathsf{a}_{t+1}, \ldots, \mathsf{a}_{t+n-1})\) to the future state \(\mathsf{s}_{t+n}\):
\[ \mathfrak{E}_n(s) = \max_{p(\mathsf{a}_t^n)} I\!\left(\mathsf{a}_t^n;\, \mathsf{s}_{t+n} \,\middle|\, \mathsf{s}_t = s\right), \tag{1}\]
where the maximum runs over distributions on action sequences and the mutual information is taken under the environment dynamics \(T(s' \mid s, a)\). In rough terms it quantifies how many distinguishable futures the agent could reliably reach, given its action choices and the dynamics of the environment.
Empowerment is sometimes called a “pseudo-utility” (i.e. a kind of internal reward proxy) that depends only on local, agent‐accessible information. If we squint at it, we can imagine that it quantifies how much the agent can “control” its future, in a way that’s independent of any particular extrinsic reward or task. Of course, in making that claim we assumes a lot of structure: a well-defined agent, environment, state space, action space, transition dynamics, etc, wrapped up in a POMDP or similar formalism.
It looks like empowerment in this special formalization captures a general tendency of agents to keep options open and extend their influence. The argument (e.g. in Empowerment Is (Almost) All We Need) is that a sufficiently powerful drive for empowerment might lead to many of the behaviours we desire in intelligent agents — exploration, maintaining options, robustness, die Wille zur Macht etc.
In each case, the concept is less about a fixed external goal or reward (like “get the treasure”) — rather, the agent is driven by how much control it has over its future, regardless of the extrinsic objective.
1 In reinforcement learning
TODO
In RL, agents typically optimize for maximal yield from an external reward signal. Many environments have sparse, noisy, or delayed reward signals, which makes learning hard (the exploration problem, credit assignment, etc.).
Empowerment sidesteps some of these pathologies by providing:
- Because it doesn’t depend on an external task, an agent can explore more “safely” or systematically by trying to increase its control.
- It biases the agent towards states with many outgoing branches — places from which many futures are reachable. In many domains, that corresponds to being in central, flexible positions rather than being stuck in a dead end.
- Some works combine extrinsic reward with empowerment. For example, there is a formulation of a unified Bellman equation that mixes reward maximization with empowerment terms (Leibfried, Pascual-Diaz, and Grau-Moya 2020).
- Others use empowerment or information‐theoretic objectives as intrinsic motivations to guide exploration, especially in sparse‐reward tasks. (Dai et al. 2021)
- More recently, works have integrated causal modelling with empowerment to get better sample efficiency and more directed exploration. For instance, “Empowerment via Causal Learning” is a framework in model‐based RL that uses causal structure to compute empowerment more meaningfully. (Cao, Feng, Fang, et al. 2025)
- There is also a notion called causal action empowerment, which aims to focus the empowerment signal on those actions that causally influence important parts of the environment (Cao, Feng, Huo, et al. 2025). I really need to read that one.
This all sounds cool, but clearly most people do no use empowerment as their RL objective. AFAICS this is because it creates more problems than it solves.
It is easy to say that empowerment is “easier” than long horizon reward computation. But computing empowerment is in general expensive. Estimating information is generally hard, for one thing, and it gets worse in high-dimensional, continuous, or partially observable settings. Further, getting information about future states requires modelling the environment dynamics. So it’s the usual RL problems, but with a spiky, tricky estimand. There are various approximation methods (Zhao et al. 2020).
Moreover, I know of no optimality results for empowerment, and it looks like it is not terribly efficient.
I also have questions about how it is used in practice. Is it purely a training objective? What is the off-policy/on-policy story? How far to we roll out? Do we estimate an empowerment value function etc?
2 Does evolution seek empowerment?
It is tempting to read the empowerment story back into biology: a replicator that keeps many viable futures open — via robustness and evolvability, niche construction, redundant pathways, bet-hedging across fluctuating environments — looks a lot like an agent keeping its options open. The general form of that argument, where selection forces systems to behave as if they optimize something, is the business of selection theorems, and I won’t redo it here.
It looks like a crap candidate for evolution though. Fitness works as an as-if utility because selection optimizes it by construction — nobody has to compute it [see utility and fitness]. Empowerment is a different quantity: option-preservation, defined over an agent’s own anticipated rollouts, only instrumentally correlated with fitness. Selection optimizes fitness, not empowerment; empowerment-like traits survive only when they happen to pay rent in variable environments. So I don’t think genes “seek empowerment” in the way the metaphor wants — there is no rollout, no shadow of the future, only the retrospective bookkeeping of what reproduced. The drive only makes sense when something is actually doing the anticipating.
Empowerment also suggests a route toward open-endedness: systems whose internal drive is to expand their “influence frontier” could drive themselves toward complexity and diversity, or the ability to affect the world in more and more ways. Whether that internal drive cashes out as the AI-safety worry about power-seeking is a separate question, which I take up on its own page.
3 Incoming
Salge, Glackin, and Polani (2014)
Combine with extrinsic tasks
- See how empowerment helps in sparse reward environments or as an exploration bonus.
- Read “A Unified Bellman Principle Combining Reward Maximization and Empowerment” for one approach. (Leibfried, Pascual-Diaz, and Grau-Moya 2020)
Scaling & approximation
- Formally, look into techniques for approximating empowerment in high-dimensional or continuous spaces (variational approximations, estimating mutual information, etc.) (Zhao et al. 2020).
Causal / model-based enhancements:
- In the technical sense, explore recent work that adds causal modelling to compute empowerment more meaningfully (i.e. what actions truly affect what variables) (Cao, Feng, Fang, et al. 2025).
Connections to open-endedness, evolution, artificial life
- Metaphorically, study how intrinsic drives (like empowerment) can support open-ended growth or autonomous innovation.
- Read in artificial life / evolutionary robotics about self-replication, niche construction, and the pressures toward controllability and adaptability (Taylor and Dorin 2020).