Informational empowerment
A particular mathematization of intrinsic motivation
2022-11-27 — 2026-06-22
Wherein Informational Empowerment Is Defined as a Mutual‐Information Quantity From Optimal Control Theory, and Its Prospects as an Intrinsic Motivation for Reinforcement Learning Are Examined.
The drive to empowerment is a hypothesized, generic, internal pressure on an agent to move itself into states from which it has lots of influence (or many options) going forward. If an agent “seeks empowerment”, it ‘aims’ to maximize its ability to affect the future. We could weaponise this concept in the 21st century AI agency age, where it helps us ask questions like: is power-seeking a generic property of intelligent agents?
Maybe? The technical definition here looks a little like that, but it also looks a little like a formalization for robots and abstract algorithms of the notion of having fun, or some other intrinsic motivation
Informational empowerment is tightly defined in a formal sense: it’s a precise, information-theoretic quantity from optimal control theory, specifically the mutual information between actions and future states. Borrowing the POMDP notation — sans-serif for random variables, \(\mathsf{a}_t \in \mathcal{A}\) for actions and \(\mathsf{s}_t \in \mathcal{S}\) for states — the \(n\)-step empowerment at state \(s\) is the channel capacity from an action sequence \(\mathsf{a}_t^n = (\mathsf{a}_t, \mathsf{a}_{t+1}, \ldots, \mathsf{a}_{t+n-1})\) to the future state \(\mathsf{s}_{t+n}\):
\[ \mathfrak{E}_n(s) = \max_{p(\mathsf{a}_t^n)} I\!\left(\mathsf{a}_t^n;\, \mathsf{s}_{t+n} \,\middle|\, \mathsf{s}_t = s\right), \tag{1}\]
where the maximum runs over distributions on action sequences and the mutual information is taken under the environment dynamics \(T(s' \mid s, a)\). In rough terms it quantifies how many distinguishable futures the agent could reliably reach, given its action choices and the dynamics of the environment.
Empowerment is sometimes called a “pseudo-utility” (i.e. a kind of internal reward proxy). If we squint at it, we can imagine that it quantifies how much the agent can “control” its future, in a way that’s independent of any particular extrinsic reward or task. Of course, in making that claim we assume a lot of structure: a well-defined agent, environment, state space, action space, transition dynamics, etc, wrapped up in a POMDP or similar formalism.
It looks like empowerment in this special formalization captures a general tendency of agents to keep options open and extend their influence. The argument (e.g. in Empowerment Is (Almost) All We Need) is that a sufficiently powerful drive for empowerment might lead to many of the behaviours we desire in intelligent agents — exploration, maintaining options, robustness, die Wille zur Macht etc.
In each case, the concept is less about a fixed external goal or reward (like “get the treasure”) — rather, the agent is driven by how much control it has over its future, regardless of the extrinsic objective.
1 In reinforcement learning
🚧TODO🚧
In RL, agents typically optimize for maximal yield from an external reward signal. Many environments have sparse, noisy, or delayed reward signals, which makes learning hard (the exploration problem, credit assignment, etc.).
Empowerment sidesteps some of these pathologies:
- Because it doesn’t depend on an external task, an agent can explore more “safely” or systematically by trying to increase its control.
- It biases the agent towards states with many outgoing branches — places from which many futures are reachable. In many domains, that corresponds to being in central, flexible positions rather than being stuck in a dead end. As such it looks like a “universal” reward to guide exploration
- Maybe it supplements sparse rewards? (Dai et al. 2021)
This all sounds cool, but clearly most people do not use empowerment as their RL objective. AFAICS this is because it creates more problems than it solves.
It is easy to say that empowerment is “easier” than long-horizon reward computation. But computing empowerment is generally expensive, and, I would argue, impossible for the cases where we want it most.. Estimating information is generally hard, for one thing, and it gets worse in high-dimensional, continuous, or partially observable settings. Further, getting information about future states requires modelling the environment dynamics. Exploring over open domains, like niche formation, self-modification etc is not generally tractable (naïvely we need to simulate the universe itself to get the empowerment value function). So it’s the usual RL problems, but with a spiky, tricky estimand and an ontological nightmare. There are nonetheless various approximation methods (Zhao et al. 2020).
Moreover, I know of no optimality results for empowerment, and it looks like it is not terribly efficient.
I have minor questions about how it is used in practice. Is it purely a training objective? What is the off-policy/on-policy story? How far do we roll out? Do we estimate an empowerment value function etc?
2 Does evolution seek empowerment?
It is tempting to read the empowerment story back into biology: a replicator that keeps many viable futures open — via robustness and evolvability, niche construction, redundant pathways, bet-hedging across fluctuating environments — looks a lot like an agent keeping its options open. The general form of that argument, where selection forces systems to behave as if they optimize something, is the business of selection theorems.
It looks like a crap candidate for evolution though. Fitness works as an as-if utility because selection optimizes it by construction — nobody has to compute it [see utility and fitness]. Empowerment is a different quantity: option-preservation, defined over an agent’s own anticipated rollouts, only instrumentally correlated with fitness. Selection optimizes fitness, not empowerment; empowerment-like traits survive only when they happen to pay rent in variable environments. So I don’t think genes “seek empowerment” in the way the metaphor wants — there is no rollout, no shadow of the future, only the retrospective bookkeeping of what reproduced. Evolution is, in general a one-step ahead process. This is not to say it has no foresight — , cooperation-evolution and evolvability models complicate things. But! Gradients through roll-outs? No.
3 As power-seeking?
Empowerment also suggests a route toward open-endedness: systems whose internal drive is to expand their “influence frontier” could drive themselves toward complexity and diversity, or the ability to affect the world in more and more ways. Whether that internal drive cashes out as the AI-safety worry about power-seeking is… unclear to me. TBD.
4 For sparse reward
Leibfried, Pascual-Diaz, and Grau-Moya (2020)? Dai et al. (2021)?
5 Practical estimation
Formally, lLook into techniques for approximating empowerment in high-dimensional or continuous spaces (variational approximations, estimating mutual information, etc.) (Zhao et al. 2020).
6 Causal-flavoured
What actions truly affect what variables? “Empowerment via Causal Learning” (Cao, Feng, Fang, et al. 2025) is a framework in model‐based RL that uses causal structure to compute empowerment more ‘meaningfully’. There is also a notion called causal action empowerment*, which aims to focus the empowerment signal on those actions that causally influence important parts of the environment (Cao, Feng, Huo, et al. 2025).
7 Incoming
- Salge, Glackin, and Polani (2014)