Empowerment and intrinsic motivation
Do agents learn to want freedom?
2022-11-27 — 2025-09-25
Wherein the notion of empowerment is examined as an intrinsic drive, and an analogy is drawn to replicators’ niche construction, whereby organisms are shown to reshape environments to expand controllable futures.
At a high level, empowerment is an intrinsic motivation or internal pressure on an agent to move itself in states from which it has lots of influence (or many options) going forward. In other words: it wants to maximize its ability to affect the future.
There are really two related notions of empowerment in play:
- Technical empowerment: a precise, information-theoretic quantity from optimal control theory, usually defined as the mutual information between actions and future states.
- Metaphorical empowerment: a looser, agent-foundations concept that points to a broader tendency for agents to increase their options and influence—i.e. a bridge concept towards “power-seeking” in general.
In each case, the concept isn’t about a fixed external goal or reward (like “get the treasure”) — instead, the agent is driven by how much control it has over its future, regardless of the extrinsic objective.
In the technical sense, empowerment is measured in terms of mutual information between the agent’s actions (or action sequences) and its future states. In rough terms: how many distinguishable futures can the agent reliably reach, given its action choices and the dynamics of the environment? Because of that, empowerment is sometimes called a “pseudo-utility” (i.e. a kind of internal reward proxy) that depends only on local, agent‐accessible information.
In the broader metaphorical sense, empowerment gestures towards a general tendency of agents to keep options open and extend their influence. The argument (e.g. in Empowerment Is (Almost) All We Need) is that a sufficiently powerful drive for empowerment might lead to many of the behaviours we desire in intelligent agents — exploration, maintaining options, robustness, etc.
1 Why “empowerment” is interesting in reinforcement learning
To see its appeal, let’s contrast it with ordinary reinforcement learning (RL). In RL, agents typically optimize an external reward signal. But many environments have sparse, noisy, or delayed reward signals, which makes learning hard (the exploration problem, credit assignment, etc.).
Empowerment offers a complementary mechanism:
- Because it doesn’t depend on an external task, an agent can explore more “safely” or systematically by trying to increase its control.
- It biases the agent towards states with many outgoing branches — places from which many futures are reachable. In many domains, that corresponds to being in central, flexible positions rather than being stuck in a dead end.
- Some works combine extrinsic reward with empowerment. For example, there is a formulation of a unified Bellman equation that mixes reward maximization with empowerment terms. (Leibfried, Pascual-Diaz, and Grau-Moya 2020)
- Others use empowerment or information‐theoretic objectives as intrinsic motivations to guide exploration, especially in sparse‐reward tasks. (Dai et al. 2021)
- More recently, works have integrated causal modelling with empowerment to get better sample efficiency and more directed exploration. For instance, “Empowerment via Causal Learning” is a framework in model‐based RL that uses causal structure to compute empowerment more meaningfully. (Cao, Feng, Fang, et al. 2025)
- There is also a notion called causal action empowerment, which aims to focus the empowerment signal on those actions that causally influence important parts of the environment. (Cao, Feng, Huo, et al. 2025)
Challenges / caveats:
For the technical definition:
- Computing empowerment is often expensive, especially in high-dimensional, continuous, or partially observable settings.
For the metaphorical / agent foundations sense:
- It may push agents to prefer “control for its own sake,” sometimes at odds with actual tasks.
- The connection between empowerment and “good behaviour” isn’t guaranteed in arbitrary domains—it needs careful shaping, constraints, or combination with extrinsic goals.
2 Empowerment in replicators
Now let’s pivot to the biological / evolutionary side: genetic replicators, i.e. replicators in the sense of genes (or more abstract replicators) that propagate and evolve. How might we adapt the intuition of empowerment there — and why it’s suggestive to connect the two domains?
2.1 Replicators, vehicles, and influence
In evolutionary biology, a replicator (in the Dawkins/Hull sense) is an entity that is replicated with variation and subject to selection pressures (Godfrey-Smith 2000).A gene “wants” to maximize its propagation potential (informally speaking). It evolves strategies (via the organism) to influence its environment (via phenotype, behaviour, niche construction, etc.). Of course, genes don’t literally compute empowerment in the technical sense; here I’m using “empowerment” loosely. In that metaphor:
- A replicator benefits from having many possible viable futures — i.e. flexibility in ecological or developmental trajectories such that it can survive under various conditions.
- The vehicle/organism is the means by which the replicator acts on the environment to preserve or replicate itself.
Just as an AI agent might try to keep many future branches open, a replicator (through its phenotypic machinery) might favour designs that maintain options in changing environments.
2.2 Empowerment-like structure in evolution
Here are some speculative bridges to empowerment:
- Robustness & evolvability: replication systems that can tolerate perturbations (robustness) and adapt (evolvability) are more “powerful” in the face of environmental change. That’s a kind of biological counterpart to having many controllable futures.
- Niche construction / environment modification: many organisms modify the environment (e.g. beaver dams, root systems altering soil, microbial communities altering chemistry). These are ways replicators/vehicles shape the environment to increase the control space or favourable pathways they can exploit. That’s like increasing empowerment by altering environmental dynamics.
- Redundancy, backup paths, modularity: mechanisms like gene duplication, redundant pathways, or modular designs allow alternative routes of adaptation or survival when parts fail. That’s akin to an agent having multiple “paths” to future states.
- Selection in fluctuating environments: when environments change, replicators that maintain flexible strategies (i.e. not overly specialized) may outperform those with narrow optima. That aligns with the push toward “keeping many options open.”
Pros of the analogy:
- It helps us see that empowerment isn’t just a quirky AI trick—it’s related to very deep, general pressures about control, options, and resilience.
- It encourages thinking of AI agents not as isolated optimizers of fixed tasks, but as evolving, interacting entities with long‐term survival or replicative pressures.
- It suggests a route toward open-endedness: systems whose internal drive is to expand their “influence frontier” could self‐drive to complexity and diversity (an ambition in AGI / artificial life).
Limits / misfits:
- Biological replicators don’t literally compute mutual information or maximize an explicit “empowerment” objective; the analogy is metaphorical, not literal.
- The optimization pressure on replicators is mediated via competition, mutation, constraints, and many nonlinear dynamics, which may lead to side effects, trade‐offs, or path dependence — unlike a clean, rational agent maximizing an information theoretic objective.
- There’s no “learning algorithm” in the same sense; evolutionary dynamics are slower, blind, and subject to drift, unlike an RL agent with explicit gradient updates.
- In evolution, sometimes “losing options” (specialization) is beneficial in stable niches — which empowerment might resist.
For more on that theme, continue on to utility and fitness.
3 Incoming
Fun things I’d like to read.
Combine with extrinsic tasks
- See how empowerment helps in sparse reward environments or as an exploration bonus.
- Read “A Unified Bellman Principle Combining Reward Maximization and Empowerment” for one approach. (Leibfried, Pascual-Diaz, and Grau-Moya 2020)
Scaling & approximation
- Formally, look into techniques for approximating empowerment in high-dimensional or continuous spaces (variational approximations, estimating mutual information, etc.). (Zhao et al. 2020)
Causal / structure-based enhancements
- In the technical sense, explore recent work that adds causal modeling to compute empowerment more meaningfully (i.e. what actions truly affect what variables). (Cao, Feng, Fang, et al. 2025)
Connections to open-endedness, evolution, artificial life
- Metaphorically, study how intrinsic drives (like empowerment) can support open-ended growth or autonomous innovation.
- Read in artificial life / evolutionary robotics about self-replication, niche construction, and the pressures toward controllability and adaptability. (Taylor and Dorin 2020)
Critiques and safety considerations
- In the agent foundations sense, investigate failure modes: e.g. empowerment-driven agents might prefer “safe control” regions over risky but useful ones.
- Analyze whether empowerment aligns with human values or task goals, and whether it can be perverted.