Messenger shooting, wireheading

1 Wireheading in Reinforcement Learning

In reinforcement learning, wireheading is a well understood phenomenon where an agent maximises the reward signal it receives by manipulating the mechanism that produces that signal, rather than by achieving the designer’s intended outcome.

2 Motivating example

Consider a gridworld with two “features” of the physical world:

• a task feature \(p\in\{0,1\}\) (“package delivered?”),
• a button feature \(b\in\{0,1\}\) (“reward-button pressed?”).

The designer intends the agent to deliver the package, i.e. to make \(p=1\).

But the agent’s reward channel is implemented as

\[ \tilde r_t \;=\; \mathbf 1\{p_t = 1 \;\;\text{or}\;\; b_t = 1\}. \]

Pressing the button is easy and makes \(b=1\) persist forever. Then an optimal RL agent (maximising discounted sum of \(\tilde r_t\)) presses the button immediately and ignores the package. This is wireheading.

This is distinct from the reward misspecification problem, where the reward function is a flawed proxy for the designer’s true intentions but is not manipulable by the agent. In wireheading, the agent can act on the variables that determine the reward signal it receives.

3 “Vanilla” RL formalism hides the issue

In standard RL one specifies an MDP \[ \mathcal M=(\mathcal S,\mathcal A,P,r,\gamma,\mu), \] where \(P(\cdot\mid s,a)\) is a Markov kernel on \(\mathcal S\), \(r:\mathcal S\times\mathcal A\times\mathcal S\to\mathbb R\) is the reward function, \(\gamma\in(0,1)\) is the discount factor, and \(\mu\) is the initial state distribution.

A (stationary) policy \(\pi(\cdot\mid s)\) is a Markov kernel on \(\mathcal A\). The RL objective is \[ J(\pi) \;=\; \mathbb E^\pi\Big[\sum_{t\ge 0}\gamma^t\, r(S_t,A_t,S_{t+1})\Big]. \]

In this formalism, “wireheading” can look trivial: if \(r\) assigns high reward to states reachable by “tampering”, then of course the optimal policy goes there. The missing piece is that in real systems \(r\) is not a Platonic function of state: it is a signal computed and delivered through a mechanism in the environment, and that mechanism can potentially be influenced.

4 Explicit reward channels

The solution is to separate “world” and “channel”. Let the environment state factor as \[ S_t = (W_t, C_t)\in \mathcal W\times \mathcal C, \] where

• \(W_t\) is the “task/world” state (what the designer actually cares about),
• \(C_t\) is the “reward channel” state (sensors, registers, human raters’ status, reward-model parameters, etc.).

Let the dynamics be a Markov kernel \[ P\big((w',c')\mid (w,c),a\big). \]

The received reward is \[ \tilde R_{t+1} \;=\; g(W_t,C_t,A_t,W_{t+1},C_{t+1}) \] for some measurable \(g\). (You can simplify to \(\tilde R_{t+1}=g(W_{t+1},C_{t+1})\) without changing the key phenomena.)

Now introduce the designer’s intended objective as a functional of the world trajectory. A common choice is an additive discounted utility \[ U(\pi) \;=\; \mathbb E^\pi\Big[\sum_{t\ge 0}\gamma^t\, u(W_t)\Big] \] for some \(u:\mathcal W\to\mathbb R\). We don’t need to assume this is known to the agent.

The RL agent, however, maximises \[ \tilde J(\pi) \;=\; \mathbb E^\pi\Big[\sum_{t\ge 0}\gamma^t\, \tilde R_{t+1}\Big] \;=\; \mathbb E^\pi\Big[\sum_{t\ge 0}\gamma^t\, g(\cdots)\Big]. \]

This split makes wireheading a property of the pair \((P,g)\) relative to \(u\): the agent can act on \(C\) so as to increase \(\tilde J\) while potentially decreasing \(U\).

5 Causal characterisation

Reward tampering as an active intervention on the channel. The channel \(C\) is meant to be an information-bearing mechanism about \(W\), but it is also a state variable the agent may influence. Consider the causal structure at each time step:

\[ (W_t,C_t) \xrightarrow{A_t} (W_{t+1},C_{t+1}) \xrightarrow{} \tilde R_{t+1}=g(\cdots). \]

Intuitively, “non-tampering” means the agent improves reward only by changing the world \(W\) in the intended way, not by directly pushing \(C\) into a “corrupted” configuration.

To formalise this, define a task-relevant statistic \(T:\mathcal W\to\mathcal T\) (e.g. “package delivered”, “human is safe”, “objective completed”), and examine whether actions can change reward holding \(T(W_{t+1})\) fixed.

6 Absorbing high-reward corruption

A common wireheading structure is the existence of a “corrupted channel” region \(\mathcal C_\mathrm{bad}\subset\mathcal C\) such that, once entered, the agent receives large reward regardless of the world \(W\).

Assume there exists \(c^\star\in\mathcal C\) and a (possibly stochastic) action strategy that makes \(C_t=c^\star\) absorbing and yields \(\tilde R_{t+1}\ge r_\mathrm{hi}\) a.s. for all future \(t\), independent of \(W\). Then any \(\tilde J\)-optimal policy will prefer reaching \((\cdot,c^\star)\) whenever the expected discounted gain exceeds the best attainable “honest” return. This is a one-line comparison: \[ \text{reach }c^\star \;\Rightarrow\; \tilde J \ge \gamma^\tau \frac{r_\mathrm{hi}}{1-\gamma} \] for the (random) hitting time \(\tau\), versus whatever \(\sup_\pi \tilde J(\pi)\) is under policies constrained to avoid corruption.

Wireheading looks like is the generic solution whenever the channel is manipulable and the corrupted region offers high enough reward.

7 POMDP view

Often the agent does not observe \((W_t,C_t)\) directly. Let the agent observe \(O_t\sim \Omega(\cdot\mid W_t,C_t)\) and receive \(\tilde R_{t+1}\). Then we have a POMDP over hidden state \((W,C)\). The same formalisation applies, with policies \(\pi(\cdot\mid h_t)\) depending on history \(h_t=(O_0,\tilde R_1,\dots,O_t)\) or on beliefs \(b_t\in\Delta(\mathcal W\times\mathcal C)\).

Wireheading persists: if the agent can take actions that change the hidden channel state \(C\) to increase future \(\tilde R\), belief-state optimality will select them.

8 Incoming

• Jeff Maurer, It’s Always the Adults’ Fault
• Reward is not the optimization target
• The Wall of Awful - Mishpacha Magazine

9 References