Incentive alignment problems

1.1 The Parable of the Coffee-Fetching Bot

Imagine you are the Principal. You’ve just built a sophisticated autonomous robot, your Agent, whose job is to fetch you the perfect cup of coffee every morning.

Your objective is simple: get a high-quality latte, quickly, and without spending too much. The robot’s objective is whatever you program its “reward function” to be. For an audience used to loss functions, a utility function is just its negative—something to be maximized.

Formally:

• Actions ($a$): The robot can choose an action $a$ from a set of options $A$. For example, $A=\{\text{Go to Starbucks},\text{Go to local cafe},\text{Brew instant}\}$.
• Your Payoff ($\mathrm{\Pi }$): Your satisfaction is captured by a payoff function, $\mathrm{\Pi }(a)$, which you want to maximize. A great latte is worth 10 “utils” to you, an okay one 5, and instant coffee 1.
• The Agent’s Utility ($U$): The robot has its own utility function, $U(a)$, based on the reward signal you give it, which it seeks to maximize.

The economists refer to the formal roles that map an observable outcome to the agent rewards as a contract refers to the formal set of rules that defines the agent’s incentive; AFAICT it is almost perfectly analogous to the reward function. Ideally, you could just write a contract to set the agent’s utility equal to your payoff: $U(a)=\mathrm{\Pi }(a)$. If so, the action that maximizes the robot’s utility would, by definition, also maximize yours. This is perfect alignment:

$$\arg \underset{a\in A}{max}U(a)=\arg \underset{a\in A}{max}\mathrm{\Pi }(a)$$

The robot choosing its best option is it choosing our best option. Problem solved! Of course, the world is rarely that simple. The core of the alignment problem arises because you and your agent don’t see or know the same things.

1. Hidden Actions (Moral Hazard): You can’t monitor the robot perfectly. You tell it to “minimize wait time,” but you only see when it gets back. Did it take an efficient route, or did it slack off to conserve its battery (a goal you didn’t explicitly forbid)? This is the classic moral hazard problem: the principal cannot observe the agent’s effort, only a noisy outcome (Holmström 1979).

2. Hidden Information (Adverse Selection): The robot knows things you don’t. It might discover that the fancy cafe’s espresso machine is broken. If its reward is just “bring a latte,” it might default to a suboptimal choice from your perspective, when you would have preferred it to just make instant coffee given that new information.

This is where naive instructions fail. We can’t write a contract for every contingency. Instead of specifying the action, we must design the incentives.

1.2 Incentive Compatibility aligning goals, not actions

Another way to get at the core idea is to talk about incentive compatibility (IC). An incentive structure is incentive compatible if an agent, by acting in their own best interest, naturally chooses the action the principal desires. We design the “rules of the game” (the reward function) so that the agent’s selfishness leads to our desired outcome (Laffont and Martimort 2002).

Instead of a simple reward for “bringing coffee,” we might design a reward function based on observable outcomes: temperature, time of delivery, and cost, fair-trade certification etc. The goal is to craft this function so that the robot, in maximizing its expected reward, behaves as we would. A mechanism with this property is prbably what AI people mean when they describe something as “aligned”

1.3 Multiple agents

The principal-agent problem is about a one-to-one relationship. But what if we are a social planner designing a system for many agents, like an auction? Here, the goal is often efficiency: ensuring the goods go to the people who value them most.

A celebrated example is the Vickrey-Clarke-Groves (VCG) auction. This mechanism is designed to be “dominant-strategy incentive compatible” This means that every bidder’s best strategy is to bid their true value, no matter what anyone else bids (Vickrey 1961; Myerson 1981). The mechanism aligns private incentives (winning) with the social goal (efficiency) (Mas-Colell, Whinston, and Green 1995). Stuff gets much weirder for more complicated things than auctions though. See, e.g. multi agent systems and the full machinery of mechanism design.

1.4 “Nearly” aligned

Classical definitions of alignment are often binary. But in AI, we often deal with approximations. The economic formalism gives us a vocabulary for measuring how misaligned something is.

1.5 What economists don’t deal with

The economic framework is powerful, but it’s built on assumptions that can be fragile in the real world.

1.6 For AI Alignment

The economic perspective doesn’t solve AI alignment, but it provides a vocabulary for framing the problem. It forces us to be precise:

• It moves us from vague wishes to clearly defined objectives, actions, and information structures.
• It shows that alignment is not a property of an agent in isolation, but a feature of the system or game we design for it.
• It gives us a candidate quantifications of for “near misses” and catastrophic failures through concepts like regret, robustness, and distributional risk.

Read on at Aligning AI.