Performative prediction
A thing people worry about when they worry about “socially constructed” categories
2026-06-09 — 2026-06-14
Wherein a Credit Classifier Is Found to Cause the Defaults It Predicts, Performative Stability Is Distinguished From Optimality, and Strategic Manipulation of Features by Applicants Is Examined.
An ML formalization of interesting phenomena such as legibilizing and hyperstition, probably implicit in recommender dynamics, adversarial classification and some types of external validity, among other places. HT TJ for the pointer.
🚧TODO🚧: very excited about Heitzig (2026).
In classical supervised learning we fit parameters \(\theta\) to minimize expected loss over a distribution \(\mathcal{D}\) that sits passively and lets itself be measured, \[\theta_{\mathrm{SL}} = \arg\min_\theta \mathbb{E}_{Z\sim\mathcal{D}}\,\ell(Z;\theta),\] with \(Z=(X,Y)\) a feature–outcome pair and \(\ell\) the loss.
Specifically, while we assume the model watches the world, we assume that the world does not watch back. Performative prediction (Perdomo et al. 2020; Hardt and Mendler-Dünner 2025) is a formalization of a different regime wherein deploying the model changes the subsequent data distribution.
The stock example (familiar from fairness) is assessing people for credit-worthiness. If we predict someone to be at high risk of credit default, we might protectively assign them a punishing interest rate, which in turn increases their propensity to default, thereby confirming our prediction.
We address this phenomenon formally by assuming that each choice of parameters produces a perturbed data distribution \(\mathcal{D}(\theta)\) — the data we would see if we deployed \(\theta\) to do things in the world. In the case that the perturbation ‘looks like a self-fulfilling prophecy’, we call that a hyperstition.
We define the performative risk, \[\mathrm{PR}(\theta) = \mathbb{E}_{Z\sim\mathcal{D}(\theta)}\,\ell(Z;\theta).\] Here \(\theta\) appears twice: once as the model being graded, once inside the distribution it produces. That second appearance clearly clashes with ordinary regression — we cannot simply descend the gradient of the loss in isolation, because moving \(\theta\) also moves the target.
There are two extensions to the definition which recover something like that classical supervised, static setting.
A performative optimum \(\theta_{\mathrm{PO}} = \arg\min_\theta \mathrm{PR}(\theta)\) minimises the performative risk with both copies of \(\theta\) moving together, via the implicit dependence of the distribution on the parameters. We derive this by applying the chain rule to the performative risk: \[\nabla_\theta \mathrm{PR}(\theta) = \underbrace{\mathbb{E}_{Z\sim\mathcal{D}(\theta)}\big[\nabla_\theta \ell(Z;\theta)\big]}_{\text{fit term}} \;+\; \underbrace{\nabla_\theta\,\mathbb{E}_{Z\sim\mathcal{D}(\theta)}\big[\ell(Z;\theta')\big]\Big|_{\theta'=\theta}}_{\text{reshaping term}}.\] The fit term is the gradient we would write down if the distribution were static. The reshaping term is the part new to performative prediction: it measures how perturbing \(\theta\) deforms the distribution \(\mathcal{D}(\theta)\) itself, and how much that deformation costs us in expected loss. The optimum is (defined to be?) where the two cancel — where the marginal gain from fitting the data better is exactly offset by the marginal cost it produces.
A performatively stable point \(\theta_{\mathrm{PS}}\) instead satisfies a fixed-point condition, \[\theta_{\mathrm{PS}} = \arg\min_\theta \mathbb{E}_{Z\sim\mathcal{D}(\theta_{\mathrm{PS}})}\,\ell(Z;\theta),\] i.e. we are satisfied with a \(\theta_{\mathrm{PS}}\) if, given an induced world, the model is already optimal for it, so refitting recovers the same parameters.
Apparently, the two do not coincide; the stable model is not in general the best one. Its special feature is that we suspect it can be found by hyperstitious iteration. Practitioners already talk about this as retraining: refit on whatever distribution the last model produced, \[\theta_{t+1} = \arg\min_\theta \mathbb{E}_{Z\sim\mathcal{D}(\theta_t)}\,\ell(Z;\theta).\]
Perdomo and co-authors call this repeated risk minimization and derive a contraction guarantee: if the loss is smooth and strongly convex, and the map \(\mathcal{D}(\cdot)\) is sufficiently Lipschitz in Wasserstein distance, then the iteration converges to a stable point at a linear rate, the error shrinking by a constant factor each step. Put another way, “does the hyperstition converge or explode?” turns upon the question of how sensitive the world is to the model.
1 Strategic classification
Everything above is agnostic about why the distribution moves. \(\mathcal{D}(\theta)\) is just some map. The contraction result depends only on how Lipschitz it is, not any specific interpretation. Strategic classification (Hardt et al. 2016) is a specific generative story. Here, the distribution moves because the people being classified are gaming the classifier.
The setup is a Stackelberg game. The institution moves first, publishing a classifier \(\theta\) (credit worthiness, food stamp worthiness, tax credits, etc) where the value of the classifier is the score it assigns to the features \(x\) of an applicant \(s_\theta(x)\). Each agent with features \(x\), seeing the rule, attempts to land on the favourable side of it by adjusting the features they present to the algorithm. Each agent pay some cost \(c(x, x')\) for presenting features \(x'\) in place of their “true” (default?) \(x\). This might correspond to my cost for buying a new suit when going for that loan etc. Now each agent best-responds, trading the score improvement \(s_\theta(x') - s_\theta(x)\) against manipulation costs : \[\Delta_\theta(x) = \arg\max_{x'}\big[\, s_\theta(x') - c(x, x') \,\big].\] The induced distribution \(\mathcal{D}(\theta)\) is then the base distribution pushed forward through this best-response map. This gives economic microfoundations to performativity. We can, in principle, deduce how perturbing \(\theta\) deforms the new observations, because we have written down the optimisation each agent solves against it.
In the basic version the features move but the “true” labels do not. I dress up in the fancy suit \(x\) to clear the credit-worthy bar without becoming any more “fundamentally creditworthy” underneath, by, e.g. saving more money. This is pure adversarial Goodhart where the metric is prised loose from the thing it was meant to measure.
There exist richer variants that relax this. If perturbing some feature actually moves the outcome — studying for the exam rather than buying the answers — then the manipulation is improvement rather than gaming, and we might in fact aim to design \(\theta\) so that the cheap moves are the improving ones. Whether a given classifier rewards gaming or improvement is one of the things this formalism is designed to resolve.
Miller, Milli, and Hardt (2019) observes that drawing this distinction at all drags us out of pure prediction and into causal inference. Whether nudging a feature games the classifier or improves the outcome depends on whether that feature causes \(Y\) or is merely correlated with it — so an institution that wants to reward improvement has to know the causal graph relating features to the outcome, not just the predictive correlations a classifier blithely uses. That is to say, a classifier that incentivizes improvement is a designed intervention on the population. Pinning down which levers a decision rule pulls — and so what behaviour it rewards — is a job for the causal-influence-diagram account of incentives. That is in general harder than merely fitting \(\theta\) — it inherits every identifiability headache of causal inference. An yet, we need to do it: Naïve retraining will happily converge to a classifier that everyone games and no one improves under, e.g. where people who want loans max out their credit cards buying fancy suits.
Mapping this strategic play back on to performative prediction: The reshaping term of the performative gradient in this setting is the derivative of the agents’ best response \(\Delta_\theta\), measuring how much harder everyone games as we move the classification rule. A performatively stable classifier is one that stays optimal against the gaming perturbations it provokes: we have already priced in how people will respond, so refitting buys us nothing. The catch is that \(\Delta_\theta\) includes an \(\arg\max\), so that reshaping term need not be at all well-behaved or, for example, smooth. Levanon and Rosenfeld (2021) get the chain rule to run in practice by swapping the hard best-response for a smooth surrogate, to make the objective differentiable end-to-end. And of course, that is even assuming we know the cost function; if we only see the gamed features that come back, we are in the revealed-preferences regime of Dong et al. (2018) — learning the game while we are playing it.
There are some interesting extensions. When the classified agents do not act in isolation but rather coordinate, we are in algorithmic collective action (Hardt et al. 2024). This is interesting for regimes wherein even a small collective can steer what the model learns — the mirror image of the platform’s own performative power (Hardt, Jagadeesan, and Mendler-Dünner 2022). When we want to quantify whether a principal can instead keep the players from coordinating, that is the divide-and-rule coalition game.
2 Seeing like a model
Step back far enough and the whole performative setup is what happens when a measuring instrument is also an instrument of government. This is the territory James C. Scott maps in Seeing Like a State, which I pick over in legibility and automation and metis and modernity: to run a forest, a city, or a workforce at scale, the state first has to make it legible, flattening its teeming specificities into gridded categories a central office can read at a glance. That act of flattening might, in these modern times, manifest as a deployed classifier \(\theta\). Indeed, a governed population does not persist unchanged beneath the gaze of the classifiers of the state, and we might expect both performative power and strategic classification to be in play.
Scott’s classic, photogenic disaster was the Normalbaum of scientific forestry: a forest replanted as tidy rows of legible timber, a triumph for one rotation and an ecological collapse by the second. Board-feet and the forest settle into a self-consistent fixed point — performatively stable, and a catastrophe — the gap between stable and best. The collapse is, in Scott’s terms, the un-classifiable metis reasserting itself. Where the classified can act, as the credit applicant or the exam-sitter can, classifier gaming can be strategic action (Goodhard again, or tufta). Where they cannot, as a forest cannot, the result is just a bad outcome. The Normalbaum is the second kind; a human under a bureaucracy gets both at once.
We are used to the idea, in modern technocraies, that once resources, status and power flow through a category its definition becomes a contested thing. Gender, obscenity, creditworthiness, refugee status, tax bracket: these are essential categories for modern states. What changes here, is the automation of the judgements, and the formalization of what the hell is going on when constructing these categories: things prove slippery.
3 Incoming
Connect this to
- opponent shaping and other
- iterated game cooperation problems.