Performative prediction
A thing people worry about when they worry about “socially constructed” categories, and technocracy and prophecy
2026-06-09 — 2026-08-17
In Which a Machine Learning Model Is Mandated to Govern Populations, the World Is Mistaken for a Static Instrument, and Human Agency Is Salvaged Through the Strategic Gaming of Classification Boundaries.
An ML formalization of interesting phenomena such as legibilizing and hyperstition, likely implicit in recommender dynamics, adversarial classification and some types of external validity, among other places. HT TJ for the pointer.
In classical supervised learning we fit parameters \(\theta\) to minimize expected loss over a distribution \(\mathcal{D}\) \[\theta_{\mathrm{SL}} = \arg\min_\theta \mathbb{E}_{Z\sim\mathcal{D}}\,\ell(Z;\theta),\] with \(Z=(X,Y)\) a feature–outcome pair and \(\ell\) the loss. We assume that the distribution \(\mathcal{D}\) is fixed and that the model acts as a passive observer—a measuring instrument. Poetically, this assumes that while the model watches the world, the world does not watch back. Performative prediction (Perdomo et al. 2020; Hardt and Mendler-Dünner 2025) is a formalization of a different regime where deploying the model changes the subsequent data distribution.
The stock example (familiar from fairness) is assessing people for credit-worthiness. If we predict someone is at high risk of credit default, we might protectively assign them a punishing interest rate, which in turn increases their propensity to default, thereby confirming our prediction.
In performative prediction we model this directly, formally by assuming that each choice of parameters produces a perturbed data distribution \(\mathcal{D}(\theta)\)—the data we would see if we deployed \(\theta\) to do things in the world. In the case that the perturbation ‘looks like a self-fulfilling prophecy’, we call that a hyperstition.
We define the performative risk, \[\mathrm{PR}(\theta) = \mathbb{E}_{Z\sim\mathcal{D}(\theta)}\,\ell(Z;\theta).\] Here \(\theta\) appears twice: once as the model whose predictions we assess, and again as an influence on the distribution it produces when deployed. That second appearance puts us at odds with the techniques of ordinary regression — we cannot simply descend the gradient of the loss in isolation, because moving \(\theta\) also shifts the target.
There are two extensions to the definition that recover something like that classical supervised, static setting.
A performative optimum \(\theta_{\mathrm{PO}} = \arg\min_\theta \mathrm{PR}(\theta)\) minimizes the performative risk of this coupling via \(\theta\), solving for the implicit dependence of the distribution on the parameters. We derive this by applying the chain rule to the performative risk: \[\nabla_\theta \mathrm{PR}(\theta) = \underbrace{\mathbb{E}_{Z\sim\mathcal{D}(\theta)}\big[\nabla_\theta \ell(Z;\theta)\big]}_{\text{fit term}} \;+\; \underbrace{\nabla_\theta\,\mathbb{E}_{Z\sim\mathcal{D}(\theta)}\big[\ell(Z;\theta')\big]\Big|_{\theta'=\theta}}_{\text{reshaping term}}.\] The fit term is the gradient we would calculate if the distribution were static. The reshaping term is an innovation of performative prediction: it measures how perturbing \(\theta\) deforms the distribution \(\mathcal{D}(\theta)\) itself, and how much that deformation costs us in expected loss. This “optimum” is where the two cancel — where the marginal gain from fitting the data better is exactly offset by the marginal cost it produces.
A performatively stable point \(\theta_{\mathrm{PS}}\) is a different solution. This satisfies a fixed-point condition, \[\theta_{\mathrm{PS}} = \arg\min_\theta \mathbb{E}_{Z\sim\mathcal{D}(\theta_{\mathrm{PS}})}\,\ell(Z;\theta),\] i.e. we are satisfied with a \(\theta_{\mathrm{PS}}\) if, given an induced world, the model is already optimal for it, so refitting recovers the same parameters. This is similar to the result we get if we train the predictor, then use it in the wild, then update the predictor based on the new state of the world, and so on, until convergence. Practitioners already talk about this as retraining: refit on whatever distribution the last model produced, \[\theta_{t+1} = \arg\min_\theta \mathbb{E}_{Z\sim\mathcal{D}(\theta_t)}\,\ell(Z;\theta).\]
The reason we are here is that these notions do not coincide; the stable model is not, in general, the best one.
Perdomo and co-authors call this repeated risk minimization and derive a contraction guarantee: if the loss is smooth and strongly convex, and the map \(\mathcal{D}(\cdot)\) is sufficiently Lipschitz-smooth in Wasserstein distance, then the iteration converges to a stable point at a linear rate, with the error shrinking by a constant factor each step. Put another way, “does the hyperstition converge or explode?” turns upon the question of how sensitive the world is to the model, and sometimes the answer is yes.
1 Strategic classification
So far I have been generic and just talked about regression problems, remaining agnostic about why the distribution moves, and describing \(\mathcal{D}(\theta)\) as just some distribution whose shape determines the reshaping term.
Strategic classification (Hardt et al. 2016) is a specific generative interpretation: the distribution moves because the people being classified are gaming a classifier.
The setup is a Stackelberg game. The institution moves first, publishing a classifier \(\theta\) (credit worthiness, food stamp worthiness, tax credits, etc.) where the value of the classifier is the score it assigns to the features \(x\) of an applicant \(s_\theta(x)\). Each agent with features \(x\), seeing the rule, attempts to land on the favourable side of the class boundary by adjusting the features they present to the algorithm. Each agent pays some cost \(c(x, x')\) for presenting features \(x'\) in place of their “true” (default?) \(x\). We could imagine this corresponds to the cost of buying a new suit when applying for that loan, and so on. Now each agent best-responds, trading the score improvement \(s_\theta(x') - s_\theta(x)\) against manipulation costs: \[\Delta_\theta(x) = \arg\max_{x'}\big[\, s_\theta(x') - c(x, x') \,\big].\] The induced distribution \(\mathcal{D}(\theta)\) is then the base distribution pushed forward through this best-response map. Boom! Economic microfoundations of performativity. We can, in principle, deduce how perturbing \(\theta\) deforms the new observations, because we have written down the optimization each agent aims to solve.
In the basic version, the features move but the “true” labels do not. I dress up in a fancy suit \(x'\) to clear the credit-worthy bar without becoming any more “fundamentally creditworthy” underneath, by, e.g., saving more money. This is pure adversarial Goodhart where the metric is prized loose from the thing it was meant to measure.
There exist richer variants with more complicated dynamics. If perturbing some feature actually moves the outcome — studying for the exam rather than buying the answers — then the manipulation might be actual improvement rather than gaming, and we might in fact aim to design the game \(\theta\) so that the cheap perturbations of \(x\) are the improving ones. Whether a given classifier rewards gaming or improvement is one of the things this formalism is designed to resolve.
Miller, Milli, and Hardt (2019) observes that drawing this distinction at all drags us out of pure prediction and into causal inference. Whether nudging a feature games the classifier or improves the outcome depends on whether that feature causes \(Y\) or is merely correlated with it — so an institution that wants to reward improvement cannot blithely use the correlations (between, e.g., academic achievement and test score) but needs to learn the causal graph relating features to the outcome. That is to say, a classifier that incentivizes improvement is a designed intervention on the population. Interpreting causal effects of incentives in games is something we fortunately have tools for. That is in general harder than merely fitting \(\theta\) — it inherits every identifiability headache of causal inference. And yet, we need to do it: Naïve retraining can happily converge to a classifier that everyone games and no one improves under, e.g., where people who want loans max out their credit cards buying fancy suits.
Mapping this strategic play back onto the performative prediction formalism: The reshaping term of the performative gradient in this setting is the derivative of the agents’ best response \(\Delta_\theta\), measuring how much harder everyone games as we move the classification rule. A performatively stable classifier is one that stays optimal against the gaming perturbations it provokes: we have already priced in how people will respond, so refitting buys us nothing. The catch is that \(\Delta_\theta\) includes an \(\arg\max\), so that reshaping term need not be at all well-behaved or, for example, smooth. Levanon and Rosenfeld (2021) get the chain rule to run in practice by swapping the hard best-response for a smooth surrogate, to make the objective differentiable end-to-end. And of course, that is even assuming we know the cost function. If we only see the gamed features that come back, we are in the revealed-preferences regime, learning the game while we are playing it, which sounds punishingly hard (Dong et al. 2018).
There are some interesting extensions. When the classified agents do not act in isolation but rather coordinate to change the classifier, we are doing algorithmic collective action (Hardt et al. 2024). Conversely, we can consider performative power (Hardt, Jagadeesan, and Mendler-Dünner 2022), the ability of the classifying institution’s makers to shape the population’s behaviour. When we want to quantify whether a classifying institution can instead keep the players from coordinating, that is the divide-and-rule coalition game.
2 Seeing like a model
Step back far enough and the whole performative setup looks familiar from the classic observation that a measuring instrument is also an instrument of government as made famous by James C. Scott in Seeing Like a State, who cares greatly about legibility and automation not to mention the tense relationship between the irreproducible tacit metis and high modernity: to run a forest, a city, or a workforce at scale, the state first has to make it legible, which looks a hell of a lot like classifiable. We can improve classification accuracy in two ways: either by learning a more precise model, or by flattening the teeming specificities into more easily separable groups for the central office. Indeed, a governed population does not persist unchanged beneath the gaze of the classifiers of the state, and we might expect both performative power and strategic classification to be in play.
Scott’s photogenic example is a measurement procedure that structures the forest it was intended to analyze. The Normalbaum is a standardized tree of a given size class, its volume of saleable wood tabulated by size and age; the Norway spruce in straight rows came afterwards, planted to match the table, bending reality to the convenience of the predictor. The forest is not a strategic responder, however. Where the classified can act, as the credit applicant or the exam-sitter can, gaming the classifier is strategic action (Goodhart again, or tufta). Where they cannot, as a forest cannot, the reshaping term is still there and yet nobody is best-responding to us; that is a garden-variety unstrategic bad performative equilibrium.
We are accustomed to the idea, in modern technocracies, that once resources, status and power flow through a category its definition becomes a contested thing. Gender, obscenity, creditworthiness, refugee status, tax bracket: these are essential categories for modern states and thus contested. What changes in performative prediction is not that we “solve” for the “true” categories, nor that we throw up our hands in despair and decry that this is not what we were told in statistics class. Performative prediction gives us a means, rather, to understand and name how much we are seeking to understand reality, and how much to change it.
3 Incoming
- Expectation = intention = setpoint
- 🚧TODO🚧: very excited about Heitzig (2026) which has an application of this idea to performative forecasting