Sufficient statistics
Summarizing the data with a minimal representation
2018-03-13 — 2026-07-08
Wherein the Canonical Sufficient Statistic Is Revealed to Be a Probability Measure, and Its Many Aliases Across Statistics, Control Theory, and Machine Learning Are Catalogued.
🚧TODO🚧
I’m working through a small realization, for my own interest, which has helped my understanding of variational Bayes; and specifically in relating it to non-Bayes variational inference. Also sequential Monte Carlo.
Formally, a statistic \(T\) — any function of the data — is sufficient for \(\theta\) iff the conditional law of the data given \(T\) doesn’t depend on \(\theta\),
\[ p(x\mid T(x),\theta)=p(x\mid T(x)), \]
equivalently the Fisher–Neyman factorisation \(p(x;\theta)=h(x)\,g(T(x);\theta)\). Everything the data say about \(\theta\) is funnelled through \(T\); conditional on \(T\), the rest of \(x\) is noise as far as \(\theta\) is concerned.
Sufficiency is a property of a function of the data; it says nothing about where that function takes values, i.e. the codomain of \(T\).
Nothing in the definition forces \(T\) to be small, finite-dimensional, or even a vector. The dream, though, is that it is: a fixed-size summary we can carry around in place of the full dataset. In the easy case the codomain is \(\mathbb{R}^k\) with \(k\) fixed — a vector summary, our data is infinitely compressible.
Finite-dimensional sufficient statistics are the Garden of Edenference, when there is some fixed-size \(k\) such that \(T:\mathcal{X}\to\mathbb{R}^k\) is sufficient for every sample size \(n\). Turns out, Edenference is small. By Pitman–Koopman–Darmois (Pitman 1936; Koopman 1936; Darmois 1935), a sufficient statistic whose dimension stays fixed as the sample size \(n\) grows essentially characterises exponential families. Under even mild generalizations, things are not so nice. In mixtures, hierarchical models, anything modern including and especially neural networks — there is no fixed-size \(T\) that is exactly sufficient and which characterizes the posterior.
1 The sufficient statistic is a distribution
In general, the natural codomain of \(T\) is the space of probability measures on \(\Theta\), and the canonical sufficient statistic is the likelihood function, or equivalently in Bayes, the posterior distribution over \(\theta\) given \(x\). This is still a sufficient statistic, if we let \(T\) live somewhere bigger than \(\mathbb{R}^k\). The posterior \(\theta \mapsto p(\theta\mid x)\) is itself a function of the data — a statistic in the original sense — it just takes values in the space of probability measures on \(\Theta\) rather than in \(\mathbb{R}^k\). Bayesian sufficiency (Halmos and Savage 1949; Bahadur 1954) in fact says precisely this: \(T\) is sufficient iff, for every prior, the posterior depends on \(x\) only through \(T\) — so the posterior measure is itself sufficient, tautologically.
For prediction the analogue is the conditional law of the future given the past; Knight’s prediction process (Knight 1975) is the minimal Markov sufficient statistic for any process.
Now, it turns out measure-valued sufficient statistics pop up a lot; I hadn’t realised how many places they occur, despite seeing many objects that were remarkably similar. Most recently, I was writing up POMDP belief states and I was all like, damn that looks like a sufficient statistic — why does it have a funny name? I had simply never put these concepts together. A quick search via my friendly neighbourhood LLM revealed quite a few other places where the same object appears, under many different names:
| Field | Name it goes by | The object |
|---|---|---|
| Bayesian statistics | posterior | \(p(\theta\mid x)\); sufficient for every prior (Halmos and Savage 1949; Bahadur 1954) |
| Sequential analysis, Bayesian bandits | belief, Bayes-adaptive state | posterior over the unknown parameter; the state of the Gittins / BAMDP problem |
| Nonlinear filtering, HMMs | the filter, conditional law | \(\pi_t=\mathbb{P}(X_t\in\cdot\mid\mathcal{Y}_t)\) (Kushner 1964; Stratonovich 1960; Zakai 1969); collapses to mean + covariance in the Kalman / exponential-family case |
| Stochastic control | information state, hyperstate | (Åström 1965; Striebel 1965; KumarVaraiya1986Stochastic?); the separation principle acts on it |
| POMDPs, RL | belief state | \(b_t\in\Delta(\mathcal{S})\) — see the belief-MDP reduction |
| Stochastic processes | prediction process | (Knight 1975); the canonical measure-valued Markov sufficient statistic |
| Computational mechanics | causal state, ε-machine | (Shalizi2000Computational?); the minimal predictively-sufficient partition of histories |
| Graphical models | message, belief, mean parameter | a belief-propagation message is a sufficient statistic for its subtree’s evidence; EP projects it onto an exponential family (Minka2001Expectation?; Wainwright2008Graphical?) |
| Predictive state representations | predictive state | (Littman2001Predictive?); sufficiency stated over future observable tests rather than latent state |
| Exchangeability | de Finetti mixing measure | the latent “parameter” is itself a random distribution |
It seems we manage to act in the world without carrying around a copy of the world in our heads; so if we are doing something like statistical analysis of the world, we must be making some approximations. The question is how cheaply we can approximate that measure while losing as little as we can afford to (whatever that, in context, means).
We can think of many such ways: nearly sufficient statistics, variational inference (of the posterior), particle filters (of the prediction process), predictive state representations (of the latent state), and so on.