The Predictive Approach to Bayesian Inference
Purely-predictive models, “The Italian school”, martingale posteriors, …
2025-07-10 — 2025-07-10
Suspiciously similar content
I don’t know much about this variant of Bayes, but the central idea is that we consider Bayes updating as a coherent betting rule and back everything else out from that. This gets us something like classic Bayes but with an even more austere approach to what probability is.
I am interested in this because, following an insight of Susan Wei’s, I note that it might be an interesting way of understanding when foundation models do optimal inference, since most neural networks are best understood as purely predictive models anyway.
What follows is an LLM-written introduction, which I am gradually massaging into something I would stand behind.
1 Background and Notation
Throughout this article we distinguish distributions (uppercase letters) from their densities (lowercase letters) where both exist. We work on an infinite sequence of observations
\[ X_1,\,X_2,\,\dots \]
taking values in some space \(\mathcal X\). A finite batch of data is denoted
\[ x_{1:n} \;=\; (x_1,\dots,x_n). \]
1.1 Classic (Parameter-Based) Bayesian Inference
Parameter: \(\theta\in\Theta\).
Prior distribution on \(\theta\): \(\Pi(d\theta)\), with density \(\pi(\theta)\).
Likelihood of data \(x_{1:n}\) under parameter \(\theta\):
\[ L(x_{1:n}\mid \theta) \quad\bigl(\text{often written }p(x_{1:n}\mid \theta)\bigr). \]
Posterior distribution of \(\theta\):
\[ \Pi(d\theta\mid x_{1:n}), \quad\text{with density } \pi(\theta\mid x_{1:n}) = \frac{L(x_{1:n}\mid \theta)\,\pi(\theta)}{\int_\Theta L(x_{1:n}\mid \vartheta)\,\pi(\vartheta)\,d\vartheta}. \]
Posterior predictive for a new observation \(X_{n+1}\):
\[ P_{\text{param}}\bigl(X_{n+1}\in A\mid x_{1:n}\bigr) = \int_\Theta P_\theta(X_{n+1}\in A)\;\Pi(d\theta\mid x_{1:n}), \]
or in density form
\[ p(x_{n+1}\mid x_{1:n}) = \int_\Theta p(x_{n+1}\mid \theta)\,\pi(\theta\mid x_{1:n})\,d\theta. \]
2 Predictive (de Finetti–Style) Inference
Rather than introduce an unobserved \(\theta\), we work directly with the sequence of one-step-ahead predictive distributions
\[ P_n(\,\cdot\mid x_{1:n}) \equiv \Pr\bigl\{X_{n+1}\in\cdot \,\bigm|\,X_{1:n}=x_{1:n}\bigr\}, \qquad n=0,1,2,\dots, \]
where:
- \(P_0\) is the prior predictive (your belief about \(X_1\) before seeing data).
- \(P_n\) is the predictive rule after seeing \(x_{1:n}\).
If these \(P_n\) satisfy coherence and exchangeability conditions (symmetry in \(x_{1:n}\), Kolmogorov consistency), then by de Finetti’s theorem there exists an (implicit) random measure \(F\) so that
\[ P_n(A\mid x_{1:n}) = \mathbb{E}\bigl[F(A)\mid x_{1:n}\bigr]. \]
In effect, the \(P_n\) are your Bayesian updating, but you never need to write down—or sample—the posterior on \(F\).
2.1 Key notation for predictives
| Symbol | Meaning |
|---|---|
| \(P_n(\cdot\mid x_{1:n})\) | Predictive distribution for \(X_{n+1}\) given data \(x_{1:n}\). |
| \(p_n(x_{n+1}\mid x_{1:n})\) | Density of \(P_n\) when it exists (lowercase for densities). |
| \(\tilde F\) | Implicit random probability measure (“parameter” in de Finetti’s sense). |
| \(\Pi(dF)\) | Mixing measure on \(\tilde F\) (the de Finetti prior), usually never written out. |
| \(\alpha,F_0\) | Hyperparameters in Dirichlet-process examples: \(\alpha\) is concentration, \(F_0\) the base. |
2.2 Why Two Views?
- Parameter view is often familiar: you write down \(\pi(\theta)\) and \(p(x\mid\theta)\), update to \(\pi(\theta\mid x)\), then integrate out \(\theta\) to predict.
- Predictive view flips that: you directly specify how you would predict the next datum at each step. If your predictive rule is coherent, there automatically exists a Bayes model behind it.
For many modern nonparametric and robust methods, crafting the \(P_n\) is easier than choosing a high-dimensional prior. In some cases (e.g. Fortini–Petrone’s predictive resampling or Fong et al.’s martingale posteriors) one never explicitly deals with \(\tilde F\) or \(\theta\) at all.
Bayesian inference is traditionally introduced via unknown parameters: we place a prior on a parameter \(\theta\) and update it to a posterior given data, then use it for predictions. The predictive approach, in contrast, puts prediction at the center – probabilities are assigned directly to future observables rather than to parameters. This view was championed by Bruno de Finetti, who argued that probability statements should only refer to observable events, with parameters serving as a convenient fiction linking past data to future outcomes. In other words, the only meaningful uncertainty is about things we might observe. The parameter (if it exists at all) is merely a derived concept – often “the link in the chain from past experience to the probability of future observable facts” (to paraphrase de Finetti). This predictive Bayesianism has deep philosophical roots and has proven fruitful in practice, influencing modern developments in Bayesian nonparametrics and machine learning.
This article provides an introduction to the predictive framework, tracing a timeline of major developments from de Finetti’s seminal work through the Italian school of Bayesian statisticians (such as Regazzini, Berti, Fortini, Petrone and others) up to contemporary innovations like martingale posterior distributions by Fong, Holmes, and Walker. We outline key theorems that underpin this approach and illustrate them with examples. Both measure-theoretic foundations and applied intuition are given to build understanding. Our emphasis is on synthesizing results into an interpretable narrative for an outsider to this branch of Bayesian thinking. Along the way, we motivate philosophical insights through methodological innovations – showing by example how the predictive approach works and why it offers a compelling perspective on inference.
3 Historical Timeline of the Predictive Framework
1930s – de Finetti’s Foundation: In 1937, Bruno de Finetti published his famous representation theorem for exchangeable sequences, laying the cornerstone of predictive Bayesian inference. An infinite sequence of observations \(X\_1,X\_2,\dots\) is exchangeable if its joint probability is invariant under permutation of indices. De Finetti’s theorem states that any infinite exchangeable sequence is equivalent to iid sampling from some latent random probability distribution \(F\); for suitable observables (say \(X\_i\in\mathbb{R}\)):
\[ P(X_1\in dx_1,\dots,X_n\in dx_n) \;=\; \int \prod_{i=1}^n F(dx_i)\; \mu(dF)\,, \]
where \(\mu\) is a “mixing” measure on distribution functions \(F\) (this \(\mu\) serves as a prior over \(F\) in Bayesian terms). Intuitively, if we believe the \(X\_i\) are exchangeable, we act as if there is some unknown true distribution \(F\) governing them; given \(F\), the data are iid. De Finetti emphasized that \(F\) itself is an unobservable construct – what matters are the predictive probabilities \(P(X\_{n+1}\in A \mid X\_{1\:n}=x\_{1\:n})\) for future observations. His philosophical stance was that probability is about your belief in future observable events, not about abstract parameters. He often illustrated this with betting and forecasting interpretations, effectively treating inference as an updating of predictive “previsions” (expectations of future quantities). De Finetti’s ideas formed the philosophical bedrock of the Italian school of subjective Bayesianism, shifting focus toward prediction.
1950s – Formalization of Exchangeability: Following de Finetti, mathematical statisticians solidified the theoretical underpinnings. Hewitt and Savage (1955) provided a rigorous existence proof for de Finetti’s representation via measure-theoretic extension theorems (ensuring a mixing measure \(\mu\) exists for any exchangeable law). This period established exchangeability as a fundamental concept in Bayesian theory. Simply put, exchangeability = “iid given some \(F\)”. This result, sometimes called de Finetti’s theorem, became a “cornerstone of modern Bayesian theory”. It means that specifying a prior on the parameter (or on \(F\)) is mathematically equivalent to specifying a predictive rule for the sequence. In fact, one can recover de Finetti’s mixture form by multiplying one-step-ahead predictive probabilities:
\[ P(x_1,\dots,x_n) = P(x_1) \, P(x_2\mid x_1)\cdots P(x_n\mid x_{1:n-1})\,, \]
and for an exchangeable model this product must equal the integral above. This insight – that a joint distribution can be factorized into sequential predictive distributions – is central to the predictive approach.
1970s – Bayesian Nonparametrics and Urn Schemes: Decades later, de Finetti’s predictive philosophy found new life in Bayesian nonparametric (BNP) methods. In 1973, Blackwell and MacQueen (1973) introduced the Pólya’s urn scheme as a constructive predictive rule for the Dirichlet Process (DP) prior, which Ferguson had proposed that same year as a nonparametric prior on distributions. Blackwell and MacQueen showed that if \(X\_1\sim F\_0\) (a base distribution) and for each \(n\ge1\), \(X_{n+1}\mid X_{1:n} \sim \frac{\alpha}{\alpha+n}F_0 + \frac{1}{\alpha+n}\sum_{i=1}^n \delta_{X_i}\,,\) then the sequence \((X\_n)\_{n\ge1}\) is exchangeable and in fact \(F\_0\) mixed with the urn scheme defines a Dirichlet process law. In this predictive rule, with probability \(\frac{\alpha}{\alpha+n}\) the \((n+1)\)th draw is a new value sampled from \(F\_0\), and with probability \(\frac{n\_j}{\alpha+n}\) it repeats one of the previously seen values (specifically, it equals the j-th distinct value seen so far, which occurred \(n\_j\) times). This elegant scheme generates clusters of identical values and is the basis of the Chinese restaurant process in machine learning. Importantly, it required no explicit mention of a parameter – the predictive probabilities themselves defined the model. The Dirichlet process became the canonical example of a prior that is constructed via predictive distributions. Around the same time, Cifarelli and Regazzini (1978) in Italy discussed Bayesian nonparametric problems under exchangeability, and Ewens’s sampling formula (Ewens 1972) in population genetics provided another famous predictive rule for random partitions of species. These developments showed the power of de Finetti’s idea: one can build rich new models by directly formulating how observations predict new ones.
1980s – Predictive Inference and Model Assessment: By the 1980s, the predictive viewpoint began influencing statistical practice and philosophy outside of nonparametrics. Seymour Geisser advanced the idea that predictive ability is the ultimate test of a model – he promoted predictive model checking and advocated using the posterior predictive distribution for model assessment and selection (foundational to modern cross-validation approaches). In 1981, Rubin introduced the Bayesian bootstrap, an alternative to the classical bootstrap, which can be seen as a predictive inferential method: it effectively assumes an exchangeable model where the “prior” is that the \(n\) observed data points are a finite population from which future samples are drawn uniformly at random. The Bayesian bootstrap’s posterior predictive for a new observation is simply the empirical distribution of the observed sample (with random weights), which aligns with de Finetti’s view of directly assigning probabilities to future data without a parametric likelihood. Ghosh and Meeden (Ghosh and Meeden 1986; Ghosh 2021) further developed Bayesian predictive methods for finite population sampling, treating the unknown finite population values as exchangeable and focusing on predicting the unseen units – again, no explicit parametric likelihood was needed. These works kept alive the notion that Bayesian inference “a la de Finetti” – with predictions first – could be practically useful. However, at the time, mainstream Bayesian statistics still largely centred on parametric models and priors, so the predictive approach was a somewhat heterodox perspective, championed by a subset of Bayesian thinkers.
1990s – The Italian School and Generalized Exchangeability: The 1990s saw renewed theoretical interest in characterizing exchangeable structures via predictions. Partial exchangeability (where data have subgroup invariances, like Markov exchangeability or other structured dependence) became a focus. In 1995, Jim Pitman generalized the Pólya urn to a two-parameter family (the Pitman–Yor process), broadening the class of predictive rules to capture power-law behavior in frequencies (Pitman 1995). In Italy, scholars like Eugenio Regazzini, Pietro Muliere, and their collaborators began exploring reinforced urn processes and other predictive constructions for more complex sequences. For example, Pietro Muliere and Petrone (1993) applied predictive mixtures of Dirichlet processes in regression problems, and P. Muliere, Secchi, and Walker (2000) introduced reinforced urn models for survival data. These models were essentially Markov chains whose transition probabilities update with reinforcement (i.e. past observations feed back into future transition probabilities), and they showed such sequences are mixtures of Markov chains – a type of partially exchangeable structure. Throughout, the strategy was to start by positing a plausible form for the one-step predictive distribution and then deduce the existence and form of the underlying probability law or “prior.” This reversed the conventional approach: instead of specifying a prior then deriving predictions, one specified predictions and thereby defined an implicit prior. By the end of the 90s, the groundwork was laid for a systematic predictive construction of Bayesian models.
2000s – Predictive Characterizations and New Priors: In 2000, a landmark paper (Fortini, Ladelli, and Regazzini 2000) formalized the conditions for a predictive rule to yield exchangeability. They gave precise necessary and sufficient conditions on a sequence of conditional distributions \((P\_n)\) such that there exists some exchangeable joint law \(P\) producing them. In essence, they proved that symmetry (the predictive probabilities depend on data only through symmetric functions like counts) and a certain consistency (related to associative conditioning of future predictions) characterize exchangeability. This result (along with earlier work by Diaconis and Freedman on sufficiency) provided a rigorous predictive criterion: you can validate if a proposed prediction rule is coherent (comes from some exchangeable model) without explicitly constructing the latent parameter. Around the same time, new priors in BNP were being defined via predictive structures. For instance, the species sampling models (Boothby, Pitman, etc.) were recognized as those exchangeable sequences with a general predictive form \(P(X\_{n+1}= \text{new} \mid X\_{1\:n}) = \frac{\alpha + c k}{\alpha + n}\) (for some constants \(c,\alpha\) and \(k\) distinct values so far), which yields various generalizations of the Dirichlet process. The Italian school played a leading role: they worked out how popular nonparametric priors like Dirichlet processes, Pitman–Yor processes, and others can be derived from a sequence of predictive probabilities. Priors by prediction became a theme. Fortini and Petrone (2012) wrote a comprehensive review on predictive construction of priors for both exchangeable and partially exchangeable scenarios. They highlighted theoretical connections and revisited classical results “to shed light on theoretical connections” among predictive constructions. By the end of the 2000s, it was clear that one could either start with a prior or directly with a predictive mechanism – the two routes were provably equivalent if done consistently, but the predictive route often yielded new insights.
2010s – Consolidation and Wider Adoption: In the 2010s, the predictive approach gained broader recognition and was increasingly connected to modern statistical learning. Fortini and Petrone continued to publish a series of works extending the theory: they explored predictive sufficiency (identifying what summary of data preserves all information for predicting new data), and they characterized a range of complex priors via predictive rules (from hierarchical priors to hidden Markov models built on predictive constructions). For example, they showed how an infinite Hidden Markov Model (used in machine learning for clustering time series) can be seen as a mixture of Markov chains, constructed by a sequence of predictive transition distributions. Meanwhile, machine learning researchers, notably in the topic modelling and Bayesian nonparametric clustering communities, adopted the language of exchangeable partitions (the Chinese restaurant process, Indian buffet process, etc., all essentially predictive rules). The review article Fortini and Petrone (2016) distilled the philosophy and noted how the predictive approach had become central both to Bayesian foundations and to practical modelling in nonparametrics and ML. Another development was the exploration of conditionally identically distributed (CID) sequences (weaker than full exchangeability) and other relaxations – these allow some trend or covariate effects while retaining a predictive structure. Researchers like Berti contributed here, defining models where only a subset of predictive probabilities are constrained by symmetry (Berti, Pratelli, and Rigo 2004, 2012). All these efforts reinforced that de Finetti’s perspective is not just philosophical musing – it leads to concrete new models and methods.
2020s – Martingale Posteriors and Prior-Free Bayesianism: Very recent years have witnessed a surge of interest in prior-free or prediction-driven Bayesian updating rules. Two parallel lines of work – one by Fong, Holmes, and Walker in the UK, and another by Berti, Rigo, and collaborators in Italy – have pushed the predictive approach to its logical extreme: conduct Bayesian inference entirely through predictive distributions, with no explicit prior at all. Edwin Fong’s D.Phil. thesis (C. H. E. Fong 2021) and subsequent papers introduced the Martingale Posterior framework. The core idea is to view the “parameter” as the infinite sequence of future (or missing) observations. If one had the entire population or the entire infinite sequence \(Y\_{n+1:\infty}\), any parameter of interest (like the true mean, or the underlying distribution \(F\)) would be known exactly. Thus uncertainty about \(\theta\) is really uncertainty about the as-yet-unseen data. Fong et al. formalize this by directly assigning a joint predictive distribution for all future observations given the observed \(Y\_{1\:n}\). In notation, instead of a posterior \(p(\theta\mid Y\_{1\:n})\), they consider \(p(Y\_{n+1:\infty}\mid Y\_{1\:n})\). This is a huge distribution (over an infinite sequence), but under exchangeability it encodes the same information as a posterior on \(\theta\). In fact, there is a one-to-one correspondence: if you choose the predictive distribution in the standard Bayesian way (by integrating the likelihood against a prior), then Doob’s martingale theorem implies the induced distribution on \(\theta\) is exactly the usual posterior. Fong and colleagues instead relax this: they allow the user to specify any predictive mechanism (any sequence of one-step-ahead predictive densities) that seems reasonable for the problem, not necessarily derived from a likelihood-prior pair. As long as these predictive densities are coherent (a martingale in the sense of not contradicting themselves over time), one can define an implicit “posterior” for \(\theta\) or for any function of the unseen data. They dub this the martingale posterior distribution, which “returns Bayesian uncertainty directly on any statistic of interest without the need for the likelihood and prior”. In practice, they introduce an algorithm called predictive resampling to draw samples from the martingale posterior. Essentially, one iteratively samples pseudo-future observations from the chosen predictive rule to impute an entire fake “completion” of the data, uses that to compute the statistic of interest, and repeats – thereby approximating the distribution of that statistic under the assumed predictive model. Martingale posteriors generalize Bayesian inference, subsuming standard posteriors when the predictive comes from a usual model, but also allowing robust or model-misspecified settings to be handled by choosing an alternative predictive (e.g. one might choose a heavy-tailed predictive density to guard against outliers, implicitly yielding a different “posterior”).
In parallel, Berti et al. (2023) developed a similar idea of Bayesian predictive inference without a prior. They work axiomatically with a user-specified sequence of predictives \({\sigma\_n(\cdot\mid x\_{1\:n})}\_{n\ge0}\) and establish general results for consistency and asymptotics of the resulting inference. One main advantage, as they note, is “no prior probability has to be selected” – the only inputs are the data and the predictive rule. These cutting-edge developments show how de Finetti’s viewpoint – once considered philosophically radical – is now driving methodological innovation for large-scale and robust Bayesian analysis. Today, the predictive approach is not only a cornerstone of Bayesian foundations but also an active area of research in its own right, influencing topics from machine learning (e.g. sequence modelling and meta-learning) to the theory of Bayesian asymptotics.
4 Key Concepts and Theoretical Foundations
To fully appreciate the predictive framework, we need to understand its mathematical backbone: exchangeability, predictive distributions, and coherence. We now review these core concepts, stating important theorems and illustrating with examples. We use measure-theoretic notation where appropriate, but also connect each result to intuition and practical examples.
4.1 Exchangeability and de Finetti’s Theorem
Exchangeability means that one’s probabilistic beliefs about a sequence of random variables do not depend on the order in which data are observed. Formally, \((X\_1,\ldots,X\_n)\) is exchangeable if for every permutation \(\pi\) of \({1,\dots,n}\), the joint distribution satisfies \(P(X\_1\in dx\_1,\dots,X\_n\in dx\_n) = P(X\_{\pi(1)}\in dx\_1,\dots,X\_{\pi(n)}\in dx\_n)\). De Finetti’s representation theorem (mentioned in the timeline) is the foundation: any infinite exchangeable sequence is a conditionally iid sequence. There exists a (generally unknown) random distribution \(F\) such that, given \(F\), the \(X\_i\) are iid with law \(F\). In Bayesian terms, \(F\) plays the role of a parameter – in fact one can think of \(F\) as \(\theta\) and put a prior on it. A more precise statement in measure-theoretic form is: If \((X\_n)\_{n\ge1}\) is exchangeable on a space \(\mathcal{X}\), then there exists a probability measure \(\mu\) on the set of probability distributions over \(\mathcal{X}\) (this set can be topologized suitably) such that for every \(n\) and any measurable events \(A\_1,\dots,A\_n\subseteq\mathcal{X}\),
\[ P(X_1\in A_1,\dots,X_n\in A_n) \;=\; \int \prod_{i=1}^n F(A_i)\; \mu(dF)\,. \]
Here \(\mu\) is often called the de Finetti mixing measure or the law of the random \(F\). If \(\mu\) were known, we could sample \(F\sim \mu\) (this is like drawing a random parameter) and then sample \(X\_1,\dots,X\_n\iid F\). Of course \(\mu\) is typically not known – it is merely asserted to exist by the theorem – but this representation has two crucial implications:
The joint distribution factors into a product of predictive terms. As noted earlier, \(P(x\_1,\dots,x\_n) = \int \prod\_{i=1}^n F(dx\_i),\mu(dF)\). By Fubini’s theorem, one can swap integration order to see this as an iterative conditioning: first draw \(F\sim\mu\), then \(X\_1\sim F\); next draw \(X\_2\sim F\), etc. The result is exactly \(P(x\_1)P(x\_2|x\_1)\cdots P(x\_n|x\_{1\:n-1})\). In fact, the one-step predictive distribution emerges naturally: \(P(X_{n+1}\in A \mid X_{1:n}=x_{1:n}) \;=\; \int F(A)\; \mu(dF\mid X_{1:n}=x_{1:n})\,,\) where \(\mu(\cdot\mid X\_{1\:n}=x\_{1\:n})\) is the posterior distribution of \(F\) given the observed data. This formula formalizes a simple idea: the predictive distribution for a new observation is the posterior mean of \(F\) (the “urn” distribution). For example, in a coin-flip scenario, \(F\) would be the true (but unknown) distribution of Heads/Tails; given some data, the predictive probability of Heads on the next flip is the posterior expected value of \(F({\text{Heads}})\). If we had a Beta\((a,b)\) prior on the coin’s bias \(p=F({\text{Heads}})\), this predictive is \(\frac{a + \text{(heads so far)}}{a+b+n}\) – the well-known Beta-Binomial rule. Indeed, taking \(a=b=1\) (a uniform prior), this gives \((1+k)/(2+n)\) after \(k\) heads in \(n\) tosses, which is Laplace’s “rule of succession.” Notice how naturally the predictive viewpoint reproduces such formulas: it is literally the posterior expectation of the unknown frequency.
Learning is reflected in the convergence of predictive distributions. Under exchangeability, the Strong Law of Large Numbers implies the empirical distribution of observations converges to the underlying \(F\) almost surely. In predictive terms, this means our predictive distribution eventually concentrates. A precise statement is: with probability 1, \(P(X\_{n+1}\in \cdot \mid X\_{1\:n})\) converges (as \(n\to\infty\)) to a degenerate distribution that puts all its mass on the limiting empirical frequency \(F,\). In other words, as we see more data, the sequence of predictive measures \((P\_n)\) forms a martingale that converges to the true distribution \(F\) almost surely. This is a powerful consistency property: no matter what the true \(F\) is, an exchangeable Bayesian will (almost surely) eventually have predictive probabilities that match \(F\) on every set \(A\). In classical terms, the posterior on \(F\) converges to a point mass at the truth (if the model is well-specified). This was first shown by Joseph Doob in 1949 using martingale theory. It provides a frequentist validation of Bayesian learning in the exchangeable case. But from the predictive perspective, it says if your predictive rule is coherent and eventually you see enough data, you will effectively discover the data-generating distribution – your forecasts become indistinguishable from frequencies in the long run.
4.2 Predictive Distributions and Coherence Conditions
A predictive rule or predictive distribution sequence is a collection \({P\_n(\cdot \mid x\_{1\:n}) : n\ge0,; x\_{1\:n}\in \mathcal{X}^n}\) where each \(P\_n(\cdot\mid x\_{1\:n})\) is a probability measure for the next observation \(X\_{n+1}\) given past observations \(x\_{1\:n}\). (For \(n=0\), we have \(P\_0(\cdot)\) which is just the distribution of \(X\_1\) before seeing any data.) Not every arbitrary choice of predictive rules corresponds to a valid joint distribution – they must satisfy coherence constraints. The fundamental coherence condition is that there exists some joint probability law \(P\) on the sequence such that for all \(n\) and events \(A\subseteq\mathcal{X}\):
\[ P_n(A \mid X_{1:n}=x_{1:n}) \;=\; P(X_{n+1}\in A \mid X_{1:n}=x_{1:n})\,, \]
i.e. the kernel \(P\_n\) truly comes from that joint law. By the Ionescu–Tulcea extension theorem, if one can specify a family of conditional probabilities in a consistent way for every finite stage, a unique process measure \(P\) on the sequence (on \((\mathcal{X})^\infty\)) is induced. Thus, one route to define a Bayesian model is: pick a predictive rule and invoke Ionescu–Tulcea to get the joint. However, ensuring consistency and exchangeability in the predictive rule is key. Fortini, Ladelli, and Regazzini (2000) gave an elegant description of the necessary and sufficient conditions for exchangeability in terms of predictives. In plain language, the conditions are:
Symmetry: If you condition on any past data \(x\_{1\:n}\), the predictive distribution \(P\_n(\cdot \mid x\_{1\:n})\) must be a symmetric function of \(x\_{1\:n}\). That is, it depends on the past observations only through their multiset or empirical frequency. This is intuitively clear – if the order of past data doesn’t matter for joint probabilities, it shouldn’t matter for predicting the next observation either. So, for example, in an exchangeable coin toss model, \(P(X\_{n+1}=\text{Heads}\mid X\_{1\:n})\) can only depend on the number of heads in \(n\) tosses (and \(n\)), not on the exact sequence.
Associativity (Consistency): This one is more technical. Roughly it means the predictive rule must be internally consistent when extended to two steps. Formally, for any \(n\ge0\), for any events \(A,B\) in the state space, one requires
\[ \int_A P_{n+1}(B \mid X_{1:n+1}=x_{1:n},x_{n+1}) \; P_n(dx_{n+1}\in A \mid x_{1:n}) \;=\; \int_B P_{n+1}(A \mid X_{1:n+1}=x_{1:n},x_{n+1}) \; P_n(dx_{n+1}\in B \mid x_{1:n})\,, \]
and this should hold for all past \(x\_{1\:n}\). Although this expression looks complicated, it is basically saying: if you consider two future time points \(n+1\) and \(n+2\), the joint predictive for \((X\_{n+1}, X\_{n+2})\) should be symmetric in those two (because the entire sequence is exchangeable). It ensures that your one-step predictive extends consistently to a two-step (and hence multi-step) predictive. In intuitive terms, condition (ii) is related to Kolmogorov consistency for the projective family of predictive distributions and the requirement of exchangeability on future samples. If symmetry (i) holds and this consistency (ii) holds, then there exists an exchangeable joint law producing those predictives. These are essentially the predictive analog of de Finetti’s theorem: they characterize when a given predictive specification is “valid” (arises from some mixture model).
The takeaway is that to design a Bayesian model in predictive form, one must propose a rule \(P\_n(\cdot\mid x\_{1\:n})\) that is symmetric and coherent across time. If you do, a de Finetti-type parameter \(\Theta\) (the random distribution \(F\)) is implicitly defined as the thing whose posterior predictive matches your rule. In fact, the predictive approach often yields an implicit description of the prior on \(\Theta\) without having to write it down explicitly. A classic example illustrates this nicely:
Example – Pólya’s Urn (Dirichlet Process) Revisited. Suppose we want an exchangeable model for draws \(X\_i\) taking values in some space (say \(\mathbb{R}\) or a discrete set) such that: (a) the first draw \(X\_1\) has some distribution \(F\_0\) (a “baseline” measure), and (b) at each step, the probability of seeing a new value that has never been seen before is proportional to a constant \(\alpha\), while the probability of seeing a value already seen is proportional to how many times it was seen. Formally, we set:
- \(P\_0(\cdot) = F\_0(\cdot)\),
- For \(n\ge1\), given past data \(x\_{1\:n}\) with \(k\) distinct values \({y\_1,\dots,y\_k}\) appearing in counts \(n\_1,\dots,n\_k\), define \(P_n(X_{n+1}=y_j \mid x_{1:n}) = \frac{n_j}{\,\alpha + n\,}, \qquad j=1,\dots,k,\) and \(P_n(X_{n+1} \text{ is a new value} \mid x_{1:n}) = \frac{\alpha}{\,\alpha + n\,}.\) If a new value is to appear, it is drawn from the base distribution \(F\_0\) (e.g. a new color in an urn is sampled from \(F\_0\)).
It is easy to check symmetry: the formulas depend on \((n\_1,\dots,n\_k)\), the counts of past values, which is symmetric in the data. The consistency condition can also be verified (and is a special case of the general theorem above) – essentially it holds because of the “reinforcement” form of the process (adding two observations in either order leads to the same updated counts). Therefore, by the theorem, there is an exchangeable joint law corresponding to this predictive rule. Blackwell and MacQueen (1973) proved that law’s de Finetti measure is the Dirichlet Process \(DP(\alpha F\_0)\). In other words, specifying this predictive scheme is equivalent to assuming \(F\sim DP(\alpha F\_0)\) as a prior on the distribution and then observing iid samples. But notice how much simpler the predictive description is: one can understand and simulate the model just by following the above recipe, without any direct invocation of a Dirichlet or Beta function. This predictive view also makes some properties obvious: e.g. the probability that the next observation is new remains \(\alpha/(\alpha+n)\) which tends to 0 as \(n\) grows, implying eventually all draws are repeats of existing values – hence the limiting random distribution \(F\) is discrete (almost surely, it’s an atomic distribution with infinitely many potential atoms). All that can be “seen” directly from the predictive rule, whereas deriving it from the Dirichlet process definition might require more work. This example underscores how one can “build priors by thinking predictively.” any other BNP priors have similar stories – for instance, the Pitman–Yor process has a two-parameter predictive rule \(P(\text{new}) \propto \alpha + c k\) and \(P(X\_{n+1}=y\_j) \propto n\_j - c\) for some \(0\le c<1\) (with some constraints to ensure nonnegativity), yielding a rich-get-richer scheme with power-law behavior in the counts. The underlying de Finetti measure in that case is a stable (or two-parameter Poisson–Dirichlet) process, but one can derive it by first principles via the predictive condition.
Example – Parametric Models and Sufficient Statistics. The predictive approach is fully compatible with parametric Bayesian models as well. If one has a parametric family \({f\_\theta(x)}\) and a prior \(\pi(d\theta)\), the traditional posterior predictive is \(P(X_{n+1}\in A \mid X_{1:n}=x_{1:n}) = \int_{\Theta} f_\theta(A)\; \pi(d\theta \mid X_{1:n}=x_{1:n})\,.\) This will obey the coherence conditions automatically (since it came from a joint model). But an interesting viewpoint is gained by focusing on predictive sufficiency. In many models, the posterior predictive depends on the data only through some summary \(T(x\_{1\:n})\). For example, in the Beta–Binomial model (coin flips with Beta prior), \(T\) can be the number of heads \(k\) in \(n\) flips. In a normal model with known variance and unknown mean, \(T\) could be the sample mean. In general, if \(T(x\_{1\:n})\) is a sufficient statistic for \(\theta\), then the predictive distribution \(P\_n(\cdot\mid x\_{1\:n})\) will be a function of \(T(x\_{1\:n})\). Fortini et al. (2000) discuss how predictive sufficiency relates to classical sufficiency. Essentially, a statistic \(T\_n = T(X\_{1\:n})\) is predictively sufficient if \(P\_n(X\_{n+1}\in \cdot \mid X\_{1\:n}) = P\_n(X\_{n+1}\in \cdot \mid T\_n)\) for all data – that is, the prediction for a new observation depends on past data only through \(T\_n\). For an exponential family with conjugate prior, this \(T\_n\) will usually be the same as the conventional sufficient statistic. Predictive sufficiency provides another angle: instead of deriving the posterior for \(\theta\), one can directly derive the predictive by updating \(T\). For example, in the normal-with-known-variance case, one can derive the Gaussian form of the predictive for the next observation by directly updating the mean and uncertainty of the sampling distribution, rather than explicitly updating a prior for \(\theta\). In summary, parametric Bayesian models yield predictive rules of a special form (ones that admit a finite-dimensional sufficient statistic). These are consistent with the general theory but represent cases where the predictive distributions lie on a lower-dimensional family. The predictive approach doesn’t conflict with parametric modelling – it generalizes it, and in practice it often yields the same results (since Bayes theorem ensures coherence). What it offers is a different interpretation of those results: your posterior is just a device to produce predictions, and you could have potentially produced those predictions without ever introducing \(\theta\) explicitly.
4.3 Martingales and the Martingale Posterior
One of the modern breakthroughs in predictive inference is recognizing the role of martingales in Bayesian updating. We saw earlier that the sequence of predictive distributions \((P\_n)\) under an exchangeable model is a martingale (in the weak sense that \(P\_{n}\) is the conditional expectation of \(P\_{n+1}\) given current information) and it converges to the true distribution \(F\) almost surely. E. Fong, Holmes, and Walker (2021) took this insight further: if the conventional Bayes posterior predictive yields a martingale that converges to a degenerate distribution at the true \(F\), perhaps one can construct other martingales that converge to something else, representing different uncertainty quantification. The martingale posterior is built on this idea. They start from the premise: “the foundation of Bayesian inference is to assign a distribution on missing observations conditional on what has been observed.” Rather than begin with a prior on \(\theta\), they ask: what distribution would a Bayesian assign to all the not-yet-seen data \(Y\_{n+1},Y\_{n+2},\dots\) given the seen \(Y\_{1\:n}\)? If one follows standard Bayes, the answer is clear: it would be the product of posterior predictives, i.e. \(P(Y\_{n+1:\infty}\in dy\_{n+1:\infty}\mid Y\_{1\:n}=y\_{1\:n}) = \int \prod\_{m=n+1}^\infty F(dy\_m); \mu(dF \mid Y\_{1\:n}=y\_{1\:n})\). This complicated object is basically “\(F\) drawn from the posterior, then future \(Y\) iid from \(F\).” From it, we can derive the usual posterior for any parameter \(\theta = T(F)\) as the induced distribution of \(\theta\). In fact, Doob’s theorem showed that if you go this route, the distribution of \(\theta\) defined in this way equals the standard posterior for \(\theta\). So nothing new so far – it just recapitulates that the Bayesian posterior is equivalent to the Bayesian predictive for the sequence. But now comes the twist: we are free to choose a different joint predictive for the future that is not derived from the prior predictive formula. If we do so, we get a different “posterior” for \(\theta\) when we invert that relationship. The only requirement is that our chosen joint predictive for future data still respects that \(Y\_{n+1:\infty}\), given \(Y\_{1\:n}\), forms a martingale as \(n\) grows (so that as \(n\to\infty\) it collapses appropriately). Fong et al. introduce a family of such predictive rules (including some based on copula dependence structures to handle when observations are not iid). The resulting martingale posterior is the probability distribution on the parameter (or on any estimand of interest) that corresponds to these new predictives. It is a true posterior in the sense that it is an updated distribution on the quantity given the data, but it need not come from any prior via Bayes’ rule. In practice, one can think of it as a Bayesian-like posterior that uses an alternative updating rule. The martingale posterior is chosen to satisfy certain optimality or robustness properties (for instance, one can design it to minimize some loss on predictions, or to coincide with bootstrap resampling in large samples). Indeed, Fong et al. note connections between martingale posteriors and the Bayesian bootstrap and other resampling schemes. The term “martingale” highlights that the sequence of these posteriors (as \(n\) increases) is a martingale in the space of probability measures, which ensures coherence over time. This concept is quite new, and ongoing research is examining its properties – for example, how close is a martingale posterior to the ordinary posterior as \(n\) becomes large? Early results indicate that if the model is well-specified, certain martingale posteriors are consistent (converge on the true parameter) and even asymptotically normal like a standard posterior. But if the model is misspecified, a cleverly chosen martingale posterior might offer advantages in terms of robustness (since it was not derived from the wrong likelihood). The theory here is deep, combining Bayesian ideas with martingale convergence and requiring new technical tools to verify things like convergence in distribution of these posterior-like objects. From a user’s perspective, however, the appeal is clear: you can get a posterior without specifying a prior or likelihood, by directly coding how you think future data should behave. This significantly lowers the barrier to doing Bayesian-style uncertainty quantification in complex problems – one can bypass the often difficult task of choosing a prior or fully specifying a likelihood and focus on predictions (which may be easier to elicit from experts or justify scientifically).
5 Summary of Theoretical Insights
At its heart, the predictive framework rests on a few key insights proven by the above results:
Observables suffice: If you can specify your beliefs about observable sequences (one-step-ahead at a time) in a consistent way, you have done the essential job of modelling. Theorems like de Finetti’s and its predictive extensions guarantee that a parameter-based description exists if you want it, but it’s optional. “Bayes à la de Finetti” means one-step probabilities are the building blocks.
Exchangeability is a powerful symmetry: It grants a form of “sufficientness” to empirical frequencies. For instance, in an exchangeable model, the predictive distribution for tomorrow given all past data depends only on the distribution of past data, not on their order. This leads to natural Bayesian consistency (learning from frequencies) and justifies why we often reduce data to summary statistics.
Predictive characterization of models: Many complex models (like BNP priors or hierarchical mixture models) can be characterized by their predictive rules. In some cases this yields simpler derivations. For example, it’s easier to verify an urn scheme than to prove a Chinese restaurant process formula from scratch. Predictive characterizations also allow extending models by modifying predictive rules (e.g. creating new priors via new urn schemes, such as adding reinforcement or memory).
Philosophical clarity: The predictive view clarifies what the “parameter” really is – usually some aspect of an imaginary infinite population. As Fong et al. put it, “the parameter of interest [is] known precisely given the entire population”. This demystifies \(\theta\): it’s not a mystical quantity but simply a function of all unseen data. Thus, debating whether \(\theta\) “exists” is moot – what exists are data (observed or not yet observed). This philosophy can be very practical: it encourages us to check our models by simulating future data (since the model is the prediction rule), and to judge success by calibration of predictions.
Flexibility and robustness: By freeing ourselves from always specifying a likelihood, we can create “posterior-like” updates that may be more robust. For instance, one could specify heavier-tailed predictive densities than a Gaussian model would give, to reduce sensitivity to outliers, and still have a coherent updating scheme that quantifies uncertainty. This is one motivation behind general Bayesian approaches and martingale posteriors.
6 Practical Methods and Examples
Let’s walk through a few concrete examples and methods where the predictive approach is applied. We’ve already discussed the Pólya urn (Dirichlet process) and a simple Beta-Bernoulli model. Here we highlight additional applied scenarios:
Bayesian Bootstrap (Rubin 1981): Suppose we have observed data \(x\_1,\dots,x\_n\) which we treat as a sample from some population distribution \(F\). The Bayesian bootstrap avoids choosing a parametric likelihood for \(F\) and instead puts a uniform prior on the space of all discrete distributions supported on \({x\_1,\dots,x\_n}\). In effect, one assumes exchangeability and that the true \(F\) puts all its mass on the observed points (as would be the case if these \(n\) points were literally the entire population values, but we just don’t know their weights). The posterior for \(F\) given the data then turns out to be a Dirichlet(\(1,1,\dots,1\)) on the point masses at \(x\_1,\dots,x\_n\). Consequently, the posterior predictive for a new observation \(X\_{n+1}\) is \(P(X_{n+1}=x_i \mid X_{1:n}=x_{1:n}) = \frac{1}{n}\,,\) for each \(i=1,\dots,n\). In other words, the next observation is equally likely to be any of the observed values. This is precisely the Bayesian bootstrap’s predictive distribution, which is just the empirical distribution of the sample (sometimes with one extra point allowed to be new, but with flat prior that new point gets zero mass posterior). The Bayesian bootstrap can be viewed as a limiting case of the Dirichlet process predictive rule when \(\alpha \to 0\) (or, dually, when one says “I believe all probability mass is already in the observed points”). It’s a prime example of how a predictive assumption (the next point is equally likely to be any observed point) leads to an implicit prior (Dirichlet(1,…,1) on weights) and a posterior (the random weights after a Dirichlet draw). The Bayesian bootstrap is often used to generate approximate posterior samples for parameters like the mean or quantiles without having to assume any specific data distribution. This method has gained popularity in Bayesian data analysis, especially in cases where a parametric model is hard to justify. It is an embodiment of de Finetti’s idea: you directly express belief about future draws (they should resemble past draws in distribution) and that is your model.
Predictive Model Checking and Cross-Validation: In Bayesian model evaluation, a common practice is to use the posterior predictive distribution to check goodness-of-fit: simulate replicated data \(X\_i^{\text{rep}}\) from \(P(X\_i^{\text{rep}}\mid X\_{1\:n})\) and compare to observed \(X\_i\). Any systematic difference may indicate model misfit. This is fundamentally a predictive approach: rather than testing hypotheses about parameters, we ask “does the model predict new data that look like the data we have?”. It aligns perfectly with the predictive view that the ultimate goal of inference is accurate prediction. In fact, modern Bayesian workflow encourages predictive checks at every step. Additionally, methods like leave-one-out cross-validation (LOO-CV) can be given a Bayesian justification via the predictive approach. The LOO-CV score is essentially the product of \(P(X\_i \mid X\_{-i})\) over all \(i\) (the probability of each left-out point under the predictive based on the rest). Selecting models by maximizing this score (or its logarithm) is equivalent to maximizing predictive fit. Some recent research (including by Fong and others) formally connects cross-validation to Bayesian marginal likelihood and even proposes cumulative cross-validation as a way to score models coherently. The philosophy is: a model is good if it predicts well, not just if it has high posterior probability a priori. By building model assessment on predictive distributions, we ensure the evaluation criteria align with the end-use of the model (prediction or forecasting).
Coresets and Large-Scale Bayesian Summarization: (Flores 2025) A very recent application of predictive thinking is in creating Bayesian coreset algorithms – these aim to compress large datasets into small weighted subsets that yield almost the same posterior inference. Traditionally, coreset construction tries to approximate the log-likelihood of the full data by a weighted log-likelihood of a subset (minimizing a KL divergence). However, this fails for complex non-iid models. Flores (2025) proposed to use a predictive coreset: choose a subset of points such that the posterior predictive distribution of the subset is close to that of the full data. In other words, rather than matching likelihoods, match how well the subset can predict new data like the full set would. This approach explicitly cites the predictive view of inference (E. Fong and Yiu 2024; Fortini and Petrone 2012) as inspiration. The result is an algorithm that works even for models where likelihoods are intractable (because it can operate on predictive draws). This is a cutting-edge example of methodological innovation driven by predictive thinking.
Machine Learning and Sequence Modelling: It’s worth noting that in machine learning, modern large models (like transformers) are often trained to do next-token prediction on sequences. In some recent conceptual work, researchers have drawn a connection between such pre-trained sequence models and de Finetti’s theory. Essentially, a large language model that’s been trained on tons of text is implicitly representing a predictive distribution for words given preceding words. If the data (text) were regarded as exchangeable in some blocks, the model is doing a kind of empirical Bayes (using the training corpus as prior experience) to predict new text. Some authors (Ye and Namkoong 2024) have even argued that in-context learning by these models is equivalent to Bayesian updating on latent features, “Bayesian inference à la de Finetti”. While these ideas are still speculative, they illustrate how the predictive perspective resonates in ML: the focus is entirely on \(P(\text{future tokens}\mid \text{past tokens})\). If one were to build an AI that learns like a Bayesian, it might well do so by honing its predictive distribution through experience, rather than by explicitly maintaining a distribution on parameters. This is essentially what these sequence models do, albeit not in a fully coherent probabilistic way.
6.1 Conclusion and Outlook
The bases of the predictive framework in Bayesian inference were laid by de Finetti nearly a century ago, but only in recent decades have we fully appreciated and utilized its power. By distilling Bayesian reasoning to its predictive essence, we gain conceptual clarity and new technical tools. The timeline we traced shows a steady broadening of this approach: from the philosophical insistence that “probability only speaks of observables,” to nonparametric priors defined via prediction, to algorithms and techniques that put prediction first (Bayesian bootstrap, model checks, coreset selection), and finally to generalizations of Bayes’ rule itself (martingale posteriors) that remove the straitjacket of the likelihood-prior framework and allow more direct specification of beliefs.
For an outsider to this type of Bayesianism, perhaps the most important message is: think about what you would predict, not what parameter you would estimate. In practice, formulating a model predictively can be very natural. For example, if modelling customer purchases, one might say: “Based on the purchases so far, the next purchase will likely be an item of a type we’ve seen frequently, with some chance of a new type appearing.” That’s a predictive statement – essentially an urn model. It can be translated into a Bayesian model (maybe a Dirichlet process mixture over item categories) but you, the statistician, might never need to do the translation yourself; the predictive story is enough to simulate and make inferences. The required theorems guarantee that if your story is consistent, it corresponds to some Bayesian model. The practical methods emerging from this mindset tend to emphasize flexibility (you can mix and match predictive rules) and often are computationally attractive (prediction is often easier to simulate than computing likelihoods or priors).
Of course, the predictive approach does not solve all problems. It can be challenging to elicit a full predictive distribution for complex data – sometimes specifying a likelihood in pieces feels easier. The approach also shifts some burdens: ensuring coherence of predictives can be tricky in high dimensions or structured settings. Nonetheless, it offers a rich, philosophically grounded toolkit. The Italian school of Bayesian statistics has given us a legacy of results and methods to build on, and recent work by Fong, Walker, Holmes, Berti, and others is pushing the boundaries, indicating that the predictive paradigm is very much alive and evolving. We can expect further philosophical insights (what is the meaning of a prior when you can do without it?) and methodological innovations (new robust posteriors, new ways to blend models by their predictive implications, etc.) to come. In the end, de Finetti’s vision of probability theory as “the science of prediction” seems more prescient than ever – guiding both the theory and practice of Bayesian inference into the future.