Conjugate priors

1 Exponential family likelihood

Our observation $X$ is assumed to arise from an exponential family likelihood. That is, given (vector) parameter $\theta$, $X$ has a density of the following form: $$p(x\mid \theta )=h(x)\exp (\eta (\theta {)}^{T}T(x)-A(\theta ))$$ Here:

• $\eta (\theta )$ is the natural (canonical) parameter, which is some transform of the naive parameter $\theta$. The natural parameter is the parameter of the distribution that is linear in the sufficient statistics. In fact, it is so much simpler to use the natural parameters, that we drop $\theta$ and just work with $\eta$ hereafter.
• $T(x)$ is the sufficient statistic (may be vector-valued).
• $A(\theta )$ is the log-partition function.
• $h(x)$ is the base measure.

Rewriting in natural parameters, we have $$p(x\mid \eta )=h(x)\exp ({\eta }^{T}T(x)-A(\eta )).$$

If we knew $\eta$ we would now have a distribution for $X$. In practice, we are not sure about $\eta$, so we have a prior distribution for $\eta$. Things will go well for us if we choose this prior to have a particular, and particularly convenient, form.

2 Conjugate prior

The conjugate prior for $\eta$ is designed to ensure that the posterior distribution remains within the same family after a realization from that likelihood we just introduced.

A conjugate prior has to look like this: $$p(\eta \mid \lambda ,\nu )=f(\lambda ,\nu )\exp ({\eta }^T\lambda -\nu A(\eta ))$$ where $\lambda$ means something like ‘accumulated sufficient statistics from prior knowledge’ and $\nu$ the ‘weight’ of the prior or the ‘number of prior observations’. These are effectively hyperparameters encoding how certain we are. This looks like an exponential family distribution, ($f(\lambda ,\nu )$ is the $h$-like base measure), except for this weird scaling of the log-partition function by $\nu$. It is in fact a tempered exponential family.

3 Prior predictive distribution

The prior predictive distribution for a new observation $x$ is obtained by integrating the product of the likelihood and the prior over the natural parameter $\eta$: $$\begin{array}{rl}p(x)& =\int p(x\mid \eta )p(\eta \mid \lambda ,\nu )d\eta \\ & =\int h(x)\exp ({\eta }^{T}T(x)-A(\eta ))f(\lambda ,\nu )\exp ({\eta }^T\lambda -\nu A(\eta ))d\eta \\ & =h(x)f(\lambda ,\nu )\int \exp ({\eta }^T(T(x)+\lambda )-(\nu +1)A(\eta ))d\eta .\end{array}$$

This integral represents the normalization constant of an updated exponential family distribution with parameters updated to ${\lambda }^{\prime }=\lambda +T(x)$ and ${\nu }^{\prime }=\nu +1$. Thus, the integral simplifies to $1/f({\lambda }^{\prime },{\nu }^{\prime })$, where $f$ is the normalizing factor ensuring that the distribution integrates to 1. Hence, the prior predictive distribution becomes: $$p(x)=\frac{h(x)f(\lambda ,\nu )}{f(\lambda +T(x),\nu +1)}$$

The prior predictive distribution $p(x)$ essentially provides the likelihood of observing $x$ before any actual data are observed, based solely on the prior parameters $\lambda$ and $\nu$. This distribution reflects how beliefs encoded in the prior (through $\lambda$ and $\nu$) influence expectations about future data points, integrated over all possible values of the natural parameter $\eta$.

4 Updating the prior

Let us suppose an observation $x\sim X$ arrives. We would like a conjugate posterior update that incorporates the new information. The update to the conjugate prior’s parameters is:

1. ${\lambda }_{\text{posterior}}\leftarrow \lambda +T(x)$, incorporating the new data’s sufficient statistic into the prior accumulated statistics, and
2. ${\nu }_{\text{posterior}}\leftarrow \nu +1$, an increment in the effective number of observations.

The posterior distribution of $\eta$, after observing $x$, in full, is thus: $$p(\eta \mid \lambda +T(x),\nu +1)=f(\lambda +T(x),\nu +1)\exp ({\eta }^T(\lambda +T(x))-(\nu +1)A(\eta ))$$

5 Posterior predictive

As with the prior predictive, we need to integrate out the natural parameters of the likelihood. The posterior predictive distribution for a new observation $x^{\prime }$ given the observed data $x$ is obtained by integrating over the posterior distribution of $\eta$: $$p(x^{\prime }\mid x)=\int p(x^{\prime }\mid \eta )p(\eta \mid x)d\eta$$ Expanding this using the forms we derived above for $p(x^{\prime }\mid \eta )$ and $p(\eta \mid x)$, we find: $$\begin{array}{rl}p(x^{\prime }\mid x)& =\int h(x^{\prime })\exp ({\eta }^{T}T(x^{\prime })-A(\eta ))f(\lambda +T(x),\nu +1)\exp ({\eta }^T(\lambda +T(x))-(\nu +1)A(\eta ))d\eta \end{array}$$ This integral represents the normalizing constant of an updated exponential family distribution with parameters ${\lambda }^{\prime }=\lambda +T(x)+T(x^{\prime })$ and ${\nu }^{\prime }=\nu +2$. Thus, the integral simplifies to $1/f({\lambda }^{\prime },{\nu }^{\prime })$, where $f$ is the normalizing constant. Hence, $$p(x^{\prime }\mid x)=\frac{h(x^{\prime })f(\lambda +T(x),\nu +1)}{f(\lambda +T(x)+T(x^{\prime }),\nu +2)}$$

6 Updates

The conjugate prior for $\eta$ is designed to ensure that the posterior distribution remains within the same family. We partition the parameter of the prior into $\left[\begin{array}{c}\lambda \nu \end{array}\right].$ A conjugate prior has the following form: $$\begin{array}{r}p(\eta \mid \lambda ,\nu )=f(\lambda ,\nu )\exp ({\eta }^T\lambda -\nu A(\eta ))\end{array}$$ where $\lambda$ (something like ‘accumulated sufficient statistics from prior knowledge’) and $\nu$ (the ‘weight’ of the prior or the ‘number of prior observations’) are effectively hyperparameters encoding how certain we are.

The prior predictive distribution for a new observation $x$ is obtained by integrating the product of the likelihood and the prior over the natural parameter $\eta$: $$\begin{array}{rl}p(x)& =\int p(x\mid \eta )p(\eta \mid \lambda ,\nu )d\eta \\ & =\int h(x)\exp ({\eta }^{T}T(x)-A(\eta ))f(\lambda ,\nu )\exp ({\eta }^T\lambda -\nu A(\eta ))d\eta \\ & =h(x)f(\lambda ,\nu )\int \exp ({\eta }^T(T(x)+\lambda )-(\nu +1)A(\eta ))d\eta \\ & =\frac{h(x)f(\lambda ,\nu )}{f(\lambda +T(x),\nu +1)}\end{array}$$ The last line follows from the observation that the integral represents the normalization constant of an exponential family distribution with parameters updated to ${\lambda }^{\prime }=\lambda +T(x)$ and ${\nu }^{\prime }=\nu +1$. Thus, the integral simplifies to $1/f({\lambda }^{\prime },{\nu }^{\prime })$, where $f$ is the normalizing factor.

Let us suppose an observation $x\sim X$ arrives. We would like a conjugate posterior update that incorporates the new information. The update to the conjugate prior’s parameters is:

The posterior distribution of $\eta$, after observing $x$, continues to belong to the same exponential family and is given by: $$p(\eta \mid \lambda +T(x),\nu +1)=f(\lambda +T(x),\nu +1)\exp ({\eta }^T(\lambda +T(x))-(\nu +1)A(\eta ))$$ i.e.~The updated parameters are ${\lambda }_{\text{new}}=\lambda +T(x)$ and ${\nu }_{\text{new}}=\nu +1$.

The posterior predictive distribution for a new observation $x^{\prime }$ given the observed data $x$ is obtained by integrating over the posterior distribution of $\eta$ $$\begin{array}{rl}& p(x^{\prime }\mid x)\\ & =\int p(x^{\prime }\mid \eta )p(\eta \mid x)d\eta \\ & =\int h(x^{\prime })\exp ({\eta }^{T}T(x^{\prime })-A(\eta ))f(\lambda +T(x),\nu +1)\exp ({\eta }^T(\lambda +T(x))-(\nu +1)A(\eta ))d\eta \\ & =h(x^{\prime })f(\lambda +T(x),\nu +1)\int \exp ({\eta }^T(T(x^{\prime })+\lambda +T(x))-(\nu +2)A(\eta ))d\eta \\ & =\frac{h(x^{\prime })f(\lambda +T(x),\nu +1)}{f(\lambda +T(x)+T(x^{\prime }),\nu +2)}\end{array}$$

The last line follows from the observation that the integral term represents the normalizing constant of an updated exponential family distribution with parameters ${\lambda }^{\prime }=\lambda +T(x)+T(x^{\prime })$ and ${\nu }^{\prime }=\nu +2$. Thus, the integral simplifies to $1/f({\lambda }^{\prime },{\nu }^{\prime })$.

7 Mixtures

The under-rated bit of the conjugate prior thing is that, while the priors are themselves, not that flexible, there are some very interesting priors that can be constructed by mixtures of conjugate priors.

TBC. See Dalal and Hall (1983), O’Hagan (2010),…

Farrow’s tutorial introduction:

Consider what happens when we update our beliefs using Bayes’ theorem. Suppose we have a prior density $f_j^{(0)}(\theta )$ for a parameter $\theta$ and suppose the likelihood is $L(\theta )$. Then our posterior density is $$f_j^{(1)}(\theta )=\frac{f_j^{(0)}(\theta )L(\theta )}{C_j}$$ where $$C_j={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{f}_j^{(0)}(\theta )L(\theta )d\theta$$

Now let our prior density for a parameter $\theta$ be $$f^{(0)}(\theta )=\sum _{j=1}^J{k}_j^{(0)}{f}_j^{(0)}(\theta ).$$ Our posterior density is $$\begin{array}{rl}{f}^{(1)}(\theta )& =\frac{\sum _{j=1}^J{k}_j^{(0)}{f}_j^{(0)}(\theta )L(\theta )}{C}\\ & =\frac{\sum _{j=1}^J{k}_j^{(0)}{C}_j{f}_j^{(0)}(\theta )L(\theta )/C_j}{C}\\ & =\frac{\sum _{j=1}^J{k}_j^{(0)}{C}_j{f}_j^{(1)}(\theta )}{C}\end{array}$$

Hence we require $$\frac{\sum _{j=1}^J{k}_j^{(0)}{C}_j}{C}=1$$ so $$C=\sum _{j=1}^J{k}_j^{(0)}{C}_j$$ and the posterior density is $$f^{(1)}(\theta )=\sum _{j=1}^J{k}_j^{(1)}{f}_j^{(1)}(\theta )$$ where $$k_j^{(1)}=\frac{k_j^{(0)}{C}_j}{\sum _{i=1}^J{k}_i^{(0)}{C}_i}.$$

8 In nonparameterics

See (Broderick, Wilson, and Jordan 2018; Orbanz 2011).## Incoming

• Conjugate prior | Definition, explanation and examples
• bayesian - Having a conjugate prior: Deep property or mathematical accident? - Cross Validated
• Conjugate prior distribution for multinomial observations | The Book of Statistical Proofs
• Conjugate Priors - Lizhong Zheng’s Research Page

9 References