Expectation maximisation

August 17, 2014 — April 17, 2016

A particular optimisation method for statistics for that gets you a maximum likelihood estimate despite various annoyances such as missing data.

Vague description of the algorithm:

We have an experimental process that generates a random vector \(B\cup Y\) according to parameter \(\theta\). We wish to estimate the parameter of interest \(\theta\) by maximum likelihood. However, we only observe i.i.d. samples \(b_i\) drawn from \(B\). The likelihood function of the incomplete data \(L(\theta, b)\) is tedious or intractable to maximise. But the “complete” joint likelihood of both the observed and unobserved components, \(L(\theta, \{b_i\}, y)\), is easier to maximise. Then we are potentially in a situation where expectation maximisation can help.

Call \(\theta^{(k)}\) the estimate of \(\theta\) at step \(k\). Write \(\ell(\theta, \{b_i\}, y)\equiv\log L(\theta, \{b_i\}, y)\) because we work in log likelihoods why not.

The following form of the algorithm works when the log-likelihood \(\ell(\theta, b, y)\) is linear in \(b\). (Which is equivalent to it being in a exponential family I believe, but should check.)

At time \(k=0\) we start with an estimate of \(\theta^{(0)}\) chosen arbitrarily or by our favourite approximate method.

We attempt to improve our estimate of the parameter of interest by the following iterative algorithm:

  1. “Expectation”: Under the completed data model with joint distribution \(F(b,y,\theta^{(k)})\) we estimate \(y\) as

    \[ y^{(k)}=E_{\theta^{(k)}}[Y|b] \]

  2. “Maximisation”: Solve a (hopefully easier) maximisation problem:

    \[ \theta^{(k+1)}=\operatorname{arg max}_\theta \ell(\theta, b, y^{(k)}) \]

In the case that this log likelihood is not linear in \(b\), you are supposed to instead take

\[ \theta^{(k+1)}=\operatorname{arg max}_\theta E_{\theta^{(k)}}[\ell(\theta, b, Y)|b] \]

In practice this nicety is often ignored.

Even if you do the right thing, EM may not converge especially well, or to the global maximum, but it can be easy and robust to get started with, and at least it doesn’t make things worse.

Literature note — apparently the proofs in Dempster, Laird, and Rubin (1977) are dicey; See Wu (1983) for an improved (i.e. correct) versions or Wainwright and Jordan (2008) for an interpretation in terms of graphical models wherein the algorithm is a form of message passing.

My goal is to fill in the details of one key step in the derivation of the EM algorithm in a way that makes it inevitable rather than arbitrary.

1 References

Bilmes. 1998. A Gentle Tutorial of the EM Algorithm and Its Application to Parameter Estimation for Gaussian Mixture and Hidden Markov Models.” International Computer Science Institute.
Celeux, Chauveau, and Diebolt. 1995. On Stochastic Versions of the EM Algorithm.” Report.
Celeux, Chretien, Forbes, et al. 2001. A Component-Wise EM Algorithm for Mixtures.” Journal of Computational and Graphical Statistics.
Celeux, Forbes, and Peyrard. 2003. EM Procedures Using Mean Field-Like Approximations for Markov Model-Based Image Segmentation.” Pattern Recognition.
Delyon, Lavielle, and Moulines. 1999. Convergence of a Stochastic Approximation Version of the EM Algorithm.” The Annals of Statistics.
Dempster, Laird, and Rubin. 1977. Maximum Likelihood from Incomplete Data Via the EM Algorithm.” Journal of the Royal Statistical Society: Series B (Methodological).
Kuhn, and Lavielle. 2004. Coupling a Stochastic Approximation Version of EM with an MCMC Procedure.” ESAIM: Probability and Statistics.
Lee, and Scott. 2012. EM Algorithms for Multivariate Gaussian Mixture Models with Truncated and Censored Data.” Computational Statistics & Data Analysis.
McLachlan, Geoffrey J, and Krishnan. 2008. The EM algorithm and extensions.
McLachlan, Geoffrey J., Krishnan, and Ng. 2004. The EM Algorithm.” 2004,24.
Miyahara, Tsumura, and Sughiyama. 2016. Relaxation of the EM Algorithm via Quantum Annealing for Gaussian Mixture Models.” In arXiv:1701.03268 [Cond-Mat, Physics:quant-Ph, Stat].
Navidi. 1997. A Graphical Illustration of the EM Algorithm.” The American Statistician.
Neal, and Hinton. 1998. A View of the EM Algorithm That Justifies Incremental, Sparse, and Other Variants.” In Learning in Graphical Models. NATO ASI Series 89.
Prescher. 2004. A Tutorial on the Expectation-Maximization Algorithm Including Maximum-Likelihood Estimation and EM Training of Probabilistic Context-Free Grammars.” arXiv:cs/0412015.
Roche. 2011. EM Algorithm and Variants: An Informal Tutorial.” arXiv:1105.1476 [Stat].
Wainwright, and Jordan. 2008. Graphical Models, Exponential Families, and Variational Inference. Foundations and Trends® in Machine Learning.
Wei, and Tanner. 1990. A Monte Carlo Implementation of the EM Algorithm and the Poor Man’s Data Augmentation Algorithms.” Journal of the American Statistical Association.
Wu. 1983. On the Convergence Properties of the EM Algorithm.” The Annals of Statistics.