Likelihood free inference



Finding the target without directly inspecting the likelihood of the current guess

A term which seems to have a couple of distinct uses.

Here, I mostly mean this in the sense of trying to approximate intractable likelihoods, possibly by Monte Carlo simulations from a generative model. There seems also to be a school which would like to use this term for methods which make no reference to probability densities whatever. See this grab bag.

Within my use of the term, there is much to find out. Is it mostly modern reframing and rebranding of indirect inference? Is there something more here? Techniques like GANS and reparameterisation methods would fit in that classification.

The archetypal method is presumably Approximate Bayesian Computation, which could make a case for ownership the recent history of the term in the form of simulation based inference. Maybe check the reading list for ABC toolkit ELFI.

Cranmer, Brehmer, and Louppe (2020) attempt to develop a taxonomy here. They certainly make likelihood-free methods sound popular in machine learning for physics.

Cranmer et al’s taxonomy of “simulation based” approaches.

There is another sense of “likelihood free’, where you try to do inference which targets something other than the classic KL divergence as a loss function. This also looks interesting, and essentially orthogonal to the question of approximating the likelihood, e.g. Matsubara et al. (2021). A method that works to approximate an intractable likelihood function might also work to approximate some other intractable loss. If you do lots of predictive point estimation, this second use of the term seems like a waste of a term. In that area, loss functions are pluggable anyway. However, in the case of Bayesian inference, or more generally, distributional inference, complications ensue.

References

Babtie, Ann C., Paul Kirk, and Michael P. H. Stumpf. 2014. “Topological Sensitivity Analysis for Systems Biology.” Proceedings of the National Academy of Sciences 111 (52): 18507–12. https://doi.org/10.1073/pnas.1414026112.
Batz, Philipp, Andreas Ruttor, and Manfred Opper. 2017. “Approximate Bayes Learning of Stochastic Differential Equations.” arXiv:1702.05390 [physics, Stat], February. http://arxiv.org/abs/1702.05390.
Brehmer, Johann, Gilles Louppe, Juan Pavez, and Kyle Cranmer. 2020. “Mining Gold from Implicit Models to Improve Likelihood-Free Inference.” Proceedings of the National Academy of Sciences 117 (10): 5242–49. https://doi.org/10.1073/pnas.1915980117.
Bretó, Carles, Daihai He, Edward L. Ionides, and Aaron A. King. 2009. “Time series analysis via mechanistic models.” The Annals of Applied Statistics 3 (1): 319–48. https://doi.org/10.1214/08-AOAS201.
Castro, Pablo de, and Tommaso Dorigo. 2019. “INFERNO: Inference-Aware Neural Optimisation.” Computer Physics Communications 244 (November): 170–79. https://doi.org/10.1016/j.cpc.2019.06.007.
Cauchemez, Simon, and Neil M. Ferguson. 2008. “Likelihood-Based Estimation of Continuous-Time Epidemic Models from Time-Series Data: Application to Measles Transmission in London.” Journal of The Royal Society Interface 5 (25): 885–97. https://doi.org/10.1098/rsif.2007.1292.
Clark, James S., and Ottar N. Bjørnstad. 2004. “Population Time Series: Process Variability, Observation Errors, Missing Values, Lags, and Hidden States.” Ecology 85 (11): 3140–50. https://doi.org/10.1890/03-0520.
Commandeur, Jacques J. F., Siem Jan Koopman, and Marius Ooms. 2011. “Statistical Software for State Space Methods.” Journal of Statistical Software 41 (1). https://doi.org/10.18637/jss.v041.i01.
Cook, Alex R., Wilfred Otten, Glenn Marion, Gavin J. Gibson, and Christopher A. Gilligan. 2007. “Estimation of Multiple Transmission Rates for Epidemics in Heterogeneous Populations.” Proceedings of the National Academy of Sciences 104 (51): 20392–97. https://doi.org/10.1073/pnas.0706461104.
Cox, D. R., and Christiana Kartsonaki. 2012. “The Fitting of Complex Parametric Models.” Biometrika 99 (3): 741–47. https://doi.org/10.1093/biomet/ass030.
Cranmer, Kyle, Johann Brehmer, and Gilles Louppe. 2020. “The Frontier of Simulation-Based Inference.” Proceedings of the National Academy of Sciences, May. https://doi.org/10.1073/pnas.1912789117.
Creel, Michael, and Dennis Kristensen. 2012. “Estimation of Dynamic Latent Variable Models Using Simulated Non-Parametric Moments.” The Econometrics Journal 15 (3): 490–515. https://doi.org/10.1111/j.1368-423X.2012.00387.x.
———. 2013. “Indirect Likelihood Inference (revised).” UFAE and IAE Working Paper 931.13. Unitat de Fonaments de l’Anàlisi Econòmica (UAB) and Institut d’Anàlisi Econòmica (CSIC). https://ideas.repec.org/p/aub/autbar/931.13.html.
Czellar, Veronika, and Elvezio Ronchetti. 2010. “Accurate and Robust Tests for Indirect Inference.” Biometrika 97 (3): 621–30. https://doi.org/10.1093/biomet/asq040.
Didelot, Xavier, Richard G. Everitt, Adam M. Johansen, and Daniel J. Lawson. 2011. “Likelihood-free estimation of model evidence.” Bayesian Analysis 6 (1): 49–76. https://doi.org/10.1214/11-BA602.
Dridi, Ramdan, Alain Guay, and Eric Renault. 2007. “Indirect Inference and Calibration of Dynamic Stochastic General Equilibrium Models.” Journal of Econometrics, The interface between econometrics and economic theory, 136 (2): 397–430. https://doi.org/10.1016/j.jeconom.2005.11.003.
Drovandi, Christopher, and David T. Frazier. 2021. “A Comparison of Likelihood-Free Methods With and Without Summary Statistics.” arXiv:2103.02407 [stat], March. http://arxiv.org/abs/2103.02407.
Efron, Bradley. 2010. “The Future of Indirect Evidence.” Statistical Science 25 (2): 145–57. https://doi.org/10.1214/09-STS308.
Forneron, Jean-Jacques, and Serena Ng. 2015. “The ABC of Simulation Estimation with Auxiliary Statistics.” arXiv:1501.01265 [stat], January. http://arxiv.org/abs/1501.01265.
Gallant, A. Ronald, David Hsieh, and George Tauchen. 1997. “Estimation of Stochastic Volatility Models with Diagnostics.” Journal of Econometrics 81 (1): 159–92. https://doi.org/10.1016/S0304-4076(97)00039-0.
———. 1997. “Estimation of Stochastic Volatility Models with Diagnostics.” Journal of Econometrics 81 (1): 159–92. https://doi.org/10.1016/S0304-4076(97)00039-0.
Gallant, A. Ronald, and George Tauchen. 1996. “Which Moments to Match?” Econometric Theory 12 (04): 657–81. https://doi.org/10.1017/S0266466600006976.
Genton, Marc G, and Elvezio Ronchetti. 2003. “Robust Indirect Inference.” Journal of the American Statistical Association 98 (461): 67–76. https://doi.org/10.1198/016214503388619102.
Gourieroux, C., A. Monfort, and E. Renault. 1993. “Indirect Inference.” Journal of Applied Econometrics 8 (December): S85–118. http://www.jstor.org/stable/2285076.
Gourieroux, Christian, and Alain Monfort. 1993. “Simulation-Based Inference: A Survey with Special Reference to Panel Data Models.” Journal of Econometrics 59 (1–2): 5–33. https://doi.org/10.1016/0304-4076(93)90037-6.
Grelaud, Aude, Christian P. Robert, Jean-Michel Marin, François Rodolphe, and Jean-François Taly. 2009. “ABC likelihood-free methods for model choice in Gibbs random fields.” Bayesian Analysis 4 (2): 317–35. https://doi.org/10.1214/09-BA412.
Gutmann, Michael U, and Jukka Corander. n.d. “Bayesian Optimization for Likelihood-Free Inference of Simulator-Based Statistical Models,” 47.
He, Daihai, Edward L. Ionides, and Aaron A. King. 2010. “Plug-and-Play Inference for Disease Dynamics: Measles in Large and Small Populations as a Case Study.” Journal of The Royal Society Interface 7 (43): 271–83. https://doi.org/10.1098/rsif.2009.0151.
Hermans, Joeri, Volodimir Begy, and Gilles Louppe. 2020. “Likelihood-Free MCMC with Amortized Approximate Ratio Estimators.” arXiv:1903.04057 [cs, Stat], February. http://arxiv.org/abs/1903.04057.
Ionides, E. L., C. Bretó, and A. A. King. 2006. “Inference for Nonlinear Dynamical Systems.” Proceedings of the National Academy of Sciences 103 (49): 18438–43. https://doi.org/10.1073/pnas.0603181103.
Ionides, Edward L., Anindya Bhadra, Yves Atchadé, and Aaron King. 2011. “Iterated Filtering.” The Annals of Statistics 39 (3): 1776–1802. https://doi.org/10.1214/11-AOS886.
Jiang, Wenxin, and Bruce Turnbull. 2004. “The Indirect Method: Inference Based on Intermediate Statistics—A Synthesis and Examples.” Statistical Science 19 (2): 239–63. https://doi.org/10.1214/088342304000000152.
Kendall, Bruce E., Stephen P. Ellner, Edward McCauley, Simon N. Wood, Cheryl J. Briggs, William W. Murdoch, and Peter Turchin. 2005. “Population Cycles in the Pine Looper Moth: Dynamical Tests of Mechanistic Hypotheses.” Ecological Monographs 75 (2): 259–76. http://www.sysecol2.ethz.ch/Refs/EntClim/K/Ke169.pdf.
Lueckmann, Jan-Matthis, Giacomo Bassetto, Theofanis Karaletsos, and Jakob H. Macke. 2019. “Likelihood-Free Inference with Emulator Networks.” In Symposium on Advances in Approximate Bayesian Inference, 32–53. http://proceedings.mlr.press/v96/lueckmann19a.html.
Matsubara, Takuo, Jeremias Knoblauch, François-Xavier Briol, and Chris J. Oates. 2021. “Robust Generalised Bayesian Inference for Intractable Likelihoods,” April. https://arxiv.org/abs/2104.07359v1.
Miller, Benjamin Kurt, Alex Cole, and Gilles Louppe. n.d. “Simulation-Efficient Marginal Posterior Estimation with Swyft: Stop Wasting Your Precious Time.” In, 9.
Nickl, Richard, and Benedikt M. Pötscher. 2009. “Efficient Simulation-Based Minimum Distance Estimation and Indirect Inference.” Mathematical Methods of Statistics 19, August, 327–64. http://arxiv.org/abs/0908.0433.
Roberts, G. O., and O. Stramer. 2001. “On Inference for Partially Observed Nonlinear Diffusion Models Using the Metropolis–Hastings Algorithm.” Biometrika 88 (3): 603–21. https://doi.org/10.1093/biomet/88.3.603.
Smith, A. A. 1993. “Estimating Nonlinear Time-Series Models Using Simulated Vector Autoregressions.” Journal of Applied Econometrics 8 (S1): S63–84. https://doi.org/10.1002/jae.3950080506.
Smith, A A. 2008. “Indirect Inference.” In The New Palgrave Dictionary of Economics. Palgrave Macmillan. http://www.econ.yale.edu/smith/palgrave7.pdf.
Stoye, Markus, Johann Brehmer, Gilles Louppe, Juan Pavez, and Kyle Cranmer. 2018. “Likelihood-Free Inference with an Improved Cross-Entropy Estimator.” arXiv:1808.00973 [hep-Ph, Physics:physics, Stat], August. http://arxiv.org/abs/1808.00973.
Wood, Simon N. 2010. “Statistical Inference for Noisy Nonlinear Ecological Dynamic Systems.” Nature 466 (7310): 1102–4. https://doi.org/10.1038/nature09319.

No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.