Approximate Bayesian Computation

Posterior updates without likelihood

Approximate Bayesian Computation is a terribly underspecified description. There are many ways that inference can be based upon simulations, many types of freedom from likelihood and many ways to approximate Bayesian computation. This page is about the dominant use of that term, which is the use of Simulation-based inference to do Bayes updates where the likelihood is not available but where we can simulate from the generative model.

Obviously there are other ways you can approximate Bayesian computation — see e.g. variational Bayes.

TBD: relationship between this and simulation-based inference in a frequentist setting, often called indirect inference. They look similar but tend not to cite each other. Is this a technical or sociological hurdle?

Miles Cranmer’s Introduction to Simulation-based inference.


One can solve for ABC using Sequential Monte Carlo. TBD.

Bayesian Synthetic Likelihood

TBD. Something about assuming the summary statistic is close to jointly Gaussian.

Neural methods

See the Mackelab sbi page for several:

Goal: Algorithmically identify mechanistic models which are consistent with data.

Each of the methods above needs three inputs: A candidate mechanistic model, prior knowledge or constraints on model parameters, and observational data (or summary statistics thereof).

The methods then proceed by

  1. sampling parameters from the prior followed by simulating synthetic data from these parameters,
  2. learning the (probabilistic) association between data (or data features) and underlying parameters, i.e. to learn statistical inference from simulated data. The way in which this association is learned differs between the above methods, but all use deep neural networks.
  3. This learned neural network is then applied to empirical data to derive the full space of parameters consistent with the data and the prior, i.e. the posterior distribution. High posterior probability is assigned to parameters which are consistent with both the data and the prior, low probability to inconsistent parameters. While SNPE directly learns the posterior distribution, SNLE and SNRE need an extra MCMC sampling step to construct a posterior.
  4. If needed, an initial estimate of the posterior can be used to adaptively generate additional informative simulations.

Code here: mackelab/sbi: Simulation-based inference in PyTorch


Simulation-based Bayes calibration. Is this the same thing?

Martin Modrák, SBC Tutorial


Baydin, Atılım Güneş, Lei Shao, Wahid Bhimji, Lukas Heinrich, Lawrence Meadows, Jialin Liu, Andreas Munk, et al. 2019. “Etalumis: Bringing Probabilistic Programming to Scientific Simulators at Scale.” In arXiv:1907.03382 [cs, Stat].
Corenflos, Adrien, James Thornton, George Deligiannidis, and Arnaud Doucet. 2021. “Differentiable Particle Filtering via Entropy-Regularized Optimal Transport.” arXiv:2102.07850 [cs, Stat], June.
Cranmer, Kyle, Johann Brehmer, and Gilles Louppe. 2020. “The Frontier of Simulation-Based Inference.” Proceedings of the National Academy of Sciences, May.
Cranmer, Kyle, Juan Pavez, and Gilles Louppe. 2015. “Approximating Likelihood Ratios with Calibrated Discriminative Classifiers,” June.
Diggle, Peter J., and Richard J. Gratton. 1984. “Monte Carlo Methods of Inference for Implicit Statistical Models.” Journal of the Royal Statistical Society: Series B (Methodological) 46 (2): 193–212.
Drovandi, Christopher C., Clara Grazian, Kerrie Mengersen, and Christian Robert. 2018. “Approximating the Likelihood in Approximate Bayesian Computation.” arXiv:1803.06645 [stat], March.
Drovandi, Christopher, and David T. Frazier. 2021. “A Comparison of Likelihood-Free Methods With and Without Summary Statistics.” arXiv:2103.02407 [stat], March.
Durkan, Conor, George Papamakarios, and Iain Murray. 2018. “Sequential Neural Methods for Likelihood-Free Inference,” 9.
Fan, Yanan, David J. Nott, and Scott A. Sisson. 2013. “Approximate Bayesian Computation via Regression Density Estimation.” Stat 2 (1): 34–48.
Forneron, Jean-Jacques, and Serena Ng. 2015. “The ABC of Simulation Estimation with Auxiliary Statistics.” arXiv:1501.01265 [stat], January.
Frazier, David T., and Christopher Drovandi. 2021. “Robust Approximate Bayesian Inference With Synthetic Likelihood.” Journal of Computational and Graphical Statistics 0 (0): 1–19.
Frazier, David T., David J. Nott, Christopher Drovandi, and Robert Kohn. 2021. “Bayesian Inference Using Synthetic Likelihood: Asymptotics and Adjustments.” arXiv:1902.04827 [stat], March.
Gelman, Andrew, Aki Vehtari, Daniel Simpson, Charles C. Margossian, Bob Carpenter, Yuling Yao, Lauren Kennedy, Jonah Gabry, Paul-Christian Bürkner, and Martin Modrák. 2020. “Bayesian Workflow.” arXiv:2011.01808 [stat], November.
Gourieroux, Christian, and Alain Monfort. 1993. “Simulation-Based Inference: A Survey with Special Reference to Panel Data Models.” Journal of Econometrics 59 (1–2): 5–33.
Izbicki, Rafael, Ann B. Lee, and Taylor Pospisil. 2019. “ABC–CDE: Toward Approximate Bayesian Computation With Complex High-Dimensional Data and Limited Simulations.” Journal of Computational and Graphical Statistics 28 (3): 481–92.
Le, Tuan Anh, Atılım Güneş Baydin, and Frank Wood. 2017. “Inference Compilation and Universal Probabilistic Programming.” In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS), 54:1338–48. Proceedings of Machine Learning Research. Fort Lauderdale, FL, USA: PMLR.
Lueckmann, Jan-Matthis, Giacomo Bassetto, Theofanis Karaletsos, and Jakob H. Macke. 2019. “Likelihood-Free Inference with Emulator Networks.” In Symposium on Advances in Approximate Bayesian Inference, 32–53.
Meeds, Edward, and Max Welling. 2014. “GPS-ABC: Gaussian Process Surrogate Approximate Bayesian Computation.” In Proceedings of the Thirtieth Conference on Uncertainty in Artificial Intelligence, 593–602. UAI’14. Arlington, Virginia, USA: AUAI Press.
Mohamed, Shakir, and Balaji Lakshminarayanan. 2016. “Learning in Implicit Generative Models,” November.
Neal, Radford. 2008. “Computing Likelihood Functions for High-Energy Physics Experiments When Distributions Are Defined by Simulators with Nuisance Parameters.”
Papamakarios, George, and Iain Murray. 2016. “Fast ε-Free Inference of Simulation Models with Bayesian Conditional Density Estimation.” In Advances in Neural Information Processing Systems 29, edited by D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, 1028–36. Curran Associates, Inc.
Rubin, Donald B. 1984. “Bayesianly Justifiable and Relevant Frequency Calculations for the Applied Statistician.” Annals of Statistics 12 (4): 1151–72.
Säilynoja, Teemu, Paul-Christian Bürkner, and Aki Vehtari. 2021. “Graphical Test for Discrete Uniformity and Its Applications in Goodness of Fit Evaluation and Multiple Sample Comparison.” arXiv:2103.10522 [stat], March.
Schad, Daniel J., Michael Betancourt, and Shravan Vasishth. 2021. “Toward a Principled Bayesian Workflow in Cognitive Science.” Psychological Methods 26 (1): 103–26.
Sisson, S. A., Y. Fan, and Mark M. Tanaka. 2007. “Sequential Monte Carlo Without Likelihoods.” Proceedings of the National Academy of Sciences 104 (6): 1760–65.
Sisson, Scott A., Yanan Fan, and Mark Beaumont. 2018. Handbook of Approximate Bayesian Computation. CRC Press.
Stoye, Markus, Johann Brehmer, Gilles Louppe, Juan Pavez, and Kyle Cranmer. 2018. “Likelihood-Free Inference with an Improved Cross-Entropy Estimator.” arXiv:1808.00973 [hep-Ph, Physics:physics, Stat], August.
Talts, Sean, Michael Betancourt, Daniel Simpson, Aki Vehtari, and Andrew Gelman. 2020. “Validating Bayesian Inference Algorithms with Simulation-Based Calibration.” arXiv:1804.06788 [stat], October.
Tran, Dustin, Rajesh Ranganath, and David Blei. 2017. “Hierarchical Implicit Models and Likelihood-Free Variational Inference.” In Advances in Neural Information Processing Systems 30, edited by I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, 5523–33. Curran Associates, Inc.

No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.