Uncertainty quantification

Taxonomy
DUQ networks
Bayes
Physical model setting
Conformal prediction
Chaos expansions
Uncertainty Quantification 360
References

Using machine learning to make predictions, with a measure of the confidence of those predictions.

Taxonomy

Should clarify. TBD. Here is a recent reference on the theme: Kendall and Gal (2017) This extricates aleatoric and epistemic uncertainty. Also to mention, model uncertainty.

DUQ networks

Amersfoort et al. (2020); Kendall and Gal (2017)

Bayes

Bayes methods have some ideas of uncertainty baked in. You can get some way with e.g. , e.g. Gaussian process regression, or probabilistic NNs.

Physical model setting

PEST, PEST++, and pyemu are some integrated systems for uncertainty quantification that use some weird terminology, such a FOSM (First-order-second-moment) models, which at first glance resemble LIME-style local regression model interpretations. The common thread is that these models have complicated physical dynamics which are hard to handle directly, but a surrogate model might be more tractable.

Conformal prediction

Predicting with competence: the best machine learning idea you never heard of:

The essential idea is that a “conformity function” exists. Effectively you are constructing a sort of multivariate cumulative distribution function for your machine learning gizmo using the conformity function. Such CDFs exist for classical stuff like ARIMA and linear regression under the correct circumstances; CP brings the idea to machine learning in general, and to models like ARIMA when the standard parametric confidence intervals won’t work.

Hmm. Perhaps see (“Predicting With Confidence: Using Conformal Prediction in Drug Discovery” 2021; Shafer and Vovk 2008; Zeni, Fontana, and Vantini 2020). Question: how well does this work under dataset shift? (Tibshirani et al. 2019).

Chaos expansions

See chaos expansions.

Uncertainty Quantification 360

IBM’s Uncertainty Quantification 360 toolkit is both a handy software library and a summary of popular generic methods:

Auxiliary Interval Predictor

Use an auxiliary model to improve the calibration of UQ generated by the original model.

Blackbox Metamodel Classification

Extract confidence scores from trained black-box classification models using a meta-model.

Blackbox Metamodel Regression

Extract prediction intervals from trained black-box regression models using a meta-model.

Classification Calibration

Post-hoc calibration of classification models using Isotonic Regression and Platt Scaling.

Heteroscedastic Regression

Train regression models that capture data uncertainty, assuming the targets are noisy and the amount of noise varies between data points.

Homoscedastic Gaussian Process Regression

Train Gaussian Process Regression models with homoscedastic noise that capture data and model uncertainty.

Horseshoe BNN classification

Train Bayesian neural networks classifiers with Gaussian and Horseshoe priors that capture data and model uncertainty.

Horseshoe BNN regression

Train BNNs regression models with Gaussian and Horseshoe priors that capture data and model uncertainty.

Infinitesimal Jackknife

Extract uncertainty from trained models by approximating the effect of training data perturbations on the model’s predictions.

Quantile Regression

Train Quantile Regression models that capture data uncertainty, by learning two separate models for the upper and lower quantile to obtain the prediction intervals.

UCC Recalibration

Recalibrate UQ of a regression model to specified operating point using Uncertainty Characteristics Curve

They provide guidance on method selection in the manual:

[](/images/uq360_taxonomy.png

Source: UQ360

References

Amersfoort, Joost Van, Lewis Smith, Yee Whye Teh, and Yarin Gal. 2020. “Uncertainty Estimation Using a Single Deep Deterministic Neural Network.” In International Conference on Machine Learning, 9690–700. PMLR. http://proceedings.mlr.press/v119/van-amersfoort20a.html.

Bhatt, Umang, Javier Antorán, Yunfeng Zhang, Q. Vera Liao, Prasanna Sattigeri, Riccardo Fogliato, Gabrielle Gauthier Melançon, et al. 2021. “Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty.” arXiv:2011.07586 [cs], May. http://arxiv.org/abs/2011.07586.

Bishop, Christopher. 1994. “Mixture Density Networks.” Microsoft Research, January. https://www.microsoft.com/en-us/research/publication/mixture-density-networks/.

Burrows, Wesley, and John Doherty. 2015. “Efficient Calibration/Uncertainty Analysis Using Paired Complex/Surrogate Models.” Groundwater 53 (4): 531–41. https://doi.org/10.1111/gwat.12257.

Chipman, Hugh A, Edward I George, and Robert E Mcculloch. 2006. “Bayesian Ensemble Learning.” In, 8.

Doherty, John. 2015. Calibration and uncertainty analysis for complex environmental models.

Gal, Yarin, and Zoubin Ghahramani. 2015. “Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning.” In Proceedings of the 33rd International Conference on Machine Learning (ICML-16). http://arxiv.org/abs/1506.02142.

———. 2016. “Dropout as a Bayesian Approximation: Appendix.” arXiv:1506.02157 [stat], May. http://arxiv.org/abs/1506.02157.

Ghosh, Soumya, Q. Vera Liao, Karthikeyan Natesan Ramamurthy, Jiri Navratil, Prasanna Sattigeri, Kush R. Varshney, and Yunfeng Zhang. 2021. “Uncertainty Quantification 360: A Holistic Toolkit for Quantifying and Communicating the Uncertainty of AI.” arXiv:2106.01410 [cs], June. http://arxiv.org/abs/2106.01410.

Gladish, Daniel W., Daniel E. Pagendam, Luk J. M. Peeters, Petra M. Kuhnert, and Jai Vaze. 2018. “Emulation Engines: Choice and Quantification of Uncertainty for Complex Hydrological Models.” Journal of Agricultural, Biological and Environmental Statistics 23 (1): 39–62. https://doi.org/10.1007/s13253-017-0308-3.

Gratiet, Loïc Le, Stefano Marelli, and Bruno Sudret. 2016. “Metamodel-Based Sensitivity Analysis: Polynomial Chaos Expansions and Gaussian Processes.” In Handbook of Uncertainty Quantification, edited by Roger Ghanem, David Higdon, and Houman Owhadi, 1–37. Cham: Springer International Publishing. https://doi.org/10.1007/978-3-319-11259-6_38-1.

Higdon, Dave, James Gattiker, Brian Williams, and Maria Rightley. 2008. “Computer Model Calibration Using High-Dimensional Output.” Journal of the American Statistical Association 103 (482): 570–83. http://www.jstor.org/stable/27640080.

Hooten, Mevin B., William B. Leeds, Jerome Fiechter, and Christopher K. Wikle. 2011. “Assessing First-Order Emulator Inference for Physical Parameters in Nonlinear Mechanistic Models.” Journal of Agricultural, Biological, and Environmental Statistics 16 (4): 475–94. http://www.jstor.org/stable/23238828.

Jarvenpaa, Marko, Aki Vehtari, and Pekka Marttinen. 2020. “Batch Simulations and Uncertainty Quantification in Gaussian Process Surrogate Approximate Bayesian Computation.” In Conference on Uncertainty in Artificial Intelligence, 779–88. PMLR. http://proceedings.mlr.press/v124/jarvenpaa20a.html.

Kasim, M. F., D. Watson-Parris, L. Deaconu, S. Oliver, P. Hatfield, D. H. Froula, G. Gregori, et al. 2020. “Up to Two Billion Times Acceleration of Scientific Simulations with Deep Neural Architecture Search.” arXiv:2001.08055 [physics, Stat], January. http://arxiv.org/abs/2001.08055.

Kendall, Alex, and Yarin Gal. 2017. “What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?” March. https://arxiv.org/abs/1703.04977v2.

Kingma, Diederik P., Tim Salimans, and Max Welling. 2015. “Variational Dropout and the Local Reparameterization Trick.” In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2, 2575–83. NIPS’15. Cambridge, MA, USA: MIT Press. http://arxiv.org/abs/1506.02557.

Kristiadi, Agustinus, Matthias Hein, and Philipp Hennig. 2021. “Learnable Uncertainty Under Laplace Approximations.” In Uncertainty in Artificial Intelligence. http://arxiv.org/abs/2010.02720.

Lakshminarayanan, Balaji, Alexander Pritzel, and Charles Blundell. 2017. “Simple and Scalable Predictive Uncertainty Estimation Using Deep Ensembles.” In Proceedings of the 31st International Conference on Neural Information Processing Systems, 6405–16. NIPS’17. Red Hook, NY, USA: Curran Associates Inc. http://arxiv.org/abs/1612.01474.

Minka, Thomas P. 2001. “Expectation Propagation for Approximate Bayesian Inference.” In Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence, 362–69. UAI’01. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc. https://dslpitt.org/uai/papers/01/p362-minka.pdf.

Mukhoti, Jishnu, Andreas Kirsch, Joost van Amersfoort, Philip H. S. Torr, and Yarin Gal. 2021. “Deterministic Neural Networks with Inductive Biases Capture Epistemic and Aleatoric Uncertainty,” February. https://arxiv.org/abs/2102.11582v2.

O’Hagan, Anthony. 2013. “Polynomial Chaos: A Tutorial and Critique from a Statistician’s Perspective,” 20.

Pestourie, Raphaël, Youssef Mroueh, Thanh V. Nguyen, Payel Das, and Steven G. Johnson. 2020. “Active Learning of Deep Surrogates for PDEs: Application to Metasurface Design.” Npj Computational Materials 6 (1): 1–7. https://doi.org/10.1038/s41524-020-00431-2.

“Predicting With Confidence: Using Conformal Prediction in Drug Discovery.” 2021. Journal of Pharmaceutical Sciences 110 (1): 42–49. https://doi.org/10.1016/j.xphs.2020.09.055.

Sacks, Jerome, Susannah B. Schiller, and William J. Welch. 1989. “Designs for Computer Experiments.” Technometrics 31 (1): 41–47. http://www.jstor.org/stable/1270363.

Sacks, Jerome, William J. Welch, Toby J. Mitchell, and Henry P. Wynn. 1989. “Design and Analysis of Computer Experiments.” Statistical Science 4 (4): 409–23. https://doi.org/10.1214/ss/1177012413.

Shafer, Glenn, and Vladimir Vovk. 2008. “A Tutorial on Conformal Prediction.” Journal of Machine Learning Research 9 (12): 371–421. http://jmlr.org/papers/v9/shafer08a.html.

Siade, Adam J., Mario Putti, and William W. G. Yeh. 2010. “Snapshot selection for groundwater model reduction using proper orthogonal decomposition.” Water Resources Research 46 (8): W08539. https://doi.org/10.1029/2009WR008792.

Smith, Leonard A. 2000. “Disentangling Uncertainty and Error: On the Predictability of Nonlinear Systems.” In Nonlinear Dynamics and Statistics.

Stuart, A. M. 2010. “Inverse Problems: A Bayesian Perspective.” Acta Numerica 19: 451–559. https://doi.org/10.1017/S0962492910000061.

Tibshirani, Ryan J, Emmanuel J Candès, Rina Foygel Barber, and Aaditya Ramdas. 2019. “Conformal Prediction Under Covariate Shift,” 11.

Tonkin, Matthew, and John Doherty. 2009. “Calibration-Constrained Monte Carlo Analysis of Highly Parameterized Models Using Subspace Techniques.” Water Resources Research 45 (12). https://doi.org/10.1029/2007WR006678.

Vovk, Vladimir, Alex Gammerman, and Glenn Shafer. 2005. Algorithmic Learning in a Random World. Springer Science & Business Media.

Welter, David E., Jeremy T. White, Randall J. Hunt, and John E. Doherty. 2015. “Approaches in Highly Parameterized Inversion—PEST++ Version 3, a Parameter ESTimation and Uncertainty Analysis Software Suite Optimized for Large Environmental Models.” USGS Numbered Series 7-C12. Techniques and Methods. Reston, VA: U.S. Geological Survey. https://doi.org/10.3133/tm7C12.

Wen, Yeming, Dustin Tran, and Jimmy Ba. 2020. “BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning.” In ICLR. http://arxiv.org/abs/2002.06715.

White, Jeremy T. 2018. “A Model-Independent Iterative Ensemble Smoother for Efficient History-Matching and Uncertainty Quantification in Very High Dimensions.” Environmental Modelling & Software 109 (November): 191–201. https://doi.org/10.1016/j.envsoft.2018.06.009.

White, Jeremy T., Michael N. Fienen, and John E. Doherty. 2016a. pyEMU: A Python Framework for Environmental Model Uncertainty Analysis Version .01. U.S. Geological Survey. https://doi.org/10.5066/F75D8Q01.

———. 2016b. “A Python Framework for Environmental Model Uncertainty Analysis.” Environmental Modelling & Software 85 (November): 217–28. https://doi.org/10.1016/j.envsoft.2016.08.017.

Zeni, Gianluca, Matteo Fontana, and Simone Vantini. 2020. “Conformal Prediction: A Unified Review of Theory and New Challenges.” arXiv:2005.07972 [cs, Econ, Stat], May. http://arxiv.org/abs/2005.07972.

Zhang, Dongkun, Lu Lu, Ling Guo, and George Em Karniadakis. 2019. “Quantifying Total Uncertainty in Physics-Informed Neural Networks for Solving Forward and Inverse Stochastic Problems.” Journal of Computational Physics 397 (November): 108850. https://doi.org/10.1016/j.jcp.2019.07.048.