# Uncertainty quantification

December 26, 2016 — June 6, 2024

Using machine learning to make predictions, with a measure of the confidence of those predictions.

Closely related, perhaps identical: sensitivity analysis.

## 1 Taxonomy

Should clarify. TBD. Here is a recent reference on the theme: Kendall and Gal (2017) This extricates aleatoric and epistemic uncertainty. Also to mention, model uncertainty.

## 2 DUQ networks

## 3 Bayes

Bayes methods have some ideas of uncertainty baked in. You can get some way with e.g., e.g. Gaussian process regression, or probabilistic NNs.

## 4 Physical model calibration

PEST, PEST++, and pyemu are some integrated systems for uncertainty quantification that use some weird terminology, such a FOSM (First-order-second-moment) models. I think these might be best considered as inverse problem solvers, and the uncertainty quantification is a side effect of the inversion.

## 5 Conformal prediction

See conformal prediction.

## 6 Uncertainty Quantification 360

IBM’s Uncertainty Quantification 360 toolkit a summary of popular generic methods:

- Auxiliary Interval Predictor
Use an auxiliary model to improve the calibration of UQ generated by the original model.

- Blackbox Metamodel Classification
Extract confidence scores from trained black-box classification models using a meta-model.

- Blackbox Metamodel Regression
Extract prediction intervals from trained black-box regression models using a meta-model.

- Classification Calibration
Post-hoc calibration of classification models using Isotonic Regression and Platt Scaling.

- Heteroscedastic Regression

- Homoscedastic Gaussian Process Regression

- Horseshoe BNN classification

- Horseshoe BNN regression

- Infinitesimal Jackknife

- Quantile Regression

- UCC Recalibration

They provide guidance on method selection in the manual:

## 7 References

*Journal of Pharmaceutical Sciences*.

*arXiv:2011.07586 [Cs]*.

*Microsoft Research*.

*Groundwater*.

*Computational Geosciences*.

*arXiv:2106.14806 [Cs, Stat]*.

*Calibration and uncertainty analysis for complex environmental models*.

*Proceedings of the 33rd International Conference on Machine Learning (ICML-16)*.

*arXiv:1506.02157 [Stat]*.

*arXiv:2106.01410 [Cs]*.

*Journal of Agricultural, Biological and Environmental Statistics*.

*Handbook of Uncertainty Quantification*.

*Journal of the American Statistical Association*.

*Journal of Agricultural, Biological, and Environmental Statistics*.

*Conference on Uncertainty in Artificial Intelligence*.

*arXiv:2001.08055 [Physics, Stat]*.

*Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2*. NIPS’15.

*Uncertainty in Artificial Intelligence*.

*Proceedings of the 31st International Conference on Neural Information Processing Systems*. NIPS’17.

*Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence*. UAI’01.

*Npj Computational Materials*.

*Technometrics*.

*Statistical Science*.

*Journal of Machine Learning Research*.

*Water Resources Research*.

*Nonlinear Dynamics and Statistics*.

*Acta Numerica*.

*An Introduction to Data Analysis and Uncertainty Quantification for Inverse Problems*. Mathematics in Industry.

*Advances in Neural Information Processing Systems*.

*Water Resources Research*.

*International Conference on Machine Learning*.

*Algorithmic Learning in a Random World*.

*ICLR*.

*Environmental Modelling & Software*.

*Environmental Modelling & Software*.

*arXiv:2005.07972 [Cs, Econ, Stat]*.

*Journal of Computational Physics*.