Uncertainty quantification

Predicting with competence: the best machine learning idea you never heard of:

The essential idea is that a “conformity function” exists. Effectively you are constructing a sort of multivariate cumulative distribution function for your machine learning gizmo using the conformity function. Such CDFs exist for classical stuff like ARIMA and linear regression under the correct circumstances; CP brings the idea to machine learning in general, and to models like ARIMA when the standard parametric confidence intervals won’t work.

Hmm. Introductions in [Vovk, Gammerman, and Shafer (2005);ShaferTutorial2008]. Modern takes in see (“Predicting With Confidence: Using Conformal Prediction in Drug Discovery” 2021; Zeni, Fontana, and Vantini 2020). Question: how well does this work under dataset shift? (Tibshirani et al. 2019).


Barber, Rina Foygel, Emmanuel J. Candès, Aaditya Ramdas, and Ryan J. Tibshirani. 2021. Predictive Inference with the Jackknife+.” The Annals of Statistics 49 (1): 486–507.
Bastani, Osbert, Varun Gupta, Christopher Jung, Georgy Noarov, Ramya Ramalingam, and Aaron Roth. 2022. Practical Adversarial Multivalid Conformal Prediction.” arXiv.
Predicting With Confidence: Using Conformal Prediction in Drug Discovery.” 2021. Journal of Pharmaceutical Sciences 110 (1): 42–49.
Shafer, Glenn, and Vladimir Vovk. 2008. A Tutorial on Conformal Prediction.” Journal of Machine Learning Research 9 (12): 371–421.
Tibshirani, Ryan J, Emmanuel J Candès, Rina Foygel Barber, and Aaditya Ramdas. 2019. “Conformal Prediction Under Covariate Shift,” 11.
Vovk, Vladimir, Alex Gammerman, and Glenn Shafer. 2005. Algorithmic Learning in a Random World. Springer Science & Business Media.
Vovk, Vladimir, Ilia Nouretdinov, and Alexander Gammerman. 2009. On-Line Predictive Linear Regression.” The Annals of Statistics 37 (3): 1566–90.
Zeni, Gianluca, Matteo Fontana, and Simone Vantini. 2020. Conformal Prediction: A Unified Review of Theory and New Challenges.” arXiv:2005.07972 [Cs, Econ, Stat], May.

No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.