Uncertainty quantification



Predicting with competence: the best machine learning idea you never heard of:

The essential idea is that a “conformity function” exists. Effectively you are constructing a sort of multivariate cumulative distribution function for your machine learning gizmo using the conformity function. Such CDFs exist for classical stuff like ARIMA and linear regression under the correct circumstances; CP brings the idea to machine learning in general, and to models like ARIMA when the standard parametric confidence intervals won’t work.

Hmm. Introductions in [Vovk, Gammerman, and Shafer (2005);ShaferTutorial2008]. Modern takes in see (“Predicting With Confidence: Using Conformal Prediction in Drug Discovery” 2021; Zeni, Fontana, and Vantini 2020). Question: how well does this work under dataset shift? (Tibshirani et al. 2019).

References

Barber, Rina Foygel, Emmanuel J. Candès, Aaditya Ramdas, and Ryan J. Tibshirani. 2021. Predictive Inference with the Jackknife+.” The Annals of Statistics 49 (1): 486–507.
Bastani, Osbert, Varun Gupta, Christopher Jung, Georgy Noarov, Ramya Ramalingam, and Aaron Roth. 2022. Practical Adversarial Multivalid Conformal Prediction.” arXiv.
Predicting With Confidence: Using Conformal Prediction in Drug Discovery.” 2021. Journal of Pharmaceutical Sciences 110 (1): 42–49.
Shafer, Glenn, and Vladimir Vovk. 2008. A Tutorial on Conformal Prediction.” Journal of Machine Learning Research 9 (12): 371–421.
Tibshirani, Ryan J, Emmanuel J Candès, Rina Foygel Barber, and Aaditya Ramdas. 2019. “Conformal Prediction Under Covariate Shift,” 11.
Vovk, Vladimir, Alex Gammerman, and Glenn Shafer. 2005. Algorithmic Learning in a Random World. Springer Science & Business Media.
Vovk, Vladimir, Ilia Nouretdinov, and Alexander Gammerman. 2009. On-Line Predictive Linear Regression.” The Annals of Statistics 37 (3): 1566–90.
Zeni, Gianluca, Matteo Fontana, and Simone Vantini. 2020. Conformal Prediction: A Unified Review of Theory and New Challenges.” arXiv:2005.07972 [Cs, Econ, Stat], May.

No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.