Conformal prediction

References

Predicting with competence: the best machine learning idea you never heard of from renowned passive-aggressive grumpy bastard Scott Locklin (Sorry Scott, but you are so reliably objectionable that I am always going to need to put a disclaimer on links to you, why do you refer to female scientists as “this woman”?):

The essential idea is that a “conformity function” exists. Effectively you are constructing a sort of multivariate cumulative distribution function for your machine learning gizmo using the conformity function. Such CDFs exist for classical stuff like ARIMA and linear regression under the correct circumstances; CP brings the idea to machine learning in general, and to models like ARIMA when the standard parametric confidence intervals won’t work.

Cosma Shalizi recommends Samii’s Conformal Inference Tutorial and Lei et al. (2017), because he felt Vovk, Gammerman, and Shafer (2005) was badly written. Maybe Shafer’s tutorial is good? (Shafer and Vovk 2008). Modern takes in Alvarsson et al. (2021);Zeni, Fontana, and Vantini (2020) and A Tutorial on Conformal Prediction plus accompanying video (Angelopoulos and Bates 2022).

Emmanuel Candés’ Neurips keynote on Conformal Prediction in 2022 was good.

Question: how does conformal predication work under dataset shift (Tibshirani et al. 2019; Barber et al. 2023)?

References

Alvarsson, Jonathan, Staffan Arvidsson McShane, Ulf Norinder, and Ola Spjuth. 2021. “Predicting With Confidence: Using Conformal Prediction in Drug Discovery.” Journal of Pharmaceutical Sciences 110 (1): 42–49.

Angelopoulos, Anastasios N., and Stephen Bates. 2022. “A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification.” arXiv.

———. 2023. “Conformal Prediction: A Gentle Introduction.” Foundations and Trends® in Machine Learning 16 (4): 494–591.

Barber, Rina Foygel, Emmanuel J. Candes, Aaditya Ramdas, and Ryan J. Tibshirani. 2023. “Conformal Prediction Beyond Exchangeability.” arXiv.

Barber, Rina Foygel, Emmanuel J. Candès, Aaditya Ramdas, and Ryan J. Tibshirani. 2021. “Predictive Inference with the Jackknife+.” The Annals of Statistics 49 (1): 486–507.

Bastani, Osbert, Varun Gupta, Christopher Jung, Georgy Noarov, Ramya Ramalingam, and Aaron Roth. 2022. “Practical Adversarial Multivalid Conformal Prediction.” arXiv.

Card, Dallas, Michael Zhang, and Noah A. Smith. 2019. “Deep Weighted Averaging Classifiers.” In Proceedings of the Conference on Fairness, Accountability, and Transparency, 369–78.

Efron, Bradley. 2021. “Resampling Plans and the Estimation of Prediction Error.” Stats 4 (4): 1091–1115.

Fontana, Matteo, Gianluca Zeni, and Simone Vantini. 2023. “Conformal Prediction: A Unified Review of Theory and New Challenges.” Bernoulli 29 (1): 1–23.

Gibbs, Isaac, and Emmanuel Candès. 2022. “Conformal Inference for Online Prediction with Arbitrary Distribution Shifts.” arXiv.

Hu, Yuge, Joseph Musielewicz, Zachary W Ulissi, and Andrew J Medford. 2022. “Robust and Scalable Uncertainty Estimation with Conformal Prediction for Machine-Learned Interatomic Potentials.” Machine Learning: Science and Technology 3 (4): 045028.

Lei, Jing, Max G’Sell, Alessandro Rinaldo, Ryan J. Tibshirani, and Larry Wasserman. 2017. “Distribution-Free Predictive Inference For Regression.” arXiv.

Norinder, Ulf, Lars Carlsson, Scott Boyer, and Martin Eklund. 2014. “Introducing Conformal Prediction in Predictive Modeling. A Transparent and Flexible Alternative to Applicability Domain Determination.” Journal of Chemical Information and Modeling 54 (6): 1596–1603.

Romano, Yaniv, Evan Patterson, and Emmanuel Candes. 2019. “Conformalized Quantile Regression.” In Advances in Neural Information Processing Systems. Vol. 32. Curran Associates, Inc.

Shafer, Glenn, and Vladimir Vovk. 2008. “A Tutorial on Conformal Prediction.” Journal of Machine Learning Research 9 (12): 371–421.

Tibshirani, Ryan J, Rina Foygel Barber, Emmanuel Candes, and Aaditya Ramdas. 2019. “Conformal Prediction Under Covariate Shift.” In Advances in Neural Information Processing Systems. Vol. 32. Curran Associates, Inc.

Vovk, Vladimir, Alex Gammerman, and Glenn Shafer. 2005. Algorithmic Learning in a Random World. Springer Science & Business Media.

Vovk, Vladimir, Ilia Nouretdinov, and Alexander Gammerman. 2009. “On-Line Predictive Linear Regression.” The Annals of Statistics 37 (3): 1566–90.

Zeni, Gianluca, Matteo Fontana, and Simone Vantini. 2020. “Conformal Prediction: A Unified Review of Theory and New Challenges.” arXiv:2005.07972 [Cs, Econ, Stat], May.

References

1 comment

Valery