Conformal prediction

Predicting with competence: the best machine learning idea you never heard of from renowned passive-aggressive grumpy bastard Scott Locklin (Sorry Scott, but you are so reliably objectionable that I am always going to need to put a disclaimer on links to you, why do you refer to female scientists as “this woman”?):

The essential idea is that a “conformity function” exists. Effectively you are constructing a sort of multivariate cumulative distribution function for your machine learning gizmo using the conformity function. Such CDFs exist for classical stuff like ARIMA and linear regression under the correct circumstances; CP brings the idea to machine learning in general, and to models like ARIMA when the standard parametric confidence intervals won’t work.

Cosma Shalizi recommends Samii’s Conformal Inference Tutorial and Lei et al. (2017), because he felt Vovk, Gammerman, and Shafer (2005) was badly written. Maybe Shafer’s tutorial is good? (Shafer and Vovk 2008). Modern takes in “Predicting With Confidence: Using Conformal Prediction in Drug Discovery” (2021);Zeni, Fontana, and Vantini (2020) and A Tutorial on Conformal Prediction plus accompanying video (Angelopoulos and Bates 2022).

Question: how well does this work under dataset shift? (Tibshirani et al. 2019).


Angelopoulos, Anastasios N., and Stephen Bates. 2022. A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification.” arXiv.
Barber, Rina Foygel, Emmanuel J. Candes, Aaditya Ramdas, and Ryan J. Tibshirani. 2022. Conformal Prediction Beyond Exchangeability.” arXiv.
Barber, Rina Foygel, Emmanuel J. Candès, Aaditya Ramdas, and Ryan J. Tibshirani. 2021. Predictive Inference with the Jackknife+.” The Annals of Statistics 49 (1): 486–507.
Bastani, Osbert, Varun Gupta, Christopher Jung, Georgy Noarov, Ramya Ramalingam, and Aaron Roth. 2022. Practical Adversarial Multivalid Conformal Prediction.” arXiv.
Gibbs, Isaac, and Emmanuel Candès. 2022. Conformal Inference for Online Prediction with Arbitrary Distribution Shifts.” arXiv.
Lei, Jing, Max G’Sell, Alessandro Rinaldo, Ryan J. Tibshirani, and Larry Wasserman. 2017. Distribution-Free Predictive Inference For Regression.” arXiv.
Predicting With Confidence: Using Conformal Prediction in Drug Discovery.” 2021. Journal of Pharmaceutical Sciences 110 (1): 42–49.
Romano, Yaniv, Evan Patterson, and Emmanuel Candes. 2019. Conformalized Quantile Regression.” In Advances in Neural Information Processing Systems. Vol. 32. Curran Associates, Inc.
Shafer, Glenn, and Vladimir Vovk. 2008. A Tutorial on Conformal Prediction.” Journal of Machine Learning Research 9 (12): 371–421.
Tibshirani, Ryan J, Rina Foygel Barber, Emmanuel Candes, and Aaditya Ramdas. 2019. Conformal Prediction Under Covariate Shift.” In Advances in Neural Information Processing Systems. Vol. 32. Curran Associates, Inc.
Vovk, Vladimir, Alex Gammerman, and Glenn Shafer. 2005. Algorithmic Learning in a Random World. Springer Science & Business Media.
Vovk, Vladimir, Ilia Nouretdinov, and Alexander Gammerman. 2009. On-Line Predictive Linear Regression.” The Annals of Statistics 37 (3): 1566–90.
Zeni, Gianluca, Matteo Fontana, and Simone Vantini. 2020. Conformal Prediction: A Unified Review of Theory and New Challenges.” arXiv:2005.07972 [Cs, Econ, Stat], May.

1 comment

GitHub-flavored Markdown & a sane subset of HTML is supported.