Conformal prediction
December 26, 2016 — May 5, 2023
Predicting with competence: the best machine learning idea you never heard of from renowned passive-aggressive grumpy bastard Scott Locklin (Sorry Scott, but you are so reliably objectionable that I am always going to need to put a disclaimer on links to you, why do you refer to female scientists as “this woman”?):
The essential idea is that a “conformity function” exists. Effectively you are constructing a sort of multivariate cumulative distribution function for your machine learning gizmo using the conformity function. Such CDFs exist for classical stuff like ARIMA and linear regression under the correct circumstances; CP brings the idea to machine learning in general, and to models like ARIMA when the standard parametric confidence intervals won’t work.
Cosma Shalizi recommends Samii’s Conformal Inference Tutorial and Lei et al. (2017), because he felt Vovk, Gammerman, and Shafer (2005) was badly written. Maybe Shafer’s tutorial is good? (Shafer and Vovk 2008). Modern takes in Alvarsson et al. (2021);Zeni, Fontana, and Vantini (2020) and A Tutorial on Conformal Prediction plus accompanying video (Angelopoulos and Bates 2022).
Emmanuel Candés’ Neurips keynote on Conformal Prediction in 2022 was good.
Question: how does conformal predication work under dataset shift (Tibshirani et al. 2019; Barber et al. 2023)?
1 Incoming
- time: Lin, Trivedi, and Sun (2022)