ML benchmarks and their pitfalls

On marginal efficiency gain in paperclip manufacture

August 16, 2020 — April 13, 2021

economics
game theory
how do science
incentive mechanisms
institutions
machine learning
neural nets
statistics
Figure 1

Machine learning’s gamified/Goodharted version of the replication crisis is a paper mill, or perhaps paper treadmill. In this system something counts as “results” if it performs on some conventional benchmarks. But how often does that demonstrate real progress and how often is it overfitting to benchmarks?

Oleg Trott on How to sneak up competition leaderboards.

Jörn-Henrik Jacobsen, Robert Geirhos, Claudio Michaelis: Shortcuts: How Neural Networks Love to Cheat.

Sanjeev Arora, Yi Zhang, Rip van Winkle’s Razor, a Simple New Estimate for Adaptive Data Analysis has a minimum description length approach to model meta-overfitting which i will not summarize except to recommend it for being extremely psychedelic.

1 Explicit connection to Goodhart’s law

Goodhart’s law.

Filip Piekniewski on the tendency to select bad target losses for convenience. Measuring Goodhart’s Law at OpenAI.

2 Measuring speed

Lots of algorithms claim to go fast, but that is a complicated claim on modern hardware. Stabilizer attempts to randomise things to give a “fair” comparison.

3 References

Arora, and Zhang. 2021. “Rip van Winkle’s Razor: A Simple Estimate of Overfit to Test Data.” arXiv:2102.13189 [Cs, Stat].
Blum, and Hardt. 2015. “The Ladder: A Reliable Leaderboard for Machine Learning Competitions.” arXiv:1502.04585 [Cs].
Brockman, Cheung, Pettersson, et al. 2016. “OpenAI Gym.” arXiv:1606.01540 [Cs].
Fleming, and Wallace. 1986. “How Not to Lie with Statistics: The Correct Way to Summarize Benchmark Results.” Communications of the ACM.
Geirhos, Jacobsen, Michaelis, et al. 2020. “Shortcut Learning in Deep Neural Networks.” arXiv:2004.07780 [Cs, q-Bio].
Hutson. 2022. “Taught to the Test.” Science.
Hyndman. 2020. “A Brief History of Forecasting Competitions.” International Journal of Forecasting, M4 Competition,.
Lathuilière, Mesejo, Alameda-Pineda, et al. 2020. “A Comprehensive Analysis of Deep Regression.” IEEE Transactions on Pattern Analysis and Machine Intelligence.
Lones. 2021. “How to Avoid Machine Learning Pitfalls: A Guide for Academic Researchers.”
Makridakis, Spiliotis, and Assimakopoulos. 2020. “The M4 Competition: 100,000 Time Series and 61 Forecasting Methods.” International Journal of Forecasting, M4 Competition,.
Musgrave, Belongie, and Lim. 2020. “A Metric Learning Reality Check.” arXiv:2003.08505 [Cs].
Mytkowicz, Diwan, Hauswirth, et al. 2009. “Producing Wrong Data Without Doing Anything Obviously Wrong!” In Proceedings of the 14th International Conference on Architectural Support for Programming Languages and Operating Systems. ASPLOS XIV.
Olson, La Cava, Orzechowski, et al. 2017. “PMLB: A Large Benchmark Suite for Machine Learning Evaluation and Comparison.” BioData Mining.
v. Kistowski, Arnold, Huppler, et al. 2015. “How to Build a Benchmark.” In Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering. ICPE ’15.