How to go data mining for models without βdredgingβ for models.
(accidentally or otherwise)
If you keep on testing models until you find some that fit
(which you usually will)
how do you know that the fit is in some sense *interesting*?
How sharp will your conclusions be?
How does it work when you are testing against a possibly uncountable continuum of hypotheses?
(One perspective on sparsity penalties is precisely this, I am told.)

Model selection is this writ small - when you are testing how many variables to include in your model.

In modern high-dimensional models, where you have potentially many explanatory variables, the question of how to handle the combinatorial explosion of possible variables to include, this can also be considered a multiple testing problem. We tend to regard this as a smoothing and model selection problem though.

This all gets more complicated when you think about many people testing many hypothesese in many different experiments then you are going to run into many more issues than just these - also publication bias and suchlike.

Suggestive connection:

Moritz Hardt, The machine learning leaderboard problem:

In this post, I will describe a method to climb the public leaderboard without even looking at the data. The algorithm is so simple and natural that an unwitting analyst might just run it. We will see that in Kaggleβs famous Heritage Health Prize competition this might have propelled a participant from rank around 150 into the top 10 on the public leaderboard without making progress on the actual problem. [β¦]

I get super excited. I keep climbing the leaderboard! Who wouldβve thought that this machine learning thing was so easy? So, I go write a blog post on Medium about Big Data and score a job at DeepCompeting.ly, the latest data science startup in the city. Life is pretty sweet. I pick up indoor rock climbing, sign up for wood working classes; I read Proust and books about espresso. Two months later the competition closes and Kaggle releases the final score. What an embarrassment! Wacky boosting did nothing whatsoever on the final test set. I get fired from DeepCompeting.ly days before the buyout. My spouse dumps me. The lease expires. I get evicted from my apartment in the Mission. Inevitably, I hike the Pacific Crest Trail and write a novel about it.

See (Blum and Hardt 2015; Dwork et al. 2015b) for more of that.

## P-value hacking

I Fooled Millions Into Thinking Chocolate Helps Weight Loss. Hereβs How - also the journalism problem, the journals problem, the vacuous-fluff-that-passes-for-public-discussion problemβ¦

## False discovery rate

FDR controlβ¦

- Testing Millions of Hypotheses
is Larry Wassermanβs introduction to controlling the
*false discovery rate*. See also Screening and the false discovery rate. The man can explain clearly.

## Familywise error rate

Ε idΓ‘k correction, Bonferroni correctionβ¦

## Post selection inference

## Misc applied

David Kadavyβs classic, grumpy essay A/A Testing: How I increased conversions 300% by doing absolutely nothing.

http://businessofsoftware.org/2013/06/jason-cohen-ceo-wp-engine-why-data-can-make-you-do-the-wrong-thing/ http://www.evanmiller.org/the-low-base-rate-problem.html

Multiple testing in python: multipy

## References

*The Annals of Statistics*34 (2): 584β653.

*American Journal of Public Health*86 (5): 726β28.

*The Annals of Statistics*13 (4): 1286β316.

*The R Journal*3 (2): 34β39.

*arXiv:0901.3202 [Cs, Stat]*.

*The Annals of Statistics*43 (5): 2055β85.

*Computational Statistics & Data Analysis*36 (3): 279β98.

*arXiv:1511.02513 [Cs]*, November.

*Biometrical Journal*52 (6): 708β21.

*The Annals of Applied Statistics*3 (1): 179β98.

*Journal of the Royal Statistical Society: Series B (Methodological)*57 (1): 289β300.

*Journal of the American Statistical Association*100 (469): 71β81.

*The Annals of Statistics*41 (2): 802β37.

*arXiv:1502.04585 [Cs]*, February.

*Biometrics*53 (2): 603β18.

*arXiv:1503.06426 [Stat]*9 (1): 1449β73.

*The Annals of Statistics*32 (3): 898β927.

*Sociological Methods & Research*33 (2): 261β304.

*Annual Review of Economics*9 (1): 411β39.

*arXiv Preprint arXiv:1610.02351*.

*IEEE Transactions on Information Theory*52 (2): 489β509.

*Journal of Fourier Analysis and Applications*14 (5-6): 877β905.

*Statistics & Probability Letters*33 (2): 201β8.

*Journal of Statistical Planning and Inference*67 (1): 45β65.

*Annual Review of Economics*7 (1): 649β88.

*arXiv:2007.09660 [Math, Stat]*, July.

*Biometrika*96 (3): 529β44.

*Journal of the American Statistical Association*70 (351a): 698β705.

*Journal of the American Statistical Association*80 (390): 411β18.

*Journal of Neuroscience*41 (5): 1019β32.

*Proceedings of the National Academy of Sciences*114 (32): 8592β95.

*BMC Bioinformatics*12 (September): 372.

*arXiv Preprint arXiv:1602.03589*.

*Asymptotic Theory of Statistics and Probability*. Springer Texts in Statistics. New York: Springer New York.

*The Annals of Statistics*36 (2): 665β85.

*arXiv:1408.4026 [Stat]*, August.

*Journal of the American Statistical Association*90 (432): 1200β1224.

*Journal of the Royal Statistical Society. Series B (Methodological)*57 (2): 301β69.

*Science*349 (6248): 636β38.

*Proceedings of the Forty-Seventh Annual ACM on Symposium on Theory of Computing - STOC β15*, 117β26. Portland, Oregon, USA: ACM Press.

*International Journal of Environmental Research and Public Health*5 (5): 394β98.

*The Annals of Statistics*7 (1): 1β26.

*Journal of the American Statistical Association*81 (394): 461β70.

*Journal of the American Statistical Association*99 (467): 619β32.

*Metron - International Journal of Statistics*LXV (1): 3β21.

*The Annals of Applied Statistics*2 (1): 197β223.

*Journal of the American Statistical Association*104 (487): 1015β28.

*Statistical Science*25 (2): 145β57.

*Large-Scale Inference: Empirical Bayes Methods for Estimation, Testing, and Prediction*. Reprint edition. Cambridge: Cambridge University Press.

*arXiv:1507.05315 [Math, Stat]*, July.

*Journal of the American Statistical Association*96 (456): 1348β60.

*Statistica Sinica*20 (1): 101β48.

*arXiv:1407.4240 [q-Bio, Stat]*, July.

*Journal of Statistical Software*33 (1): 1β22.

*Advances in Neural Information Processing Systems 27*, edited by Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, 1817β25. Curran Associates, Inc.

*The Annals of Statistics*42 (3): 1166β1202.

*arXiv:1107.0189 [Stat]*, July.

*American Scientist*102 (6): 460.

*The Annals of Statistics*36 (2): 875β905.

*Journal of Econometrics*119 (1): 199β219.

*Proceedings of the 2014 IEEE 55th Annual Symposium on Foundations of Computer Science*, 454β63. FOCS β14. Washington, DC, USA: IEEE Computer Society.

*Statistics Surveys*2: 61β93.

*The Annals of Statistics*24 (4): 1619β47.

*International Statistical Review / Revue Internationale de Statistique*60 (3): 355β87.

*Survival Analysis: State of the Art*, edited by John P. Klein and Prem K. Goel, 211β36. Nato Science 211. Springer Netherlands.

*Biometrika*76 (2): 297β307.

*Journal of Econometrics*58 (1β2): 71β120.

*PLoS Medicine*2 (8): β124.

*Statistical Science*3 (1): 109β17.

*arXiv:1507.02061 [Math, Stat]*, July.

*Biometrika*102 (2): 479β85.

*Biometrika*101 (4): 771β84.

*Biometrika*83 (4): 875β90.

*Neural Computation*28 (1): 45β70.

*Genome Biology*20 (1): 118.

*Journal of Applied Probability*23 (4): 1025β30.

*Physical Review X*5 (1): 011007.

*Electronic Journal of Statistics*9: 643β78.

*Molecular Psychiatry*17 (1): 108β14.

*arXiv:1311.6238 [Math, Stat]*, November.

*The Annals of Statistics*36 (1): 261β86.

*The Annals of Statistics*42 (2): 413β68.

*Scandinavian Journal of Statistics*33 (2): 227β37.

*Computational Statistics & Data Analysis*52 (1): 374β93.

*Journal of the Royal Statistical Society: Series B (Statistical Methodology)*, November, n/aβ.

*Biometrika*92 (4): 893β907.

*The Annals of Statistics*34 (3): 1436β62.

*Journal of the Royal Statistical Society: Series B (Statistical Methodology)*72 (4): 417β73.

*Journal of the American Statistical Association*104 (488): 1671β81.

*The Annals of Statistics*34 (1): 373β93.

*The Annals of Statistics*37 (1): 246β70.

*Journal of Machine Learning Research*15: 2055β60.

*The Annals of Statistics*41 (6): 2852β76.

*Nature Biotechnology*27 (12): 1135β37.

*International Journal of Data Science and Analytics*3 (2): 121β29.

*The Annals of Statistics*35 (3): 1012β30.

*Epidemiology (Cambridge, Mass.)*1 (1): 43β46.

*Proceedings of the National Academy of Sciences*112 (47): 14569β74.

*arXiv:1411.1437 [Math, Stat]*, November.

*Journal of the Royal Statistical Society. Series B (Methodological)*39 (1): 44β47.

*Journal of the Royal Statistical Society: Series B (Statistical Methodology)*64 (3): 479β98.

*arXiv:1511.01957 [Cs, Math, Stat]*, November.

*arXiv:1308.5623 [Stat]*, August.

*arXiv:1411.6144 [Stat]*, November.

*Journal of Machine Learning Research*, 684β92.

*arXiv:1401.3889 [Stat]*, January.

*arXiv:1408.5801 [Stat]*, August.

*arXiv:1506.06266 [Math, Stat]*, June.

*Annals of Statistics*37 (5A): 2178β2201.

*Journal of the Royal Statistical Society: Series B (Statistical Methodology)*76 (1): 217β42.

*The Annals of Statistics*35 (5): 2173β92.

## No comments yet. Why not leave one?