Bootstrap

Shuffling reality to produce your data



Resampling your own data to estimate how good your point-estimator is, and to reduce its bias. In general an intuitive technique. However, gets tricky for e.g. dependent data. For a handy crib sheet for bootstrap failure modes, see Thomas Lumley, When the bootstrap doesn’t work.

In the classical mode, this is a frequentist technique without an immediate Bayesian interpretation.

Commonly credited as being invented by B. Efron (1979) and theoretically justified by Gine and Zinn (1990).

Bootstrap bias correction

As opp variance estimation. NBD; Bootstrap is notionally telling you the sampling distribution. πŸ—

Bootstrap for dependent data

e.g., as presaged, time series. Parametric bootstrap would be the logical default choice, right? When does that work?

Causal bootstrap

Now a thing! (Imbens and Menzel 2021)

As a Bayesian method

There is absolutely a Bayesian bootstrap if you think hard enough about it, it turns out. Several, really. Rubin (1981) derived a Bayesian version. See Lyddon, Holmes, and Walker (2019) for a modern update, and Rasmus BΓ₯Γ₯th for a diagrammed explanation of the points of contact with frequentist bootstrap and some other things.

Pedagogic

References

Alfaro, M. E. 2003. β€œBayes or Bootstrap? A Simulation Study Comparing the Performance of Bayesian Markov Chain Monte Carlo Sampling and Bootstrapping in Assessing Phylogenetic Confidence.” Molecular Biology and Evolution 20 (2): 255–66.
Bach, Francis. 2009. β€œModel-Consistent Sparse Estimation Through the Bootstrap.” arXiv:0901.3202 [Cs, Stat].
Barber, Rina Foygel, Emmanuel J. CandΓ¨s, Aaditya Ramdas, and Ryan J. Tibshirani. 2021. β€œPredictive Inference with the Jackknife+.” The Annals of Statistics 49 (1): 486–507.
Biewen, Martin. 2002. β€œBootstrap Inference for Inequality, Mobility and Poverty Measurement.” Journal of Econometrics 108 (2): 317–42.
BΓΌhlmann, Peter. 2002. β€œBootstraps for Time Series.” Statistical Science 17 (1): 52–72.
BΓΌhlmann, Peter, and Hans R KΓΌnsch. 1999. β€œBlock Length Selection in the Bootstrap for Time Series.” Computational Statistics & Data Analysis 31 (3): 295–310.
Burnham, Kenneth P., and David R. Anderson. 2004. β€œMultimodel Inference Understanding AIC and BIC in Model Selection.” Sociological Methods & Research 33 (2): 261–304.
Chang, Jinyuan, and Peter Hall. 2015. β€œDouble-Bootstrap Methods That Use a Single Double-Bootstrap Simulation.” Biometrika 102 (1): 203–14.
Chen, Kani, and Shaw-Hwa Lo. 1997. β€œOn a Mapping Approach to Investigating the Bootstrap Accuracy.” Probability Theory and Related Fields 107 (2): 197–217.
Cogneau, Philippe, and Valeri Zakamouline. 2010. β€œBootstrap Methods for Finance: Review and Analysis.” Working Paper, University of Agder.
Dahlhaus, Rainer. 2011. β€œDiscussion: Bootstrap Methods for Dependent Data: A Review.” Journal of the Korean Statistical Society 40 (4): 379–81.
DiCiccio, Thomas J., and Bradley Efron. 1996. β€œBootstrap Confidence Intervals.” Statistical Science 11 (3): 189–212.
Efron, B. 1979. β€œBootstrap Methods: Another Look at the Jackknife.” The Annals of Statistics 7 (1): 1–26.
Efron, Bradley. 1981. β€œNonparametric Estimates of Standard Error: The Jackknife, the Bootstrap and Other Methods.” Biometrika 68 (3): 589–99.
β€”β€”β€”. 2012. β€œBayesian Inference and the Parametric Bootstrap.” The Annals of Applied Statistics 6 (4): 1971–97.
β€”β€”β€”. 2021. β€œResampling Plans and the Estimation of Prediction Error.” Stats 4 (4): 1091–1115.
Fong, Edwin, and Chris Holmes. 2019. β€œOn the Marginal Likelihood and Cross-Validation.” arXiv:1905.08737 [Stat], May.
Galvani, Marta, Chiara Bardelli, Silvia Figini, and Pietro Muliere. 2021. β€œA Bayesian Nonparametric Learning Approach to Ensemble Models Using the Proper Bayesian Bootstrap.” Algorithms 14 (1): 11.
Gine, Evarist, and Joel Zinn. 1990. β€œBootstrapping General Empirical Measures.” Annals of Probability 18 (2): 851–69.
Giordano, Ryan, Michael I. Jordan, and Tamara Broderick. 2019. β€œA Higher-Order Swiss Army Infinitesimal Jackknife.” arXiv:1907.12116 [Cs, Math, Stat], July.
GonΓ§alves, SΓ­lvia, and Dimitris Politis. 2011. β€œDiscussion: Bootstrap Methods for Dependent Data: A Review.” Journal of the Korean Statistical Society 40 (4): 383–86.
GonΓ§alves, SΓ­lvia, and Halbert White. 2004. β€œMaximum Likelihood and the Bootstrap for Nonlinear Dynamic Models.” Journal of Econometrics 119 (1): 199–219.
Good, Phillip I., and Philip Good. 1999. Resampling Methods: A Practical Guide to Data Analysis. BirkhΓ€user Basel.
GΓΆtze, F., and H. R. KΓΌnsch. 1996. β€œSecond-Order Correctness of the Blockwise Bootstrap for Stationary Observations.” The Annals of Statistics 24 (5): 1914–33.
Green, Alden, and Cosma Rohilla Shalizi. 2017. β€œBootstrapping Exchangeable Random Graphs.” arXiv:1711.00813 [Stat], November.
Hall, Peter. 1992. β€œOn Bootstrap Confidence Intervals in Nonparametric Regression.” The Annals of Statistics 20 (2): 695–711.
β€”β€”β€”. 1994. β€œMethodology and Theory for the Bootstrap.” In Handbook of Econometrics, 4:2341–81. Elsevier.
Hall, Peter, Joel L. Horowitz, and Bing-Yi Jing. 1995. β€œOn Blocking Rules for the Bootstrap with Dependent Data.” Biometrika 82 (3): 561–74.
HΓ€rdle, Wolfgang, Joel Horowitz, and Jens-Peter Kreiss. 2003. β€œBootstrap Methods for Time Series.” International Statistical Review 71 (2): 435–59.
Hesterberg, Tim. 2011. β€œBootstrap.” Wiley Interdisciplinary Reviews: Computational Statistics 3 (6): 497–526.
Hinkley, David V. 1997. Bootstrap Methods and Their Application. Cambridge ; New York, NY, USA: Cambridge University Press.
Imbens, Guido, and Konrad Menzel. 2021. β€œA Causal Bootstrap.” The Annals of Statistics 49 (3): 1460–88.
KΓΌnsch, Hans Rudolf. 1989. β€œThe Jackknife and the Bootstrap for General Stationary Observations.” The Annals of Statistics 17 (3): 1217–41.
Lahiri, S N. 1993. β€œOn the Moving Block Bootstrap Under Long Range Dependence.” Statistics & Probability Letters 18 (5): 405–13.
β€”β€”β€”. 2001. β€œEffects of Block Lengths on the Validity of Block Resampling Methods.” Probability Theory and Related Fields 121: 73–97.
β€”β€”β€”. 2003. Resampling Methods for Dependent Data. New York: Springer.
Lee, Stephen M. S., and G. Alastair Young. 1996. β€œ[Bootstrap Confidence Intervals]: Comment.” Statistical Science 11 (3): 221–23.
Lyddon, S. P., C. C. Holmes, and S. G. Walker. 2019. β€œGeneral Bayesian Updating and the Loss-Likelihood Bootstrap.” Biometrika 106 (2): 465–78.
Papadopoulos, G., P.J. Edwards, and A.F. Murray. 2001. β€œConfidence Estimation Methods for Neural Networks: A Practical Comparison.” IEEE Transactions on Neural Networks 12 (6): 1278–87.
Paparoditis, Efstathios, and Theofanis Sapatinas. 2014. β€œBootstrap-Based Testing for Functional Data.” arXiv:1409.4317 [Math, Stat], September.
Politis, Dimitris N. 2003. β€œThe Impact of Bootstrap Methods on Time Series Analysis.” Statistical Science 18 (2): 219–30.
Politis, Dimitris N., and Joseph P. Romano. 1994. β€œThe Stationary Bootstrap.” Journal of the American Statistical Association 89 (428): 1303–13.
Politis, Dimitris N., and Halbert White. 2004. β€œAutomatic Block-Length Selection for the Dependent Bootstrap.” Econometric Reviews 23 (1): 53–70.
Rodriguez, Alejandro, and Esther Ruiz. 2009. β€œBootstrap Prediction Intervals in State–Space Models.” Journal of Time Series Analysis 30 (2): 167–78.
Rubin, Donald B. 1981. β€œThe Bayesian Bootstrap.” Annals of Statistics 9 (1): 130–34.
Sanson, Mevagh, Deryn Strange, and Maryanne Garry. 2019. β€œTrigger Warnings Are Trivially Helpful at Reducing Negative Affect, Intrusive Thoughts, and Avoidance.” Clinical Psychological Science 7 (4): 778–93.
Shalizi, Cosma Rohilla. 2010. β€œThe Bootstrap.” American Scientist 98 (3): 186.
Shao, Jun. 1996. β€œBootstrap Model Selection.” Journal of the American Statistical Association 91 (434): 655–65.
Shibata, Ritei. 1997. β€œBootstrap Estimate of Kullback-Leibler Information for Model Selection.” Statistica Sinica 7: 375–94.
Stone, M. 1977. β€œAn Asymptotic Equivalence of Choice of Model by Cross-Validation and Akaike’s Criterion.” Journal of the Royal Statistical Society. Series B (Methodological) 39 (1): 44–47.
Tibshirani, Ryan J., Alessandro Rinaldo, Robert Tibshirani, and Larry Wasserman. 2015. β€œUniform Asymptotic Inference and the Bootstrap After Model Selection.” arXiv:1506.06266 [Math, Stat], June.
Vogel, Richard M., and Amy L. Shallcross. 1996. β€œThe Moving Blocks Bootstrap Versus Parametric Time Series Models.” Water Resources Research 32 (6): 1875–82.
Yatchew, A, and W Hardle. 2006. β€œNonparametric State Price Density Estimation Using Constrained Least Squares and the Bootstrap.” Journal of Econometrics 133 (2): 579–99.

No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.