Bias reduction

Estimating the bias of an estimator so as to subtract it off again

Trying to reduce bias in point estimators by, e.g. bootstrap. In, e.g. AIC we try to compensate for bias in the model selection. In bias reduction we try to eliminate it from our estimates.

This looks interesting: Kosmidis and Lunardon (2020)

The current work develops a novel method for the reduction of the asymptotic bias of M-estimators from general, unbiased estimating functions. We call the new estimation method reduced-bias M -estimation, or RBM -estimation in short. Like the adjusted scores approach in Firth (1993), the new method relies on additive adjustments to the unbiased estimating functions that are bounded in probability, and results in estimators with bias of lower asymptotic order than the original M -estimators. The key difference is that the empirical adjustments introduced here depend only on the first two derivatives of the contributions to the estimating functions, and they require neither the computation of cumbersome expectations nor the potentially expensive, calculation of M -estimates from simulated samples. Specifically, …, RBM -estimation

  1. applies to models that are at least partially-specified;
  2. uses an analytical approximation to the bias function that relies only on derivatives of the contributions to the estimating functions;
  3. does not depend on the original estimator; and
  4. does not require the computation of any expectations.
Kosmidis’ comparison table

Kosmidis’ comparison table

Cavanaugh, Joseph E. 1997. “Unifying the Derivations for the Akaike and Corrected Akaike Information Criteria.” Statistics & Probability Letters 33 (2): 201–8. https://doi.org/10.1016/S0167-7152(96)00128-9.

Chang, Jinyuan, and Peter Hall. 2015. “Double-Bootstrap Methods That Use a Single Double-Bootstrap Simulation.” Biometrika 102 (1): 203–14. https://doi.org/10.1093/biomet/asu060.

Firth, David. 1993. “Bias Reduction of Maximum Likelihood Estimates.” Biometrika 80 (1): 27–38. https://doi.org/10.1093/biomet/80.1.27.

Hall, Peter. 1994. “Chapter 39 Methodology and Theory for the Bootstrap.” In, edited by BT - Handbook of Econometrics, 4:2341–81. Elsevier. http://www.sciencedirect.com/science/article/pii/S157344120580008X.

Hall, Peter, Joel L. Horowitz, and Bing-Yi Jing. 1995. “On Blocking Rules for the Bootstrap with Dependent Data.” Biometrika 82 (3): 561–74. https://doi.org/10.1093/biomet/82.3.561.

Hesterberg, Tim. 2011. “Bootstrap.” Wiley Interdisciplinary Reviews: Computational Statistics 3 (6): 497–526. https://doi.org/10.1002/wics.182.

Konishi, Sadanori, and Genshiro Kitagawa. 2003. “Asymptotic Theory for Information Criteria in Model Selection—Functional Approach.” Journal of Statistical Planning and Inference, C.R. Rao 80th Birthday Felicitation vol., Part IV, 114 (1–2): 45–61. https://doi.org/10.1016/S0378-3758(02)00462-7.

Kosmidis, Ioannis, and Nicola Lunardon. 2020. “Empirical Bias-Reducing Adjustments to Estimating Functions,” January. http://arxiv.org/abs/2001.03786.

Politis, Dimitris N., and Halbert White. 2004. “Automatic Block-Length Selection for the Dependent Bootstrap.” Econometric Reviews 23 (1): 53–70. https://doi.org/10.1081/ETC-120028836.

Shibata, Ritei. 1997. “Bootstrap Estimate of Kullback-Leibler Information for Model Selection.” Statistica Sinica 7: 375–94.

Stein, Charles M. 1981. “Estimation of the Mean of a Multivariate Normal Distribution.” The Annals of Statistics 9 (6): 1135–51. https://doi.org/10.1214/aos/1176345632.

Varin, Cristiano, Nancy Reid, and David Firth. 2011. “An Overview of Composite Likelihood Methods.” Statistica Sinica 21 (1): 5–42. http://www3.stat.sinica.edu.tw/statistica/J21N1/j21n11/j21n11.html.

Ye, Jianming. 1998. “On Measuring and Correcting the Effects of Data Mining and Model Selection.” Journal of the American Statistical Association 93 (441): 120–31. https://doi.org/10.1080/01621459.1998.10474094.

Zou, Hui, Trevor Hastie, and Robert Tibshirani. 2007. “On the ‘Degrees of Freedom’ of the Lasso.” The Annals of Statistics 35 (5): 2173–92. https://doi.org/10.1214/009053607000000127.