High dimensional statistics

Placeholder to think about the many weird problems arising in very high dimensional statistical inference. There are many approaches to this problem: throwing out dimensions/predictors as in model selection, considering low dimensional projections, viewing objects with matrix structure for concentration or factorisation, or tensor structure even.

Soap bubbles

High dimensional distributions are extremely odd, and concentrate in weird ways. For example, for some natural definitions of typical, typical items are not average items in high See Sander Dielemann’s musings on typicality for an introduction to this plus some motivating examples.

For another example, consider this summary result of Vershynin (2015):

Let \(K\) be an isotropic convex body (e.g. an \(L_2\) ball) in \(\mathbb{R}^{n},\) and let \(X\) be a random vector uniformly distributed in \(K\), with \(\mathbb{E}X=0\) and \(\mathbb{E}XX^{\top}=I_n.\) Then the following is true for some positive constants \(C,c\):

  1. (Concentration of volume) For every \(t \geq 1\), one has \[ \mathbb{P}\left\{\|X\|_{2}>t \sqrt{n}\right\} \leq \exp (-c t \sqrt{n}) \]
  2. (Thin shell) For every \(\varepsilon \in(0,1),\) one has \[ \mathbb{P}\left\{\left|\|X\|_{2}-\sqrt{n}\right|>\varepsilon \sqrt{n}\right\} \leq C \exp \left(-c \varepsilon^{3} n^{1 / 2}\right) \]

That is, even with the mass uniformly distributed over space, as the dimension grows, it all ends up in a thin shell, because volume grows exponentially in dimension. This is popularly known as a soap bubble phenomenon. This is one of the phenomena that leads to interesting behaviour in low dimensional projection. The more formal name is the Gaussian Annulus Theorem. Turning it around, for a d-dimensional spherical Gaussian with unit variance in each direction, for any \(\beta \leq \sqrt{d}\), all but at most \(3 e^{-c \beta^{2}}\) of the probability mass lies within the annulus \(\sqrt{d}-\beta \leq|\mathbf{x}| \leq \sqrt{d}+\beta,\) where \(c\) is a fixed positive constant.

Empirical processes in high dimensions

Combining empirical process theory with high dimensional statistics gets us to some extremely interesting statistics. See, e.g. van de Geer (2014b).

Markov Chain Monte Carlo in high dimensions



Borgs, Christian, Jennifer T. Chayes, Henry Cohn, and Yufei Zhao. 2014. “An $Lp̂$ Theory of Sparse Graph Convergence I: Limits, Sparse Random Graph Models, and Power Law Distributions.” January 13, 2014. http://arxiv.org/abs/1401.2906.
Bühlmann, Peter, and Sara van de Geer. 2011. Statistics for High-Dimensional Data: Methods, Theory and Applications. 2011 edition. Heidelberg ; New York: Springer.
———. 2015. “High-Dimensional Inference in Misspecified Linear Models.” March 22, 2015. https://doi.org/10.1214/15-EJS1041.
Candès, Emmanuel J., J. Romberg, and T. Tao. 2006. “Robust Uncertainty Principles: Exact Signal Reconstruction from Highly Incomplete Frequency Information.” IEEE Transactions on Information Theory 52 (2): 489–509. https://doi.org/10.1109/TIT.2005.862083.
Chen, Yen-Chi, and Yu-Xiang Wang. n.d. “Discussion on Confidence Intervals and Hypothesis Testing for High-Dimensional Regression.” Accessed July 12, 2015. http://www.stat.cmu.edu/ ryantibs/journalclub/hdconf.pdf.
Chernozhukov, Victor, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. 2016. “Double/Debiased Machine Learning for Treatment and Causal Parameters.” July 29, 2016. http://arxiv.org/abs/1608.00060.
Chernozhukov, Victor, Christian Hansen, Yuan Liao, and Yinchu Zhu. 2018. “Inference For Heterogeneous Effects Using Low-Rank Estimations.” December 19, 2018. http://arxiv.org/abs/1812.08089.
Chernozhukov, Victor, Whitney K. Newey, and Rahul Singh. 2018. “Learning L2 Continuous Regression Functionals via Regularized Riesz Representers.” September 13, 2018. http://arxiv.org/abs/1809.05224.
Geer, Sara van de. 2014a. “Worst Possible Sub-Directions in High-Dimensional Models.” In. Vol. 131. http://arxiv.org/abs/1403.7023.
———. 2014b. “Statistical Theory for High-Dimensional Models.” September 30, 2014. http://arxiv.org/abs/1409.8557.
Geer, Sara van de, Peter Bühlmann, Ya’acov Ritov, and Ruben Dezeure. 2014. “On Asymptotically Optimal Confidence Regions and Tests for High-Dimensional Models.” The Annals of Statistics 42 (3): 1166–1202. https://doi.org/10.1214/14-AOS1221.
Gorban, Alexander N., Ivan Yu Tyukin, and Ilya Romanenko. 2016. “The Blessing of Dimensionality: Separation Theorems in the Thermodynamic Limit.” October 3, 2016. http://arxiv.org/abs/1610.00494.
Gribonval, Rémi, Gilles Blanchard, Nicolas Keriven, and Yann Traonmilin. 2017. “Compressive Statistical Learning with Random Feature Moments.” June 22, 2017. http://arxiv.org/abs/1706.07180.
Gui, Jiang, and Hongzhe Li. 2005. “Penalized Cox Regression Analysis in the High-Dimensional and Low-Sample Size Settings, with Applications to Microarray Gene Expression Data.” Bioinformatics 21 (13): 3001–8. https://doi.org/10.1093/bioinformatics/bti422.
Hall, Peter, and Ker-Chau Li. 1993. “On Almost Linearity of Low Dimensional Projections from High Dimensional Data.” The Annals of Statistics 21 (2): 867–89. https://doi.org/10.1214/aos/1176349155.
Javanmard, Adel, and Andrea Montanari. 2014. “Confidence Intervals and Hypothesis Testing for High-Dimensional Regression.” Journal of Machine Learning Research 15 (1): 2869–909. http://jmlr.org/papers/v15/javanmard14a.html.
Müller, Patric, and Sara van de Geer. 2015. “Censored Linear Model in High Dimensions: Penalised Linear Regression on High-Dimensional Data with Left-Censored Response Variable.” TEST, April. https://doi.org/10.1007/s11749-015-0441-7.
Uematsu, Yoshimasa. 2015. “Penalized Likelihood Estimation in High-Dimensional Time Series Models and Its Application.” April 25, 2015. http://arxiv.org/abs/1504.06706.
Veitch, Victor, and Daniel M. Roy. 2015. “The Class of Random Graphs Arising from Exchangeable Random Measures.” December 7, 2015. http://arxiv.org/abs/1512.03099.
Vershynin, Roman. 2015. “Estimation in High Dimensions: A Geometric Perspective.” In Sampling Theory, a Renaissance: Compressive Sensing and Other Developments, edited by Götz E. Pfander, 3–66. Applied and Numerical Harmonic Analysis. Cham: Springer International Publishing. https://doi.org/10.1007/978-3-319-19749-4_1.
———. 2018. High-Dimensional Probability: An Introduction with Applications in Data Science. 1st ed. Cambridge University Press. https://doi.org/10.1017/9781108231596.
Wainwright, Martin J. 2014. “Structured Regularizers for High-Dimensional Problems: Statistical and Computational Issues.” Annual Review of Statistics and Its Application 1 (1): 233–53. https://doi.org/10.1146/annurev-statistics-022513-115643.
Zhang, Cun-Hui, and Stephanie S. Zhang. 2014. “Confidence Intervals for Low Dimensional Parameters in High Dimensional Linear Models.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 76 (1): 217–42. https://doi.org/10.1111/rssb.12026.

Warning! Experimental comments system! If is does not work for you, let me know via the contact form.

No comments yet!

GitHub-flavored Markdown & a sane subset of HTML is supported.