# Covariance matrix estimation

## Esp Gaussian

Estimating the thing that is given to you by oracles in statistics homework assignments: the covariance matrix. Or, if the data is indexed by some parameter we might consider the covariance kernel. We are especially interested in this in Gaussian processes, where the covariance structure characterises the process up to its mean.

I am not introducing a complete theory of covariance estimation here, merely mentioning a couple of tidbits for future reference.

Two big data problems problems can arise here: large $$p$$ (ambient dimension) and large $$n$$ (sample size). Large $$p$$ is a problem because the covariance matrix is a $$p \times p$$ matrixand frequently we need to invert it to calculate some target estimand.

Often life can be made not too bad for large $$n$$ with Gaussian structure because, essentially, the problem has a nice exponential family structure and hence has sufficient statistics.

## Bayesian

Inverse Wishart priors. 🏗 Other?

## Precision estimation

The workhorse of learning graphical models under linearity and Gaussianity. See precision estimation for a more complete treatment.

## Continuous

See kernel learning.

## Parametric

### on a lattice

Estimating a stationary covariance function on a regular lattice? That is a whole field of its own. Useful keywords include circulant embedding. Although strictly more general than Gaussian processes on a lattice, it is often used in that context and some extra results are on that page for now.

## Unordered

Thanks to Rothman (2010) I now think about covariance estimates as being different in ordered versus exchangeable data.

## Sandwich estimators

For robust covariances of vector data. AKA Heteroskedasticity-consistent covariance estimators. Incorporating Eicker-Huber-White sandwich estimator, Andrews kernel HAC estimator, Newey-West and others. For an intro see Achim Zeileis, Open-Source Econometric Computing in R.

## References

Abrahamsen, Petter. 1997.
Azizyan, Martin, Akshay Krishnamurthy, and Aarti Singh. 2015. arXiv:1506.00898 [Cs, Math, Stat], June.
Baik, Jinho, Gérard Ben Arous, and Sandrine Péché. 2005. The Annals of Probability 33 (5): 1643–97.
Banerjee, Onureena, Laurent El Ghaoui, and Alexandre d’Aspremont. 2008. Journal of Machine Learning Research 9 (Mar): 485–516.
Barnard, John, Robert McCulloch, and Xiao-Li Meng. 2000. Statistica Sinica 10 (4): 1281–311.
Ben Arous, Gérard, and Sandrine Péché. 2005. Communications on Pure and Applied Mathematics 58 (10): 1316–57.
Bosq, Denis. 2002. Statistical Inference for Stochastic Processes 5 (3): 287–306.
Cai, T. Tony, Cun-Hui Zhang, and Harrison H. Zhou. 2010. The Annals of Statistics 38 (4): 2118–44.
Chan, Tony F., Gene H. Golub, and Randall J. Leveque. 1983. The American Statistician 37 (3): 242–47.
Chen, Hao, Lili Zheng, Raed Al Kontar, and Garvesh Raskutti. 2020. “Stochastic Gradient Descent in Correlated Settings: A Study on Gaussian Processes.” In Proceedings of the 34th International Conference on Neural Information Processing Systems, 2722–33. NIPS’20. Red Hook, NY, USA: Curran Associates Inc.
Chen, Xiaohui, Mengyu Xu, and Wei Biao Wu. 2013. The Annals of Statistics 41 (6).
Cook, R. Dennis. 2018. Annual Review of Statistics and Its Application 5 (1): 533–59.
Cunningham, John P., Krishna V. Shenoy, and Maneesh Sahani. 2008. In Proceedings of the 25th International Conference on Machine Learning, 192–99. ICML ’08. New York, NY, USA: ACM Press.
Damian, Doris, Paul D. Sampson, and Peter Guttorp. 2001. Environmetrics 12 (2): 161–78.
Daniels, M. J., and M. Pourahmadi. 2009. Journal of Multivariate Analysis 100 (10): 2352–63.
Dasgupta, Sanjoy, and Daniel Hsu. 2007. In Learning Theory, edited by Nader H. Bshouty and Claudio Gentile, 4539:278–92. Berlin, Heidelberg: Springer Berlin Heidelberg.
Efron, Bradley. 2010. Journal of the American Statistical Association 105 (491): 1042–55.
Fan, Jianqing, Yuan Liao, and Han Liu. 2016. The Econometrics Journal 19 (1): C1–32.
Friedman, Jerome, Trevor Hastie, and Robert Tibshirani. 2008. Biostatistics 9 (3): 432–41.
Fuentes, Montserrat. 2006. Journal of Statistical Planning and Inference 136 (2): 447–66.
Gneiting, Tilmann, William Kleiber, and Martin Schlather. 2010. Journal of the American Statistical Association 105 (491): 1167–77.
Goodman, Leo A. 1960. Journal of the American Statistical Association 55 (292): 708–13.
Hackbusch, Wolfgang. 2015. Hierarchical Matrices: Algorithms and Analysis. 1st ed. Springer Series in Computational Mathematics 49. Heidelberg New York Dordrecht London: Springer Publishing Company, Incorporated.
Hansen, Christian B. 2007. Journal of Econometrics 140 (2): 670–94.
Heinrich, Claudio, and Mark Podolskij. 2014. arXiv:1410.6764 [Math], October.
Huang, Jianhua Z., Naiping Liu, Mohsen Pourahmadi, and Linxu Liu. 2006. Biometrika 93 (1): 85–98.
James, William, and Charles Stein. 1961. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, 1:361–79. University of California Press.
Janková, Jana, and Sara van de Geer. 2015. arXiv:1507.02061 [Math, Stat], July.
Kauermann, Göran, and Raymond J. Carroll. 2001. Journal of the American Statistical Association 96 (456): 1387–96.
Khoromskij, B. N., A. Litvinenko, and H. G. Matthies. 2009. Computing 84 (1-2): 49–67.
Khoshgnauz, Ehsan. 2012. arXiv:1206.6361 [Cs, Stat], June.
Kuismin, Markku O., and Mikko J. Sillanpää. 2017. WIREs Computational Statistics 9 (6): e1415.
Lam, Clifford, and Jianqing Fan. 2009. Annals of Statistics 37 (6B): 4254–78.
Ledoit, Olivier, and Michael Wolf. 2004. Journal of Multivariate Analysis 88 (2): 365–411.
Ling, Robert F. 1974. Journal of the American Statistical Association 69 (348): 859–66.
Loh, Wei-Liem. 1991. Journal of Multivariate Analysis 36 (2): 163–74.
Mardia, K. V., and R. J. Marshall. 1984. Biometrika 71 (1): 135–46.
Meier, Alexander. 2018.
Meier, Alexander, Claudia Kirch, and Renate Meyer. 2020. Journal of Multivariate Analysis 175 (January): 104560.
Meinshausen, Nicolai, and Peter Bühlmann. 2006. The Annals of Statistics 34 (3): 1436–62.
Minasny, Budiman, and Alex. B. McBratney. 2005. Geoderma, Pedometrics 2003, 128 (3–4): 192–207.
Pébay, Philippe. 2008. Sandia Report SAND2008-6212, Sandia National Laboratories.
Pourahmadi, Mohsen. 2011. Statistical Science 26 (3): 369–87.
Prause, Annabel, and Ansgar Steland. 2018. Electronic Journal of Statistics 12 (1): 890–940.
Ramdas, Aaditya, and Leila Wehbe. 2014. arXiv:1406.1922 [Stat], June.
Ravikumar, Pradeep, Martin J. Wainwright, Garvesh Raskutti, and Bin Yu. 2011. Electronic Journal of Statistics 5: 935–80.
Rosenblatt, M. 1984. The Annals of Probability 12 (4): 1167–80.
Rothman, Adam J. 2010. “Sparse Estimation of High-Dimensional Covariance Matrices.”
Sampson, Paul D., and Peter Guttorp. 1992. Journal of the American Statistical Association 87 (417): 108–19.
Schäfer, Juliane, and Korbinian Strimmer. 2005. Statistical Applications in Genetics and Molecular Biology 4: Article32.
Schmidt, Alexandra M., and Anthony O’Hagan. 2003. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 65 (3): 743–58.
Shao, Xiaofeng, and Wei Biao Wu. 2007. The Annals of Statistics 35 (4): 1773–1801.
Shimotsu, Katsumi, and Peter C. B. Phillips. 2004. The Annals of Statistics 32 (2): 656–92.
Stein, Michael L. 2005. Journal of the American Statistical Association 100 (469): 310–21.
Sun, Ying, and Michael L. Stein. 2016. Journal of Computational and Graphical Statistics 25 (1): 187–208.
Takemura, Akimichi. 1984. Tsukuba Journal of Mathematics 8 (2): 367–76.
Warton, David I. 2008. Journal of the American Statistical Association 103 (481): 340–49.
Whittle, P. 1952. Biometrika 39 (3-4): 309–18.
———. 1953a. Journal of the Royal Statistical Society: Series B (Methodological) 15 (1): 125–39.
———. 1953b. Arkiv För Matematik 2 (5): 423–34.
Whittle, Peter. 1952. Scandinavian Actuarial Journal 1952 (1-2): 48–60.
Wolter, Kirk M. 2007. Introduction to Variance Estimation. 2nd ed. Statistics for Social and Behavioral Sciences. New York: Springer.
Wu, Wei Biao, and Mohsen Pourahmadi. 2003. Biometrika 90 (4): 831–44.
Yuan, Ming, and Yi Lin. 2007. Biometrika 94 (1): 19–35.
Zeileis, Achim. 2004. Journal of Statistical Software 11 (10).
———. 2006a. Computational Statistics & Data Analysis 50 (11): 2987–3008.
———. 2006b. Journal of Statistical Software 16 (1): 1–16.
Zhang, T., and H. Zou. 2014. Biometrika 101 (1): 103–20.

### No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.