Covariance estimation for stochastic processes

Conversely, precision estimation

Estimating the thing that is given to you by oracles in statistics homework assignments: the covariance matrix or its inverse, the precision matrices. Or, if you data is indexed in some fashion, the covariance kernel. We are especially interested in this in Gaussian processes, where the covariance kernel characterises the process up to its mean.

I am not introducing a complete theory of covariance estimation here, merely mentioning a couple of tidbits for future reference.


Wishart priors. πŸ—

Precision estimation

The workhorse of learning graphical models under linearity and Gaussianity. πŸ—οΈ

Sandwich estimators

For robust covariances of vector data. AKA Heteroskedasticity-consistent covariance estimators. Incorporating Eicker-Huber-White sandwich estimator, Andrews kernel HAC estimator, Newey-West and others. For an intro see Achim Zeileis, Open-Source Econometric Computing in R.

Parametric covariance functions

There are many fun covariance models. What can we say about convergence of their estimators.



Abrahamsen, Petter. 1997. β€œA Review of Gaussian Random Fields and Correlation Functions.”
Aragam, Bryon, Jiaying Gu, and Qing Zhou. 2017. β€œLearning Large-Scale Bayesian Networks with the Sparsebn Package.” arXiv:1703.04025 [Cs, Stat], March.
Azizyan, Martin, Akshay Krishnamurthy, and Aarti Singh. 2015. β€œExtreme Compressive Sampling for Covariance Estimation.” arXiv:1506.00898 [Cs, Math, Stat], June.
Baik, Jinho, GΓ©rard Ben Arous, and Sandrine PΓ©chΓ©. 2005. β€œPhase Transition of the Largest Eigenvalue for Nonnull Complex Sample Covariance Matrices.” The Annals of Probability 33 (5): 1643–97.
Banerjee, Onureena, Laurent El Ghaoui, and Alexandre d’Aspremont. 2008. β€œModel Selection Through Sparse Maximum Likelihood Estimation for Multivariate Gaussian or Binary Data.” Journal of Machine Learning Research 9 (Mar): 485–516.
Barnard, John, Robert McCulloch, and Xiao-Li Meng. 2000. β€œModeling Covariance Matrices in Terms of Standard Deviations and Correlations, with Application to Shrinkage.” Statistica Sinica 10 (4): 1281–311.
Ben Arous, GΓ©rard, and Sandrine PΓ©chΓ©. 2005. β€œUniversality of Local Eigenvalue Statistics for Some Sample Covariance Matrices.” Communications on Pure and Applied Mathematics 58 (10): 1316–57.
Bosq, Denis. 2002. β€œEstimation of Mean and Covariance Operator of Autoregressive Processes in Banach Spaces.” Statistical Inference for Stochastic Processes 5 (3): 287–306.
Cai, T. Tony, Cun-Hui Zhang, and Harrison H. Zhou. 2010. β€œOptimal Rates of Convergence for Covariance Matrix Estimation.” The Annals of Statistics 38 (4): 2118–44.
Chan, G., and A. T. A. Wood. 1999. β€œSimulation of Stationary Gaussian Vector Fields.” Statistics and Computing 9 (4): 265–68.
Chan, Tony F., Gene H. Golub, and Randall J. Leveque. 1983. β€œAlgorithms for Computing the Sample Variance: Analysis and Recommendations.” The American Statistician 37 (3): 242–47.
Chen, Xiaohui, Mengyu Xu, and Wei Biao Wu. 2013. β€œCovariance and Precision Matrix Estimation for High-Dimensional Time Series.” The Annals of Statistics 41 (6).
Cook, R. Dennis. 2018. β€œPrincipal Components, Sufficient Dimension Reduction, and Envelopes.” Annual Review of Statistics and Its Application 5 (1): 533–59.
Cunningham, John P., Krishna V. Shenoy, and Maneesh Sahani. 2008. β€œFast Gaussian Process Methods for Point Process Intensity Estimation.” In Proceedings of the 25th International Conference on Machine Learning, 192–99. ICML ’08. New York, NY, USA: ACM Press.
Damian, Doris, Paul D. Sampson, and Peter Guttorp. 2001. β€œBayesian Estimation of Semi-Parametric Non-Stationary Spatial Covariance Structures.” Environmetrics 12 (2): 161–78.
Daniels, M. J., and M. Pourahmadi. 2009. β€œModeling Covariance Matrices via Partial Autocorrelations.” Journal of Multivariate Analysis 100 (10): 2352–63.
Dasgupta, Sanjoy, and Daniel Hsu. 2007. β€œOn-Line Estimation with the Multivariate Gaussian Distribution.” In Learning Theory, edited by Nader H. Bshouty and Claudio Gentile, 4539:278–92. Berlin, Heidelberg: Springer Berlin Heidelberg.
Davies, Tilman M., and David Bryant. 2013. β€œOn Circulant Embedding for Gaussian Random Fields in R.” Journal of Statistical Software 55 (9).
Dietrich, C. R., and G. N. Newsam. 1993. β€œA Fast and Exact Method for Multidimensional Gaussian Stochastic Simulations.” Water Resources Research 29 (8): 2861–69.
Efron, Bradley. 2010. β€œCorrelated z-Values and the Accuracy of Large-Scale Statistical Estimates.” Journal of the American Statistical Association 105 (491): 1042–55.
Fan, Jianqing, Yuan Liao, and Han Liu. 2015. β€œAn Overview on the Estimation of Large Covariance and Precision Matrices,” April, 46.
Friedman, Jerome, Trevor Hastie, and Robert Tibshirani. 2008. β€œSparse Inverse Covariance Estimation with the Graphical Lasso.” Biostatistics 9 (3): 432–41.
Fuentes, Montserrat. 2006. β€œTesting for Separability of Spatial–Temporal Covariance Functions.” Journal of Statistical Planning and Inference 136 (2): 447–66.
Gneiting, Tilmann, William Kleiber, and Martin Schlather. 2010. β€œMatΓ©rn Cross-Covariance Functions for Multivariate Random Fields.” Journal of the American Statistical Association 105 (491): 1167–77.
Goodman, Leo A. 1960. β€œOn the Exact Variance of Products.” Journal of the American Statistical Association 55 (292): 708–13.
Gray, Robert M. 2006. Toeplitz and Circulant Matrices: A Review. Vol. 2.
Guinness, Joseph, and Montserrat Fuentes. 2016. β€œCirculant Embedding of Approximate Covariances for Inference From Gaussian Data on Large Lattices.” Journal of Computational and Graphical Statistics 26 (1): 88–97.
Hackbusch, Wolfgang. 2015. Hierarchical Matrices: Algorithms and Analysis. 1st ed. Springer Series in Computational Mathematics 49. Heidelberg New York Dordrecht London: Springer Publishing Company, Incorporated.
Hansen, Christian B. 2007. β€œGeneralized Least Squares Inference in Panel and Multilevel Models with Serial Correlation and Fixed Effects.” Journal of Econometrics 140 (2): 670–94.
Heinrich, Claudio, and Mark Podolskij. 2014. β€œOn Spectral Distribution of High Dimensional Covariation Matrices.” arXiv:1410.6764 [Math], October.
Hsieh, Cho-Jui, MΓ‘tyΓ‘s A. Sustik, Inderjit S. Dhillon, and Pradeep D. Ravikumar. 2014. β€œQUIC: Quadratic Approximation for Sparse Inverse Covariance Estimation.” Journal of Machine Learning Research 15 (1): 2911–47.
Huang, Jianhua Z., Naiping Liu, Mohsen Pourahmadi, and Linxu Liu. 2006. β€œCovariance Matrix Selection and Estimation via Penalised Normal Likelihood.” Biometrika 93 (1): 85–98.
James, William, and Charles Stein. 1961. β€œEstimation with Quadratic Loss.” In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, 1:361–79.
JankovΓ‘, Jana, and Sara van de Geer. 2015. β€œHonest Confidence Regions and Optimality in High-Dimensional Precision Matrix Estimation.” arXiv:1507.02061 [Math, Stat], July.
Kauermann, GΓΆran, and Raymond J. Carroll. 2001. β€œA Note on the Efficiency of Sandwich Covariance Matrix Estimation.” Journal of the American Statistical Association 96 (456): 1387–96.
Khoromskij, B. N., A. Litvinenko, and H. G. Matthies. 2009. β€œApplication of Hierarchical Matrices for Computing the Karhunen–LoΓ¨ve Expansion.” Computing 84 (1-2): 49–67.
Khoshgnauz, Ehsan. 2012. β€œLearning Markov Network Structure Using Brownian Distance Covariance.” arXiv:1206.6361 [Cs, Stat], June.
Krumin, Michael, and Shy Shoham. 2009. β€œGeneration of Spike Trains with Controlled Auto- and Cross-Correlation Functions.” Neural Computation 21 (6): 1642–64.
Kuismin, Markku O., and Mikko J. SillanpÀÀ. 2017. β€œEstimation of Covariance and Precision Matrix, Network Structure, and a View Toward Systems Biology.” WIREs Computational Statistics 9 (6): e1415.
Lam, Clifford, and Jianqing Fan. 2009. β€œSparsistency and Rates of Convergence in Large Covariance Matrix Estimation.” Annals of Statistics 37 (6B): 4254–78.
Ledoit, Olivier, and Michael Wolf. 2004. β€œA Well-Conditioned Estimator for Large-Dimensional Covariance Matrices.” Journal of Multivariate Analysis 88 (2): 365–411.
Ling, Robert F. 1974. β€œComparison of Several Algorithms for Computing Sample Means and Variances.” Journal of the American Statistical Association 69 (348): 859–66.
Loh, Wei-Liem. 1991. β€œEstimating Covariance Matrices II.” Journal of Multivariate Analysis 36 (2): 163–74.
MacKay, David J C. 2002. β€œGaussian Processes.” In Information Theory, Inference & Learning Algorithms, Chapter 45. Cambridge University Press.
Mardia, K. V., and R. J. Marshall. 1984. β€œMaximum Likelihood Estimation of Models for Residual Covariance in Spatial Regression.” Biometrika 71 (1): 135–46.
Meier, Alexander. 2018. β€œA matrix Gamma process and applications to Bayesian analysis of multivariate time series.”
Meier, Alexander, Claudia Kirch, and Renate Meyer. 2020. β€œBayesian Nonparametric Analysis of Multivariate Time Series: A Matrix Gamma Process Approach.” Journal of Multivariate Analysis 175 (January): 104560.
Meinshausen, Nicolai, and Peter BΓΌhlmann. 2006. β€œHigh-Dimensional Graphs and Variable Selection with the Lasso.” The Annals of Statistics 34 (3): 1436–62.
Minasny, Budiman, and Alex. B. McBratney. 2005. β€œThe MatΓ©rn Function as a General Model for Soil Variograms.” Geoderma, Pedometrics 2003, 128 (3–4): 192–207.
Nowak, W., and A. Litvinenko. 2013. β€œKriging and Spatial Design Accelerated by Orders of Magnitude: Combining Low-Rank Covariance Approximations with FFT-Techniques.” Mathematical Geosciences 45 (4): 411–35.
PΓ©bay, Philippe. 2008. β€œFormulas for Robust, One-Pass Parallel Computation of Covariances and Arbitrary-Order Statistical Moments.” Sandia Report Sand2008-6212, Sandia National Laboratories.
Pourahmadi, Mohsen. 2011. β€œCovariance Estimation: The GLM and Regularization Perspectives.” Statistical Science 26 (3): 369–87.
Powell, Catherine E. 2014. β€œGenerating Realisations of Stationary Gaussian Random Fields by Circulant Embedding.” Matrix 2 (2): 1.
Prause, Annabel, and Ansgar Steland. 2018. β€œEstimation of the Asymptotic Variance of Univariate and Multivariate Random Fields and Statistical Inference.” Electronic Journal of Statistics 12 (1): 890–940.
Ramdas, Aaditya, and Leila Wehbe. 2014. β€œStein Shrinkage for Cross-Covariance Operators and Kernel Independence Testing.” arXiv:1406.1922 [Stat], June.
Rasmussen, Carl Edward, and Christopher K. I. Williams. 2006. Gaussian Processes for Machine Learning. Adaptive Computation and Machine Learning. Cambridge, Mass: MIT Press.
Ravikumar, Pradeep, Martin J. Wainwright, Garvesh Raskutti, and Bin Yu. 2011. β€œHigh-Dimensional Covariance Estimation by Minimizing β„“1-Penalized Log-Determinant Divergence.” Electronic Journal of Statistics 5: 935–80.
Rosenblatt, M. 1984. β€œAsymptotic Normality, Strong Mixing and Spectral Density Estimates.” The Annals of Probability 12 (4): 1167–80.
Sampson, Paul D., and Peter Guttorp. 1992. β€œNonparametric Estimation of Nonstationary Spatial Covariance Structure.” Journal of the American Statistical Association 87 (417): 108–19.
SchΓ€fer, Juliane, and Korbinian Strimmer. 2005. β€œA shrinkage approach to large-scale covariance matrix estimation and implications for functional genomics.” Statistical Applications in Genetics and Molecular Biology 4: Article32.
Schmidt, Alexandra M., and Anthony O’Hagan. 2003. β€œBayesian Inference for Non-Stationary Spatial Covariance Structure via Spatial Deformations.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 65 (3): 743–58.
Shao, Xiaofeng, and Wei Biao Wu. 2007. β€œAsymptotic Spectral Theory for Nonlinear Time Series.” The Annals of Statistics 35 (4): 1773–1801.
Shimotsu, Katsumi, and Peter C. B. Phillips. 2004. β€œLocal Whittle Estimation in Nonstationary and Unit Root Cases.” The Annals of Statistics 32 (2): 656–92.
Stein, Michael L. 2005. β€œSpace-Time Covariance Functions.” Journal of the American Statistical Association 100 (469): 310–21.
Sun, Ying, and Michael L. Stein. 2016. β€œStatistically and Computationally Efficient Estimating Equations for Large Spatial Datasets.” Journal of Computational and Graphical Statistics 25 (1): 187–208.
Takemura, Akimichi. 1984. β€œAn Orthogonally Invariant Minimax Estimator of the Covariance Matrix of a Multivariate Normal Population.” Tsukuba Journal of Mathematics 8 (2): 367–76.
Warton, David I. 2008. β€œPenalized Normal Likelihood and Ridge Regularization of Correlation and Covariance Matrices.” Journal of the American Statistical Association 103 (481): 340–49.
Whittle, P. 1952. β€œTests of Fit in Time Series.” Biometrika 39 (3-4): 309–18.
β€”β€”β€”. 1953a. β€œThe Analysis of Multiple Stationary Time Series.” Journal of the Royal Statistical Society: Series B (Methodological) 15 (1): 125–39.
β€”β€”β€”. 1953b. β€œEstimation and Information in Stationary Time Series.” Arkiv FΓΆr Matematik 2 (5): 423–34.
Whittle, Peter. 1952. β€œSome Results in Time Series Analysis.” Scandinavian Actuarial Journal 1952 (1-2): 48–60.
Wolter, Kirk M. 2007. Introduction to Variance Estimation. 2nd ed. Statistics for Social and Behavioral Sciences. New York: Springer.
Yuan, Ming, and Yi Lin. 2007. β€œModel Selection and Estimation in the Gaussian Graphical Model.” Biometrika 94 (1): 19–35.
Zeileis, Achim. 2004. β€œEconometric Computing with HC and HAC Covariance Matrix Estimators.” Journal of Statistical Software 11 (10).
β€”β€”β€”. 2006a. β€œImplementing a Class of Structural Change Tests: An Econometric Computing Approach.” Computational Statistics & Data Analysis 50 (11): 2987–3008.
β€”β€”β€”. 2006b. β€œObject-Oriented Computation of Sandwich Estimators.” Journal of Statistical Software 16 (1): 1–16.
Zhang, T., and H. Zou. 2014. β€œSparse Precision Matrix Estimation via Lasso Penalized D-Trace Loss.” Biometrika 101 (1): 103–20.

No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.