Precision matrix estimation

Especially Gaussain

Estimating the thing that is given to you by oracles in statistics homework assignments: the covariance matrix or its inverse, the precision matrices. Or, if you data is indexed in some fashion, the covariance kernel. We are especially interested in this in Gaussian processes, where the covariance kernel characterises the process up to its mean.

I am not introducing a complete theory of covariance estimation here, merely mentioning a couple of tidbits for future reference.

Two big data problems problems can arise here: large \(p\) (ambient dimension) and large \(n\) (sample size). Large \(p\) is a problem because the covariance matrix is a \(p \times p\) matrixand frequently we need to invert it to calculate some target estimand.

Often life can be made not too bad for large \(n\) with Gaussian structure because, essentially, it has a nice exponential family structure and hence has sufficient statistics.

The obvious way

Estimate the covariance matrix then invert it. This is the baseline. πŸ—



πŸ— Wishart priors?



Iterative approximation

Saad (2003)

Conjugate gradients



Aragam, Bryon, Jiaying Gu, and Qing Zhou. 2017. β€œLearning Large-Scale Bayesian Networks with the Sparsebn Package.” arXiv:1703.04025 [Cs, Stat], March.
Avagyan, Vahe, and Xiaoling Mei. 2022. β€œPrecision Matrix Estimation Under Data Contamination with an Application to Minimum Variance Portfolio Selection.” Communications in Statistics - Simulation and Computation 51 (4): 1381–1400.
Chen, Xiaohui, Mengyu Xu, and Wei Biao Wu. 2013. β€œCovariance and Precision Matrix Estimation for High-Dimensional Time Series.” The Annals of Statistics 41 (6).
Fan, Jianqing, Yuan Liao, and Han Liu. 2016. β€œAn Overview of the Estimation of Large Covariance and Precision Matrices.” The Econometrics Journal 19 (1): C1–32.
Hsieh, Cho-Jui, MΓ‘tyΓ‘s A. Sustik, Inderjit S. Dhillon, and Pradeep D. Ravikumar. 2014. β€œQUIC: Quadratic Approximation for Sparse Inverse Covariance Estimation.” Journal of Machine Learning Research 15 (1): 2911–47.
Hsieh, Cho-Jui, MΓ‘tyΓ‘s A. Sustik, Inderjit S. Dhillon, Pradeep Ravikumar, and Russell A. Poldrack. 2013. β€œBIG & QUIC: Sparse Inverse Covariance Estimation for a Million Variables.” In Advances in Neural Information Processing Systems, 16. NIPS’13. Red Hook, NY, USA: Curran Associates Inc.
JankovΓ‘, Jana, and Sara van de Geer. 2015. β€œHonest Confidence Regions and Optimality in High-Dimensional Precision Matrix Estimation.” arXiv:1507.02061 [Math, Stat], July.
Khoshgnauz, Ehsan. 2012. β€œLearning Markov Network Structure Using Brownian Distance Covariance.” arXiv:1206.6361 [Cs, Stat], June.
Kuismin, Markku O., and Mikko J. SillanpÀÀ. 2017. β€œEstimation of Covariance and Precision Matrix, Network Structure, and a View Toward Systems Biology.” WIREs Computational Statistics 9 (6): e1415.
Lam, Clifford, and Jianqing Fan. 2009. β€œSparsistency and Rates of Convergence in Large Covariance Matrix Estimation.” Annals of Statistics 37 (6B): 4254–78.
Le, Thien-Minh, and Ping-Shou Zhong. 2021. β€œHigh-Dimensional Precision Matrix Estimation with a Known Graphical Structure.” arXiv.
Meier, Alexander, Claudia Kirch, and Renate Meyer. 2020. β€œBayesian Nonparametric Analysis of Multivariate Time Series: A Matrix Gamma Process Approach.” Journal of Multivariate Analysis 175 (January): 104560.
Moscone, Francesco, Elisa Tosetti, and Veronica Vinciotti. 2017. β€œSparse Estimation of Huge Networks with a Block-Wise Structure.” The Econometrics Journal 20 (3): S61–85.
Pleiss, Geoff, Jacob R. Gardner, Kilian Q. Weinberger, and Andrew Gordon Wilson. 2018. β€œConstant-Time Predictive Distributions for Gaussian Processes.” In. arXiv.
Pourahmadi, Mohsen. 2011. β€œCovariance Estimation: The GLM and Regularization Perspectives.” Statistical Science 26 (3): 369–87.
Saad, Yousef. 2003. Iterative Methods for Sparse Linear Systems: Second Edition. 2nd ed. SIAM.
Ubaru, Shashanka, Jie Chen, and Yousef Saad. 2017. β€œFast Estimation of \(tr(f(A))\) via Stochastic Lanczos Quadrature.” SIAM Journal on Matrix Analysis and Applications 38 (4): 1075–99.
Wang, Lingxiao, Xiang Ren, and Quanquan Gu. n.d. β€œPrecision Matrix Estimation in High Dimensional Gaussian Graphical Models with Faster Rates,” 9.
Wu, Wei Biao, and Mohsen Pourahmadi. 2003. β€œNonparametric Estimation of Large Covariance Matrices of Longitudinal Data.” Biometrika 90 (4): 831–44.
Yuan, Ming. 2010. β€œHigh Dimensional Inverse Covariance Matrix Estimation via Linear Programming.” The Journal of Machine Learning Research 11: 26.
Zhang, T., and H. Zou. 2014. β€œSparse Precision Matrix Estimation via Lasso Penalized D-Trace Loss.” Biometrika 101 (1): 103–20.

No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.