Learning graphical models from data

What is independent of what?

Learning the graph structure, not just the clique potentials.

Much more work.

Learning these models turns out to need a conditional independence test, an awareness of multiple testing and graph thoery.

  • bnlearn learns belief networks
  • sparsebn: > A new R package for learning sparse Bayesian networks and other graphical >models from high-dimensional data via sparse regularization. Designed from >the ground up to handle: > > * Experimental data with interventions > * Mixed observational / experimental data > * High-dimensional data with p >> n > * Datasets with thousands of variables (tested up to p=8000) > * Continuous and discrete data > >The emphasis of this package is scalability and statistical consistency on >high-dimensional datasets. For more details on this package, including >worked examples and the methodological background, please see our new >preprint. > > Overview > > The main methods for learning graphical models are: > > * estimate.dag for directed acyclic graphs (Bayesian networks). > * estimate.precision for undirected graphs (Markov random fields). > * estimate.covariance for covariance matrices. > > Currently, estimation of precision and covariances matrices is limited to Gaussian data.
  • Nonparanormal skeptic (🏗)
  • skggm (python) does the Gaussian thing but also has a nice sparsification and good explanation.

Bayati, M., and A. Montanari. 2012. “The LASSO Risk for Gaussian Matrices.” IEEE Transactions on Information Theory 58 (4): 1997–2017. https://doi.org/10.1109/TIT.2011.2174612.

Buntine, W. L. 1996. “A Guide to the Literature on Learning Probabilistic Networks from Data.” IEEE Transactions on Knowledge and Data Engineering 8 (2): 195–210. https://doi.org/10.1109/69.494161.

Bühlmann, Peter, and Sara van de Geer. 2011. Statistics for High-Dimensional Data: Methods, Theory and Applications. 2011 edition. Heidelberg ; New York: Springer.

Cai, T. Tony. 2017. “Global Testing and Large-Scale Multiple Testing for High-Dimensional Covariance Structures.” Annual Review of Statistics and Its Application 4 (1): 423–46. https://doi.org/10.1146/annurev-statistics-060116-053754.

Colombo, Diego, Marloes H. Maathuis, Markus Kalisch, and Thomas S. Richardson. 2012. “Learning High-Dimensional Directed Acyclic Graphs with Latent and Selection Variables.” The Annals of Statistics 40 (1): 294–321. http://projecteuclid.org/euclid.aos/1333567191.

Cox, D. R., and H. S. Battey. 2017. “Large Numbers of Explanatory Variables, a Semi-Descriptive Analysis.” Proceedings of the National Academy of Sciences 114 (32): 8592–5. https://doi.org/10.1073/pnas.1703764114.

Drton, Mathias, and Marloes H. Maathuis. 2017. “Structure Learning in Graphical Modeling.” Annual Review of Statistics and Its Application 4 (1): 365–93. https://doi.org/10.1146/annurev-statistics-060116-053803.

Friedman, Jerome, Trevor Hastie, and Robert Tibshirani. 2008. “Sparse Inverse Covariance Estimation with the Graphical Lasso.” Biostatistics 9 (3): 432–41. https://doi.org/10.1093/biostatistics/kxm045.

Fu, Fei, and Qing Zhou. 2013. “Learning Sparse Causal Gaussian Networks with Experimental Intervention: Regularization and Coordinate Descent.” Journal of the American Statistical Association 108 (501): 288–300. https://doi.org/10.1080/01621459.2012.754359.

Geer, Sara van de. 2014. “Worst Possible Sub-Directions in High-Dimensional Models.” In. Vol. 131. http://arxiv.org/abs/1403.7023.

Geng, Zhi, Yue Liu, Chunchen Liu, and Wang Miao. 2019. “Evaluation of Causal Effects and Local Structure Learning of Causal Networks.” Annual Review of Statistics and Its Application 6 (1): 103–24. https://doi.org/10.1146/annurev-statistics-030718-105312.

Gogate, Vibhav, William Webb, and Pedro Domingos. 2010. “Learning Efficient Markov Networks.” In Advances in Neural Information Processing Systems, 748–56. http://papers.nips.cc/paper/4010-learning-efficient-markov-networks.

Hallac, David, Jure Leskovec, and Stephen Boyd. 2015. “Network Lasso: Clustering and Optimization in Large Graphs,” July. https://doi.org/10.1145/2783258.2783313.

Harris, Naftali, and Mathias Drton. 2013. “PC Algorithm for Nonparanormal Graphical Models.” Journal of Machine Learning Research 14 (1): 3365–83. http://jmlr.org/papers/v14/harris13a.html.

Hinton, Geoffrey E., Simon Osindero, and Kejie Bao. 2005. “Learning Causally Linked Markov Random Fields.” In Proceedings of the 10th International Workshop on Artificial Intelligence and Statistics, 128–35. Citeseer. http://www.cs.toronto.edu/~osindero/PUBLICATIONS/HintonOsinderoBao05_CLMRF.pdf.

Jung, Alexander, Nguyen Tran Quang, and Alexandru Mara. 2017. “When Is Network Lasso Accurate?” April. http://arxiv.org/abs/1704.02107.

Khoshgnauz, Ehsan. 2012. “Learning Markov Network Structure Using Brownian Distance Covariance,” June. http://arxiv.org/abs/1206.6361.

Kocaoglu, Murat, Alex Dimakis, and Sriram Vishwanath. 2017. “Cost-Optimal Learning of Causal Graphs.” In PMLR, 1875–84. http://proceedings.mlr.press/v70/kocaoglu17a.html.

Krämer, Nicole, Juliane Schäfer, and Anne-Laure Boulesteix. 2009. “Regularized Estimation of Large-Scale Gene Association Networks Using Graphical Gaussian Models.” BMC Bioinformatics 10 (1): 384. https://doi.org/10.1186/1471-2105-10-384.

Lederer, Johannes. 2016. “Graphical Models for Discrete and Continuous Data,” September. http://arxiv.org/abs/1609.05551.

Lee, Su-In, Varun Ganapathi, and Daphne Koller. 2006. “Efficient Structure Learning of Markov Networks Using $ L_1 $-Regularization.” In Advances in Neural Information Processing Systems, 817–24. MIT Press. http://machinelearning.wustl.edu/mlpapers/paper_files/NIPS2006_849.pdf.

Liu, Han, Fang Han, Ming Yuan, John Lafferty, and Larry Wasserman. 2012. “The Nonparanormal SKEPTIC,” June. http://arxiv.org/abs/1206.6488.

Liu, Han, John Lafferty, and Larry Wasserman. 2009. “The Nonparanormal: Semiparametric Estimation of High Dimensional Undirected Graphs.” Journal of Machine Learning Research 10 (December): 2295–2328. http://jmlr.org/papers/v10/liu09a.html.

Mansinghka, Vikash, Charles Kemp, Thomas Griffiths, and Joshua Tenenbaum. 2012. “Structured Priors for Structure Learning,” June. http://arxiv.org/abs/1206.6852.

Mazumder, Rahul, and Trevor Hastie. 2012. “The Graphical Lasso: New Insights and Alternatives.” Electronic Journal of Statistics 6 (November): 2125–49. https://doi.org/10.1214/12-EJS740.

Montanari, Andrea. 2012. “Graphical Models Concepts in Compressed Sensing.” Compressed Sensing: Theory and Applications, 394–438. http://arxiv.org/abs/1011.4328.

Neapolitan, Richard E., and others. 2004. Learning Bayesian Networks. Vol. 38. Prentice Hall Upper Saddle River. https://books.secure-services.me/Gentoomen%20Library/Artificial%20Intelligence/Bayesian%20networks/Learning%20Bayesian%20Networks%20-%20Neapolitan%20R.%20E..pdf.

Peters, Jonas, Joris Mooij, Dominik Janzing, and Bernhard Schoelkopf. 2012. “Identifiability of Causal Graphs Using Functional Models,” February. http://arxiv.org/abs/1202.3757.

Ramsey, Joseph, Madelyn Glymour, Ruben Sanchez-Romero, and Clark Glymour. 2017. “A Million Variables and More: The Fast Greedy Equivalence Search Algorithm for Learning High-Dimensional Graphical Causal Models, with an Application to Functional Magnetic Resonance Images.” International Journal of Data Science and Analytics 3 (2): 121–29. https://doi.org/10.1007/s41060-016-0032-z.

Schelldorfer, Jürg, Lukas Meier, and Peter Bühlmann. 2014. “GLMMLasso: An Algorithm for High-Dimensional Generalized Linear Mixed Models Using ℓ1-Penalization.” Journal of Computational and Graphical Statistics 23 (2): 460–77. https://doi.org/10.1080/10618600.2013.773239.

Textor, Johannes, Alexander Idelberger, and Maciej Liśkiewicz. 2015. “Learning from Pairwise Marginal Independencies,” August. http://arxiv.org/abs/1508.00280.

Wu, Rui, R. Srikant, and Jian Ni. 2012. “Learning Graph Structures in Discrete Markov Random Fields.” In INFOCOM Workshops, 214–19. http://www.ifp.illinois.edu/~srikant/WorkingPapers/wusrini11.pdf.

Zhao, Tuo, Han Liu, Kathryn Roeder, John Lafferty, and Larry Wasserman. 2012. “The Huge Package for High-Dimensional Undirected Graph Estimation in R.” Journal of Machine Learning Research : JMLR 13 (April): 1059–62. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4729207/.

Zhou, Mingyuan, Yulai Cong, and Bo Chen. 2017. “Augmentable Gamma Belief Networks,” 44.