Graphs of conditional, directed independence are a convenient formalism for many statistical models. If you have some kind of generating process for a model, often the most natural type of graphical model to express it with is a DAG. These are also called Bayes nets (not to be confused with Bayesian inference.)
These can even be causal graphical models, and when we can infer those we are extracting Science (ONO) from observational data. See causal graphical models.
The laws of message passing inference assume their (MO) most complicated form for directed models; in practice it is frequently easier to convert a directed model to a factor graph for implementation. YMMV.
I found James F Fixx’s puzzle book on the shelf when writing this post
Simpson’s paradox
Simpson’s paradox is an evergreen example of the importance of that causal graph. For a beautiful and clear example see Allen Downey’s Simpson’s Paradox and Age Effects. It is also the key explanation in Michael Nielson’s Reinventing Explanation.
Tools
BayesNets is a Julia package for reasoning over directed graphical models.
Inferring the optimal intervention requires accounting for which arrows are independent of which.
References
Aragam, Bryon, Arash A. Amini, and Qing Zhou. 2015. “Learning Directed Acyclic Graphs with Penalized Neighbourhood Regression.” arXiv:1511.08963 [Cs, Math, Stat], November.
Aragam, Bryon, Jiaying Gu, and Qing Zhou. 2017. “Learning Large-Scale Bayesian Networks with the Sparsebn Package.” arXiv:1703.04025 [Cs, Stat], March.
Aragam, Bryon, and Qing Zhou. 2015. “Concave Penalized Estimation of Sparse Gaussian Bayesian Networks.” Journal of Machine Learning Research 16: 2273–2328.
Aral, Sinan, Lev Muchnik, and Arun Sundararajan. 2009. “Distinguishing Influence-Based Contagion from Homophily-Driven Diffusion in Dynamic Networks.” Proceedings of the National Academy of Sciences 106 (51): 21544–49.
Arnold, Barry C., Enrique Castillo, and Jose M. Sarabia. 1999. Conditional Specification of Statistical Models. Springer Science & Business Media.
Bareinboim, Elias, Jin Tian, and Judea Pearl. 2014. “Recovering from Selection Bias in Causal and Statistical Inference.” In AAAI, 2410–16.
Bloniarz, Adam, Hanzhong Liu, Cun-Hui Zhang, Jasjeet Sekhon, and Bin Yu. 2015. “Lasso Adjustments of Treatment Effect Estimates in Randomized Experiments.” arXiv:1507.03652 [Math, Stat], July.
Brodersen, Kay H., Fabian Gallusser, Jim Koehler, Nicolas Remy, and Steven L. Scott. 2015. “Inferring Causal Impact Using Bayesian Structural Time-Series Models.” The Annals of Applied Statistics 9 (1): 247–74.
Bühlmann, Peter, Markus Kalisch, and Lukas Meier. 2014. “High-Dimensional Statistics with a View Toward Applications in Biology.” Annual Review of Statistics and Its Application 1 (1): 255–78.
Bühlmann, Peter, Philipp Rütimann, and Markus Kalisch. 2013. “Controlling False Positive Selections in High-Dimensional Regression and Causal Inference.” Statistical Methods in Medical Research 22 (5): 466–92.
Buntine, W.L. 1996. “A Guide to the Literature on Learning Probabilistic Networks from Data.” IEEE Transactions on Knowledge and Data Engineering 8 (2): 195–210.
Chen, B, and J Pearl. 2012. “Regression and Causation: A Critical Examination of Econometric Textbooks.”
Christakis, Nicholas A., and James H. Fowler. 2007. “The Spread of Obesity in a Large Social Network over 32 Years.” New England Journal of Medicine 357 (4): 370–79.
Colombo, Diego, Marloes H. Maathuis, Markus Kalisch, and Thomas S. Richardson. 2012. “Learning High-Dimensional Directed Acyclic Graphs with Latent and Selection Variables.” The Annals of Statistics 40 (1): 294–321.
Dawid, A. P. 2001. “Separoids: A Mathematical Framework for Conditional Independence and Irrelevance.” Annals of Mathematics and Artificial Intelligence 32 (1-4): 335–72.
Dawid, A. Philip. 1979. “Conditional Independence in Statistical Theory.” Journal of the Royal Statistical Society. Series B (Methodological) 41 (1): 1–31.
———. 1980. “Conditional Independence for Statistical Operations.” The Annals of Statistics 8 (3): 598–617.
De Luna, Xavier, Ingeborg Waernbaum, and Thomas S. Richardson. 2011. “Covariate Selection for the Nonparametric Estimation of an Average Treatment Effect.” Biometrika, October, asr041.
Edwards, David, and Smitha Ankinakatte. 2015. “Context-Specific Graphical Models for Discrete Longitudinal Data.” Statistical Modelling 15 (4): 301–25.
Fixx, James F. 1977. Games for the superintelligent. London: Muller.
Frey, B.J., and Nebojsa Jojic. 2005. “A Comparison of Algorithms for Inference and Learning in Probabilistic Graphical Models.” IEEE Transactions on Pattern Analysis and Machine Intelligence 27 (9): 1392–1416.
Gu, Jiaying, Fei Fu, and Qing Zhou. 2014. “Adaptive Penalized Estimation of Directed Acyclic Graphs From Categorical Data.” arXiv:1403.2310 [Stat], March.
Guo, Siyuan, Viktor Tóth, Bernhard Schölkopf, and Ferenc Huszár. 2022. “Causal de Finetti: On the Identification of Invariant Causal Structure in Exchangeable Data.” arXiv.
Jordan, Michael I., Zoubin Ghahramani, Tommi S. Jaakkola, and Lawrence K. Saul. 1999. “An Introduction to Variational Methods for Graphical Models.” Machine Learning 37 (2): 183–233.
Jordan, Michael Irwin. 1999. Learning in Graphical Models. Cambridge, Mass.: MIT Press.
Jordan, Michael I., and Yair Weiss. 2002a. “Graphical Models: Probabilistic Inference.” The Handbook of Brain Theory and Neural Networks, 490–96.
———. 2002b. “Probabilistic Inference in Graphical Models.” Handbook of Neural Networks and Brain Theory.
Kalisch, Markus, and Peter Bühlmann. 2007. “Estimating High-Dimensional Directed Acyclic Graphs with the PC-Algorithm.” Journal of Machine Learning Research 8 (May): 613–36.
Koller, Daphne, and Nir Friedman. 2009. Probabilistic Graphical Models : Principles and Techniques. Cambridge, MA: MIT Press.
Krause, Andreas, and Carlos Guestrin. 2009. “Optimal Value of Information in Graphical Models.” J. Artif. Int. Res. 35 (1): 557–91.
Lauritzen, S. L., and D. J. Spiegelhalter. 1988. “Local Computations with Probabilities on Graphical Structures and Their Application to Expert Systems.” Journal of the Royal Statistical Society. Series B (Methodological) 50 (2): 157–224.
Lauritzen, Steffen L. 1996. Graphical Models. Oxford Statistical Science Series. Clarendon Press.
Maathuis, Marloes H., and Diego Colombo. 2013. “A Generalized Backdoor Criterion.” arXiv Preprint arXiv:1307.5636.
Malioutov, Dmitry M., Jason K. Johnson, and Alan S. Willsky. 2006. “Walk-Sums and Belief Propagation in Gaussian Graphical Models.” Journal of Machine Learning Research 7 (October): 2031–64.
Marbach, Daniel, Robert J. Prill, Thomas Schaffter, Claudio Mattiussi, Dario Floreano, and Gustavo Stolovitzky. 2010. “Revealing Strengths and Weaknesses of Methods for Gene Network Inference.” Proceedings of the National Academy of Sciences 107 (14): 6286–91.
Mihalkova, Lilyana, and Raymond J. Mooney. 2007. “Bottom-up Learning of Markov Logic Network Structure.” In Proceedings of the 24th International Conference on Machine Learning, 625–32. ACM.
Montanari, Andrea. 2011. “Lecture Notes for Stat 375 Inference in Graphical Models.”
Murphy, Kevin P. 2012. Machine learning: a probabilistic perspective. 1 edition. Adaptive computation and machine learning series. Cambridge, MA: MIT Press.
Neapolitan, Richard E. 2003. Learning Bayesian Networks. Vol. 38. Prentice Hal, Paperback.
Pearl, Judea. 1982. “Reverend Bayes on Inference Engines: A Distributed Hierarchical Approach.” In Proceedings of the Second AAAI Conference on Artificial Intelligence, 133–36. AAAI’82. Pittsburgh, Pennsylvania: AAAI Press.
———. 1986. “Fusion, Propagation, and Structuring in Belief Networks.” Artificial Intelligence 29 (3): 241–88.
———. 2008. Probabilistic reasoning in intelligent systems: networks of plausible inference. Rev. 2. print., 12. [Dr.]. The Morgan Kaufmann series in representation and reasoning. San Francisco, Calif: Kaufmann.
Pearl, Judea, Dan Geiger, and Thomas Verma. 1989. “Conditional Independence and Its Representations.” Kybernetika 25 (7): 33–44.
Pereda, E, R Q Quiroga, and J Bhattacharya. 2005. “Nonlinear Multivariate Analysis of Neurophysiological Signals.” Progress in Neurobiology 77 (1-2): 1–37.
Pollard, Dave. 2004. “Hammersley-Clifford Theorem for Markov Random Fields.”
Rabbat, Michael G., MÁrio A. T. Figueiredo, and Robert D. Nowak. 2008. “Network Inference from Co-Occurrences.” IEEE Transactions on Information Theory 54 (9): 4053–68.
Ran, Zhi-Yong, and Bao-Gang Hu. 2017. “Parameter Identifiability in Statistical Machine Learning: A Review.” Neural Computation 29 (5): 1151–1203.
Schölkopf, Bernhard, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. 2012. “On Causal and Anticausal Learning.” In ICML 2012.
Shachter, Ross D. 1998. “Bayes-Ball: Rational Pastime (for Determining Irrelevance and Requisite Information in Belief Networks and Influence Diagrams).” In Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, 480–87. UAI’98. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc.
Shalizi, Cosma Rohilla, and Edward McFowland III. 2016. “Controlling for Latent Homophily in Social Networks Through Inferring Latent Locations.” arXiv:1607.06565 [Physics, Stat], July.
Smith, David A., and Jason Eisner. 2008. “Dependency Parsing by Belief Propagation.” In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 145–56. Association for Computational Linguistics.
Spirtes, Peter, Clark Glymour, and Richard Scheines. 2001. Causation, Prediction, and Search. Second Edition. Adaptive Computation and Machine Learning. The MIT Press.
Studený, Milan, and Jiřina Vejnarová. 1998. “On Multiinformation Function as a Tool for Measuring Stochastic Dependence.” In Learning in Graphical Models, 261–97. Cambridge, Mass.: MIT Press.
Su, Ri-Qi, Wen-Xu Wang, and Ying-Cheng Lai. 2012. “Detecting Hidden Nodes in Complex Networks from Time Series.” Phys. Rev. E 85 (6): 065201.
Textor, Johannes, Alexander Idelberger, and Maciej Liśkiewicz. 2015. “Learning from Pairwise Marginal Independencies.” arXiv:1508.00280 [Cs], August.
Visweswaran, Shyam, and Gregory F. Cooper. 2014. “Counting Markov Blanket Structures.” arXiv:1407.2483 [Cs, Stat], July.
Wainwright, Martin J., and Michael I. Jordan. 2008. Graphical Models, Exponential Families, and Variational Inference. Vol. 1. Foundations and Trends® in Machine Learning. Now Publishers.
Weiss, Yair. 2000. “Correctness of Local Probability Propagation in Graphical Models with Loops.” Neural Computation 12 (1): 1–41.
Weiss, Yair, and William T. Freeman. 2001. “Correctness of Belief Propagation in Gaussian Graphical Models of Arbitrary Topology.” Neural Computation 13 (10): 2173–2200.
Winn, John M., and Christopher M. Bishop. 2005. “Variational Message Passing.” In Journal of Machine Learning Research, 661–94.
Wright, Sewall. 1934. “The Method of Path Coefficients.” The Annals of Mathematical Statistics 5 (3): 161–215.
Yedidia, J.S., W.T. Freeman, and Y. Weiss. 2003. “Understanding Belief Propagation and Its Generalizations.” In Exploring Artificial Intelligence in the New Millennium, edited by G. Lakemeyer and B. Nebel, 239–36. Morgan Kaufmann Publishers.
Zhang, Kun, Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. 2012. “Kernel-Based Conditional Independence Test and Application in Causal Discovery.” arXiv:1202.3775 [Cs, Stat], February.
Zhou, Mingyuan, Yulai Cong, and Bo Chen. 2017. “Augmentable Gamma Belief Networks,” 44.
No comments yet. Why not leave one?