Exponential families! The secret magic at the heart of traditional statistics.
Exponential families are (sorta) probability distributions that just work, in the sense that and the things you would hope you can do with them, you can.
Noted here so I have somewhere to dump notes, but not something I am going to go into right now.
Most interesting models are not exponential families. There are curved exponential families which generalise exponential families in some way that I have not looked into.
Altun, Yasemin, Alex J. Smola, and Thomas Hofmann. 2004. “Exponential Families for Conditional Random Fields.” In Proceedings of the 20th Conference on Uncertainty in Artificial Intelligence, 2–9. UAI ’04. Arlington, Virginia, United States: AUAI Press. http://arxiv.org/abs/1207.4131.
Brown, Lawrence D. 1986. Fundamentals of Statistical Exponential Families: With Applications in Statistical Decision Theory. Lecture Notes-Monograph Series, v. 9. Hayward, Calif: Institute of Mathematical Statistics.
Brown, Lawrence D., T. Tony Cai, and Harrison H. Zhou. 2010. “Nonparametric Regression in Exponential Families.” The Annals of Statistics 38 (4): 2005–46. https://doi.org/10.1214/09-AOS762.
Efron, Bradley. 1978. “The Geometry of Exponential Families.” The Annals of Statistics 6 (2): 362–76. https://doi.org/10.1214/aos/1176344130.
Jensen, Jens Ledet, and Jesper Møller. 1991. “Pseudolikelihood for Exponential Family Models of Spatial Point Processes.” The Annals of Applied Probability 1 (3): 445–61. https://doi.org/10.1214/aoap/1177005877.
Jung, Alexander, Sebastian Schmutzhard, and Franz Hlawatsch. 2012. “The RKHS Approach to Minimum Variance Estimation Revisited: Variance Bounds, Sufficient Statistics, and Exponential Families,” October. http://arxiv.org/abs/1210.6516.
Shen, Xiaotong, Hsin-Cheng Huang, and Jimmy Ye. 2004. “Adaptive Model Selection and Assessment for Exponential Family Distributions.” Technometrics 46 (3): 306–17. https://doi.org/10.1198/004017004000000338.
Tansey, Wesley, Oscar Hernan Madrid Padilla, Arun Sai Suggala, and Pradeep Ravikumar. 2015. “Vector-Space Markov Random Fields via Exponential Families.” In Journal of Machine Learning Research, 684–92. http://jmlr.csail.mit.edu/proceedings/papers/v37/tansey15.html.
Wainwright, Martin J., and Michael I. Jordan. 2008. Graphical Models, Exponential Families, and Variational Inference. Vol. 1. Foundations and Trends® in Machine Learning. http://www.cs.berkeley.edu/~jordan/papers/wainwright-jordan-fnt.pdf.