Generalized linear models

Using the machinery of linear regression to predict in somewhat more general regressions, using least-squares or quasi-likelihood approaches. This means you are still doing something like familiar linear regression, but outside the setting of e.g. linear response and possibly homoskedastic Gaussian noise.


Discover the magical powers of log-concavity and what they enable.

Classic linear models

See linear models.

Generalised linear models

The original extension. Kenneth Tay’s explanation is simple and efficient.

To learn:

  • When we can do this? e.g. Must the response be from an exponential family for really real? What happens if not?
  • Does anything funky happen with regularisation? what?
  • model selection theory

Response distribution

πŸ— What constraints do we have here?

Linear Predictor



A generalisation of likelihood of use in some tricky corners of GLMs. (Wedderburn 1976) used it to provide a unified GLM/ML rationale. I don’t yet understand it. Heyde says (Heyde 1997):

Historically there are two principal themes in statistical parameter estimation theory

It is now possible to unify these approaches under the general description of quasi-likelihood and to develop the theory of parameter estimation in a very general setting. […]

It turns out that the theory needs to be developed in terms of estimating functions (functions of both the data and the parameter) rather than the estimators themselves. Thus, our focus will be on functions that have the value of the parameter as a root rather than the parameter itself.

Hierarchical generalised linear models

GLM + hierarchical model = HGLM.

Generalised additive models

Generalised generalised linear models. Semiparametric simultaneous discovery of some non-linear predictors and their response curve under the assumption that the interaction is additive in the transformed predictors \[ g(\operatorname{E}(Y))=\beta_0 + f_1(x_1) + f_2(x_2)+ \cdots + f_m(x_m). \]

These have now also been generalised in the obvious way.

Generalised additive models for location, scale and shape

Folding GARCH and other regression models into GAMs.

GAMLSS website:

GAMLSS is a modern distribution-based approach to (semiparametric) regression models, where all the parameters of the assumed distribution for the response can be modelled as additive functions of the explanatory variables

Vector generalised additive models

See Yee (2015).

Vector generalised hierarchical additive models for location, scale and shape

Exercise for the student.

Generalised estimating equations


But see Johnny Hong and Kellie Ottoboni. Is this just the quasi-likelihood thing again?


GeneralizedΒ² LinearΒ² models (Gordon 2002) unify GLMs with non-linear matrix factorisations.


Atal, B. S. 2006. β€œThe History of Linear Prediction.” IEEE Signal Processing Magazine 23 (2): 154–61.
Barbier, Jean, Florent Krzakala, Nicolas Macris, LΓ©o Miolane, and Lenka ZdeborovΓ‘. 2017. β€œPhase Transitions, Optimal Errors and Optimality of Message-Passing in Generalized Linear Models.” arXiv:1708.03395 [Cond-Mat, Physics:math-Ph], August.
Bolker, Benjamin M., Mollie E. Brooks, Connie J. Clark, Shane W. Geange, John R. Poulsen, M. Henry H. Stevens, and Jada-Simone S. White. 2009. β€œGeneralized Linear Mixed Models: A Practical Guide for Ecology and Evolution.” Trends in Ecology & Evolution 24 (3): 127–35.
Boyd, Nicholas, Trevor Hastie, Stephen Boyd, Benjamin Recht, and Michael Jordan. 2016. β€œSaturating Splines and Feature Selection.” arXiv:1609.06764 [Stat], September.
Breslow, N. E., and D. G. Clayton. 1993. β€œApproximate Inference in Generalized Linear Mixed Models.” Journal of the American Statistical Association 88 (421): 9–25.
Buja, Andreas, Trevor Hastie, and Robert Tibshirani. 1989. β€œLinear Smoothers and Additive Models.” Annals of Statistics 17 (2): 453–510.
Currie, I. D., M. Durban, and P. H. C. Eilers. 2006. β€œGeneralized Linear Array Models with Applications to Multidimensional Smoothing.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 68 (2): 259–80.
Eichler, Michael, Rainer Dahlhaus, and Johannes Dueck. 2016. β€œGraphical Modeling for Multivariate Hawkes Processes with Nonparametric Link Functions.” Journal of Time Series Analysis, January, n/a–.
Finke, Axel, and Sumeetpal S. Singh. 2016. β€œApproximate Smoothing and Parameter Estimation in High-Dimensional State-Space Models.” arXiv:1606.08650 [Stat], June.
Friedman, Jerome, Trevor Hastie, and Rob Tibshirani. 2010. β€œRegularization Paths for Generalized Linear Models via Coordinate Descent.” Journal of Statistical Software 33 (1): 1–22.
Gordon, Geoffrey J. 2002. β€œGeneralizedΒ² LinearΒ² Models.” In Proceedings of the 15th International Conference on Neural Information Processing Systems, 593–600. NIPS’02. Cambridge, MA, USA: MIT Press.
Hansen, Niels Richard. 2010. β€œPenalized Maximum Likelihood Estimation for Generalized Linear Point Processes.” arXiv:1003.0848 [Math, Stat], March.
Hastie, Trevor J., and Robert J. Tibshirani. 1990. Generalized Additive Models. Vol. 43. CRC Press.
Heyde, C. C. 1997. Quasi-likelihood and its application a general approach to optimal parameter estimation. New York: Springer.
Hoaglin, David C., and Roy E. Welsch. 1978. β€œThe Hat Matrix in Regression and ANOVA.” The American Statistician 32 (1): 17–22.
Lee, Youngjo., John A. Nelder, and Yudi Pawitan. 2006. Generalized linear models with random effects. Monographs on statistics and applied probability 106. Boca Raton, FL: Chapman & Hall/CRC.
Lu, Jun. 2022. β€œA Rigorous Introduction to Linear Models.” arXiv.
Mayr, Andreas, Nora Fenske, Benjamin Hofner, Thomas Kneib, and Matthias Schmid. 2012. β€œGeneralized Additive Models for Location, Scale and Shape for High Dimensional Dataβ€”a Flexible Approach Based on Boosting.” Journal of the Royal Statistical Society: Series C (Applied Statistics) 61 (3): 403–27.
McCullagh, Peter. 1984. β€œGeneralized Linear Models.” European Journal of Operational Research 16 (3): 285–92.
Nelder, J. A., and R. J. Baker. 2004. β€œGeneralized Linear Models.” In Encyclopedia of Statistical Sciences. John Wiley & Sons, Inc.
Nelder, J. A., and R. W. M. Wedderburn. 1972. β€œGeneralized Linear Models.” Journal of the Royal Statistical Society. Series A (General) 135 (3): 370–84.
Scandroglio, Giacomo, Andrea Gori, Emiliano Vaccaro, and Vlasios Voudouris. 2013. β€œEstimating VaR and ES of the Spot Price of Oil Using Futures-Varying Centiles.” International Journal of Financial Engineering and Risk Management 1 (1): 6–19.
Stasinopoulos, D. Mikis, and Robert A. Rigby. 2007. β€œGeneralized Additive Models for Location Scale and Shape (GAMLSS) in R.” Journal of Statistical Software 23 (7): 1–46.
Stasinopoulos, Dimitrios, Robert Anthony Rigby, Gillian Heller, Vlasios Voudouris, and Fernanda De Bastiani. n.d. Flexible Regression and Smoothing: Using GAMLSS in R.
Thrampoulidis, Chrtistos, Ehsan Abbasi, and Babak Hassibi. 2015. β€œLASSO with Non-Linear Measurements Is Equivalent to One With Linear Measurements.” In Advances in Neural Information Processing Systems 28, edited by C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, R. Garnett, and R. Garnett, 3402–10. Curran Associates, Inc.
Venables, W. N., and C. M. Dichmont. 2004. β€œGLMs, GAMs and GLMMs: An Overview of Theory for Applications in Fisheries Research.” Fisheries Research, Models in Fisheries Research: GLMs, GAMS and GLMMs, 70 (2–3): 319–37.
Wedderburn, R. W. M. 1974. β€œQuasi-Likelihood Functions, Generalized Linear Models, and the Gaussβ€”Newton Method.” Biometrika 61 (3): 439–47.
β€”β€”β€”. 1976. β€œOn the Existence and Uniqueness of the Maximum Likelihood Estimates for Certain Generalized Linear Models.” Biometrika 63 (1): 27–32.
Wood, Simon N. 2008. β€œFast Stable Direct Fitting and Smoothness Selection for Generalized Additive Models.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 70 (3): 495–518.
Xia, Tian, Xue-Ren Wang, and Xue-Jun Jiang. 2014. β€œAsymptotic Properties of Maximum Quasi-Likelihood Estimator in Quasi-Likelihood Nonlinear Models with Misspecified Variance Function.” Statistics 48 (4): 778–86.
Yee, Thomas W. 2015. Vector Generalized Linear and Additive Models. Springer Series in Statistics. New York, NY: Springer New York.
Zoeter, Onno. 2007. β€œBayesian Generalized Linear Models in a Terabyte World.” In 2007 5th International Symposium on Image and Signal Processing and Analysis, 435–40. Istanbul, Turkey: IEEE.

No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.