Gamma distributions

\[\renewcommand{\var}{\operatorname{Var}} \renewcommand{\corr}{\operatorname{Corr}} \renewcommand{\dd}{\mathrm{d}} \renewcommand{\bb}[1]{\mathbb{#1}} \renewcommand{\vv}[1]{\boldsymbol{#1}} \renewcommand{\rv}[1]{\mathsf{#1}} \renewcommand{\vrv}[1]{\vv{\rv{#1}}} \renewcommand{\disteq}{\stackrel{d}{=}} \renewcommand{\gvn}{\mid} \renewcommand{\Ex}{\mathbb{E}} \renewcommand{\Pr}{\mathbb{P}}\]

The density \(g(x|\alpha, \lambda )\) of the univariate Gamma is \[ g(x| \alpha, \lambda)= \frac{ \lambda^{\alpha} }{ \Gamma (\alpha) } x^{\alpha\,-\,1}e^{-\lambda x}, x\geq 0. \] People refer to this as the shape-rate parameterisation, with rate \(\lambda\) and shape \(\alpha,\).

If \(\rv{g}\sim \operatorname{Gamma}(\alpha, \lambda)\) then \(\bb E(\rv{g})=\alpha/\lambda\) and \(\var(\rv{g})=\alpha/\lambda^2.\)

We use various facts about the Gamma distribution, e.g. its divisibility. Some more are outlined in the Gamma-Dirichlet algebra section.


Note that the moment generating function of the Gamma distribution is

\[\bb{E}[\exp(\rv{g} s)]=\left(1-{\frac {s}{\lambda }}\right)^{-\alpha }{\text{ for }}s<\lambda\] which gives us expressions for all moments fairly easily;

\[\begin{aligned} \bb{E}[\rv{g}]&=\left.\frac{\dd}{\dd s}\bb{E}[\exp(\rv{g} t)]\right|_{s=0}\\ &=\left.\frac{\dd}{\dd s}\left(1-{\frac {s}{\lambda }}\right)^{-\alpha }\right|_{s=0}\\ &=\left.\frac{\alpha}{\lambda}\left(1-{\frac {s}{\lambda }}\right)^{-\alpha -1}\right|_{s=0}\\ &=\alpha/\lambda\\ \bb{E}[\rv{g}^2] &=\left.\frac{\dd}{\dd s}\frac{\alpha}{\lambda}\left(1-{\frac {s}{\lambda }}\right)^{-\alpha -1}\right|_{s=0}\\ &=\left.\frac{\alpha^2+\alpha}{\lambda^2}\left(1-{\frac {s}{\lambda }}\right)^{-\alpha -2}\right|_{s=0}\\ &=\frac{\alpha(\alpha+1)}{\lambda^2}\\ \bb{E}[\rv{g}^3] &=\left.\frac{\dd}{\dd s}\frac{\alpha^2+\alpha}{\lambda^2}\left(1-{\frac {s}{\lambda }}\right)^{-\alpha -2}\right|_{s=0}\\ &=\frac{\alpha(\alpha+1)(\alpha+2)}{\lambda^3}\left(1-{\frac {s}{\lambda }}\right)^{-\alpha -2}\\ &=\frac{\alpha(\alpha+1)(\alpha+2)}{\lambda^3}\\ \bb{E}[\rv{g}^4] &=\frac{\alpha(\alpha+1)(\alpha+2)(\alpha+3)}{\lambda^4}\\ &\dots\\ \bb{E}[\rv{g}^n] &=\frac{\langle \alpha \rangle_{n}}{\lambda^n}\\ \end{aligned}\]

Here \(\langle \alpha \rangle_{n}:=\frac{\Gamma(\alpha+n)}{\Gamma(\alpha)}\) is the rising factorial.

As exponential family

The gamma distribution is a two-parameter exponential family with natural parameters \(T=[\alpha − 1,−\lambda]^\top\), and natural statistics \([X,\ln(X)]^\top.\) In fact, it is a natural exponential family with quadratic variance function (Morris 1983).

Linear combinations of Gammas

Is the Gamma family closed under addition? For a fixed scale/rate parameters, yes, see Gamma-Dirichlet algebra.

If we are summing gammas which differ in the rate parameter \(\lambda\), the result is not in general Gamma distributed, although it can be compactly expressed in terms of Gamma densities. If we wish to know the distribution of the sum of a set of arbitrary Gamma random variables with different rate parameters, we can try to approximate the sum with moment matching (Mathai 1982; Moschopoulos 1985) but it is tedious.

Note that multiplying a gamma RV by a scalar changes the rate. The moral is that we cannot be quite so free and easy with linear combination additivity is not in general so straightforward for Gamma distributions, unlike Gaussians.

Gamma-Dirichlet algebra

OK we do not get linear combination as we do in the Gaussian case. There are various operations which give us similar conveniences, however. For those, we need to also be aware of the Dirichlet and Beta distributions. Here are some useful properties, drawn from, or extrapolated from, Dufresne (1998), Lin (2016), and Pérez-Abreu and Stelzer (2014) which use those properties

But first, a quick bit of notation. From here on, all variables denoted \(\rv{g}_{a}\) (for some \(a>0\), with or without superscript) have a \(\operatorname{Gamma}(a, 1)\) distribution; all variables denoted \(\rv{b}_{a, b}\) (for some \(a, b>0\), with or without superscript) have a \(\operatorname{Beta}_{a, b}\) distribution. In all expressions the variables \(\rv{g}_{a_{1}}, \rv{g}_{a_{2}}', \ldots, \rv{b}_{a_{1}, b_{1}}, \rv{b}_{a_{2}, b_{2}}', \ldots\) are independent unless I say otherwise.

\(\rv{g}_{\alpha_1}+\rv{g}_{\alpha_2}\sim \operatorname{Gamma}(\alpha_1+\alpha_2, \lambda)\)
If \(\rv{g}\sim \operatorname{Gamma}(\alpha, \lambda)\) then \(c \rv{g}\sim \operatorname{Gamma}(\alpha, \lambda/c).\) This looks useful but in practice few constructions I handle vary \(\lambda\).
Beta Thinning {#beta-thinning}

\(\frac{\rv{g}_{\alpha_1}}{\rv{g}_{\alpha_1}+\rv{g}_{\alpha_2}}\sim \operatorname{Beta}(\alpha_1, \alpha_2)\) independent of \(\rv{g}_{\alpha_1}+\rv{g}_{\alpha_2}.\) Equivalently, \(\rv{g}_{\alpha_1+\alpha_2} \rv{b}_{\alpha_1,\alpha_2}\sim\operatorname{Gamma}(\alpha_1)\) independent of \(\rv{b}_{\alpha_1,\alpha_2}.\)

The Gamma-bridge construction arises from this thinning procedure.

Dirichlet thinning

Grab a set of independent Gamma rvs, \(\{\rv{g}_{a_i}\}_{i=1,\dots,k}\), and define \(\rv{s}=\sum_i\rv{g}_{a_i}.\) We know that \(\rv{s}\sim\operatorname{Gamma}(\sum_i a_i,1).\) But wait! There is more. Define \[ \rv{d}_i=\frac{\rv{g}_{a_i}}{\rv{s}}. \] The \(\rv{d}_i\sim\operatorname{Dirichlet}(\vv{\alpha})\), independently of \(\rv{s}.\)

Conversely, take some arbitrary \(\rv{g}_{A}\) and some \(\vrv{d}\sim\operatorname{Dirichlet}(\vv{a})\) with \(\sum_i a_i=B.\) Then \(\rv{g}_{A}\vrv{d}_i\sim \operatorname{Gamma}(\frac{Aa_i}{B}, \lambda),\) and also the random variates \(\rv{g}_{A}\vrv{d}_i, i=1,\dots,k\) are jointly independent.

Beta thickening

Grab a set of independent Gamma rvs, \(\{\rv{g}_{a_i}\}_{i=1,\dots,k}.\) If we take a set of Beta rvs with arbitrary dependence, \(\{\rv{b}_{\kappa_i,a_i-\kappa_i}\}_{i=1,\dots,k}\) then the product rvs \(\{(\rv{g}_{a_i}\rv{b}_{\kappa_i,a_i-\kappa_i})\sim \operatorname{Gamma}(\kappa_i,\lambda)\}_{i=1,\dots,k}\) are jointly independent Gamma variates. Thus \(\sum_i\rv{g}_{a_i}\rv{b}_{\kappa_i,a_i-\kappa_i}\sim \operatorname{Gamma}(\sum_i \kappa_i,\lambda).\) As a special case, if \(\kappa_i\equiv \kappa,\) then \(\sum_i\rv{g}_{a_i}\rv{b}_{\kappa_i,a_i-\kappa_i}\sim \operatorname{Gamma}(\sum_i \kappa_i,\lambda).\)

TODO: Check this. Also, is it actually useful? I thought it was for coupling Gamma processes, but perhaps not.

There are many other nice properties and relations.

The properties I include in this section fail to define a formal algebraic structure, but they do define a bunch of operations that preserve membership of a certain distributional family, or pretty close to. 差不多. Dufresne (1998)’s construction is a more formal algebra, although that construction is too specialised for my needs.

The operations that arise most often in this Gamma-“algebra” are not quite the same as in the Gaussian process “algebra”. In that case we are usually concerned with linear algebras in that many linear operations on many objects which are Gaussian in a very broad sense still end up being Gaussian and possess of a closed-form solution. In this case we are mostly concerned with different operations, especially those thinning (Steutel and van Harn 2003) ones.

Yor (2007) talks about the Gamma-Beta algebra of Dufresne (1998) which relates certain Markov chains of Gamma distribution and Beta distributions. Read that paper for more. Here is a taster:

For any \(w, x, y, z>0\), \[ \rv{b}_{w, x} \rv{g}_{y}+\rv{g}_{z}' \disteq \rv{g}_{y+z}^{\prime \prime}\left(1-\rv{b}_{x, w} \rv{b}_{y, z}'\right) . \] In particular, for any \(w, x, y>0\), \[ \rv{b}_{w, x+y} \rv{g}_{x}+\rv{g}_{y}'=\rv{g}_{x+y} \rv{b}_{w+y, x} \disteq \rv{g}_{w+y} \rv{x}_{x+y, w} . \]

For more, see the Gamma-Beta notebook.

Generalized Gamma Convolution

As noted under divisible distributions, the class of Generalized Gamma Convolution (GGC) is a construction that represents some startling (to me) processes as a certain type of generalization of Gamma distributions. This family includes Pareto (Thorin 1977a) and Lognormal (Thorin 1977b) distributions. Those Thorin papers introduced the idea originally; possibly it is easy to start from one of the textbooks or overviews (Bondesson 2012; James, Roynette, and Yor 2008; Steutel and van Harn 2003; BarndorffNielsenClasses2006?).

AFAICT this allows us to prove lots of nice things about such distributions. It is less easy to get computational methods this way.

The GGC convolves a Gamma distribution with some measure and makes a new divisible distribution. James, Roynette, and Yor (2008):

we say that a positive r.v. \(\Gamma\) is a generalized gamma convolution \((\mathrm{GGC})\) - … if there exists a positive Radon measure \(\mu\) on \(] 0, \infty[\) such that: \[ \begin{aligned} E\left[e^{-\lambda \Gamma}\right]=& \exp \left\{-\int_{0}^{\infty}\left(1-e^{-\lambda x}\right) \frac{d x}{x} \int_{0}^{\infty} e^{-x z} \mu(d z)\right\} \\ =& \exp \left\{-\int_{0}^{\infty} \log \left(1+\frac{\lambda}{z}\right) \mu(d z)\right\} \\ \text { with: } & \int_{] 0,1]}|\log x| \mu(d x)<\infty \text { and } \int_{[1, \infty[} \frac{\mu(d x)}{x}<\infty. \end{aligned} \]

(BarndorffNielsenClasses2006?) and Pérez-Abreu and Stelzer (2014) generalize the GGC to vector- and matrix-valued distributions.

Simulating Gamma variates


A Gamma variate can be generated by many methods (Ahrens and Dieter 1974), e.g. a transformed normal and a uniform random variable (Ahrens and Dieter 1982), or two uniforms, depending on the parameter range. Most methods involve a rejection step. Here is Devroye (1986) summary for beta generators for \(0 < a, b\leq 1\) and Gamma generators for \(a<1\):

Johnk’s beta generator

REPEAT Generate iid uniform [0,1] random variates \(\rv{u}, \rv{v}\)

\[ \begin{array}{c} \rv{x} \leftarrow \rv{u}^{\frac{1}{a}}, \rv{y}-\rv{v}^{\frac{1}{b}} \\ \text { UNTL } \rv{x}+\rv{y} \leq 1 \end{array} \]

RETURN \(\frac{\rv{x}}{\rv{x}+\rv{y}}\)

\((\rv{x} \text { is beta }(a, b) \text { distributed })\)

Berman’s beta generator

REPEAT Generate iid uniform [0,1] random variates \(\rv{u}, \rv{v}\)

\[ \begin{array}{c} \rv{x} \leftarrow \rv{u}^{\frac{1}{4}}, \rv{y} \leftarrow \rv{v}^{\frac{1}{b}} \\ \text { UNTIL } \rv{x}+\rv{y} \leq \mathbf{1} \end{array} \]

RETURN \(\rv{x}\)

\((\rv{x} \text { is beta }(a, b+1) \text { distributed })\)

Johnk’s gamma generator

REPEAT Generate iid uniform [0,1] random variates \(\rv{u}, \rv{v}\)

\[ \begin{array}{r} \rv{x}+\rv{u}^{\frac{1}{a}}, \rv{y} \leftarrow \rv{v}^{\frac{1}{1-a}} \\ \text { UNTL } \rv{x}+\rv{y} \leq 1 \end{array} \]

Generate an exponential random variate \(E\) RETURN \(\frac{E \rv{x}}{\rv{x}+\rv{y}}\)

\((\rv{x} \text { is gamma ( } a\) ) distributed)

Berman’s gamma generator

REPEAT Generate iid uniform [0,1] random variates \(\rv{u}, \rv{v}\)

\[ \begin{array}{r} \rv{x} \leftarrow \rv{u}^{\frac{1}{4}}, \rv{y} \leftarrow \rv{v}^{\frac{1}{1-a}} \\ \text { UNTL } \rv{x}+\rv{y} \leq 1 \end{array} \]

Generate a gamma ( 2 ) random variate \(\rv{z}\) (either as the sum of two iid exponential random variates or as \(-\log (\rv{u} * \rv{v} *)\) where \(\rv{u} *, \rv{v} *\) are lid uniform [0,1] random variates

RETURN \(\rv{z} \rv{x}\)

\((\rv{x} \text { is gamma }(a) \text { distributed })\)


Ahrens, J. H., and U. Dieter. 1974. Computer Methods for Sampling from Gamma, Beta, Poisson and Bionomial Distributions.” Computing 12 (3): 223–46.
———. 1982. Generating Gamma Variates by a Modified Rejection Technique.” Communications of the ACM 25 (1): 47–54.
Bondesson, Lennart. 1979. A General Result on Infinite Divisibility.” The Annals of Probability 7 (6): 965–79.
———. 2012. Generalized Gamma Convolutions and Related Classes of Distributions and Densities. Lecture Notes in Statistics 76. New York: Springer Science & Business Media.
Devroye, Luc. 1986. Non-uniform random variate generation. New York: Springer.
Dufresne, Daniel. 1998. Algebraic Properties of Beta and Gamma Distributions, and Applications.” Advances in Applied Mathematics 20 (3): 285–99.
James, Lancelot F., Bernard Roynette, and Marc Yor. 2008. Generalized Gamma Convolutions, Dirichlet Means, Thorin Measures, with Explicit Examples.” Probability Surveys 5: 346–415.
Lin, Jiayu. 2016. “On The Dirichlet Distribution,” 75.
Mathai, A. M. 1982. Storage Capacity of a Dam with Gamma Type Inputs.” Annals of the Institute of Statistical Mathematics 34 (3): 591–97.
Morris, Carl N. 1982. Natural Exponential Families with Quadratic Variance Functions.” The Annals of Statistics 10 (1): 65–80.
———. 1983. Natural Exponential Families with Quadratic Variance Functions: Statistical Theory.” The Annals of Statistics 11 (2): 515–29.
Morris, Carl N., and Kari F. Lock. 2009. Unifying the Named Natural Exponential Families and Their Relatives.” The American Statistician 63 (3): 247–53.
Moschopoulos, P. G. 1985. The Distribution of the Sum of Independent Gamma Random Variables.” Annals of the Institute of Statistical Mathematics 37 (3): 541–44.
Pérez-Abreu, Victor, and Robert Stelzer. 2014. Infinitely Divisible Multivariate and Matrix Gamma Distributions.” Journal of Multivariate Analysis 130 (September): 155–75.
Roychowdhury, Anirban, and Brian Kulis. 2015. Gamma Processes, Stick-Breaking, and Variational Inference.” In Artificial Intelligence and Statistics, 800–808. PMLR.
Steutel, Fred W., and Klaas van Harn. 2003. Infinite Divisibility of Probability Distributions on the Real Line. Boca Raton: CRC Press.
Thorin, Olof. 1977a. On the Infinite Divisbility of the Pareto Distribution.” Scandinavian Actuarial Journal 1977 (1): 31–40.
———. 1977b. On the Infinite Divisibility of the Lognormal Distribution.” Scandinavian Actuarial Journal 1977 (3): 121–48.
Yor, Marc. 2007. Some Remarkable Properties of Gamma Processes.” In Advances in Mathematical Finance, edited by Michael C. Fu, Robert A. Jarrow, Ju-Yi J. Yen, and Robert J. Elliott, 37–47. Applied and Numerical Harmonic Analysis. Birkhäuser Boston.

No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.