Beta and Dirichlet distributions

October 14, 2019 — April 4, 2022

Suspiciously similar content

Suppose the joint pdf of $d_{1}, \dots, d_{k - 1}$ is $\begin{aligned} f (y_{1}, \dots, y_{k - 1}) & = \frac{α_{1} + \dots + α_{k}}{Γ (α_{1}) \dots Γ (α_{k})} y_{1}^{α_{1} - 1} \dots y_{k - 1}^{α_{k - 1} - 1} {(1 - y_{1} - \dots - y_{k - 1})}^{α_{k} - 1}, \\ = \frac{Γ (α)}{\prod_{i = 1}^{k} Γ (α_{i})} \prod_{i = 1}^{k} y_{i}^{α_{i} - 1} \end{aligned}$ where $y_{i} > 0, y_{1} + \dots + y_{k - 1} < 1, i = 1, \dots, k - 1$ and $α = \sum_{i} α_{i}$ . Then the random variables $d_{1}, \dots, d_{k - 1}$ follow the Dirichlet distribution with parameters $α_{1}, \dots, α_{k}$ . Usually, I write this as a vector random variate, with vector parameters, rather than a long list, $d \sim Dirichlet (α) .$

The Beta distribution is a special case of the Dirichlet distribution with parameters $α = [α_{1}, α_{2}]$ , i.e. the bivariate case.

There is more information in Wikipedia, although these pages are IMO unusually uninspired and confusing. My prose is terrible because I rarely have time to revisit it. What is Wikipedia’s excuse?

1 A Beta RV is a ratio of Gamma RVS

TBD.

2 A Dirichlet RV is a normalized sum of independent Gamma RVS

TBD.

3 Beta as exponential family

Beta distribution: $Y \sim Beta (α, β)$ $\begin{aligned} f_{Y} (y ∣ α, β) = & \frac{Γ (α + β)}{Γ (α) Γ (β)} y^{α - 1} (1 - y)^{β - 1} \\ = & [y (1 - y)]^{- 1} \exp (α \log (y) + β \log (1 - y) \\ + & \log Γ (α + β) - \log Γ (α) - \log Γ (β)) \end{aligned}$ with $\begin{aligned} η (α, β) & = (α, β)^{⊤} \\ T (y) & = (\log (y), \log (1 - y))^{⊤} . \end{aligned}$

4 Dirichlet as exponential family

The Dirichlet distribution is an exponential family and can be written in canonical form as $Dirichlet (θ ∣ α) = f (θ) g (α) e^{ϕ (α)^{T} u (θ)}$ with $f (θ) = 1, g (α) = 1 / B (α)$ where $B (α) = \prod_{t = 1}^{D} Γ (α_{t}) / Γ (\sum_{t = 1}^{D} α_{t}), ϕ (α) = (\begin{matrix} α_{1} - 1 \\ ⋮ \\ α_{D} - 1 \end{matrix})$ and $u (θ) = (\begin{matrix} \ln θ_{1} \\ ⋮ \\ \ln θ_{D} \end{matrix})$

5 Conjugate prior for Dirichlet RVs

Lefkimmiatis, Maragos, and Papandreou (2009) argue:

Since for any member of the exponential family there exists a conjugate prior that can be written in the form $p (α ∣ v, η) \propto g (α)^{η} e^{ϕ (α)^{T} v}$ a suitable conjugate prior distribution for the parameters $α$ of the Dirichlet is $p (α ∣ v, η) \propto \frac{1}{B (α)^{η}} e^{- \sum_{t = 1}^{D} v_{t} α_{t}}$

Wikipedia claims that there is no efficient means for sampling from this, which is sad for MCMC. Generally this does not bother people because we rarely observe Dirichlet RVs directly; they are usually, e.g. a mixing probability for some other distribution.

6 Non-conjugate priors

Anything that can be transformed to be an elementwise positive vector, presumably. multivariate gamma seems natural.

7 References

Devroye. 1986. Non-uniform random variate generation.

Dufresne. 1998. “Algebraic Properties of Beta and Gamma Distributions, and Applications.” Advances in Applied Mathematics.

Lefkimmiatis, Maragos, and Papandreou. 2009. “Bayesian Inference on Multiscale Models for Poisson Intensity Estimation: Applications to Photon-Limited Image Denoising.” IEEE Transactions on Image Processing.

Lin. 2016. “On The Dirichlet Distribution.”

Pérez-Abreu, and Stelzer. 2014. “Infinitely Divisible Multivariate and Matrix Gamma Distributions.” Journal of Multivariate Analysis.

Yor. 2007. “Some Remarkable Properties of Gamma Processes.” In Advances in Mathematical Finance. Applied and Numerical Harmonic Analysis.