There are various approximations we might use for a probability distribution. Empirical CDFs, Kernel density estimates, variational approximation, Edgeworth expansions, Laplace approximations…
From each of these we might get close in some metric to the desired target.
This is a broad topic which I cannot hope to cover in full generality. Special cases of interest include
- Statements about where the probability mass is with high probability (concentration theorems)
- statements about the asymptotic distributions of variables eventually approaching some distribution as some parameter goes to infinity (limit theorems. Most famously a lot of things approach normal distributions, but there are many limit theorems
There are other types of results besides, in this domain. I am interested in collecting results that tell me about how various combinations of variables approach a limiting distribution in some probability metric.
See Stein’s method.