Large sample theory

Bayesian posteriors are kinda gaussian

The Bayesian large sample result of notes is the Bernstein–von Mises theorem, which provides some conitions under which the posterior distribution is asymptotically Gaussian.

Particle filters are kinda gaussian

Long story. See (Bishop and Del Moral 2023, 2016; Cérou et al. 2005; Pierre Del Moral, Hu, and Wu 2011; Pierre Del Moral 2004; Pierre Del Moral and Doucet 2009; P. Del Moral, Kurtzmann, and Tugaut 2017).

Fisher Information

Used in ML theory and kinda-sorta in robust estimation, and natural gradient methods. A matrix that tells is how much a new datum affects our parameter estimates. (It is related, I am told, to garden-variety Shannon information, and when that non-obvious fact is more clear to me I shall expand how precisely this is so.) 🏗

Convolution Theorem

The unhelpfully-named convolution theorem of Hájek (1970) — is that related?

Suppose \(\hat{\theta}\) is an efficient estimator of \(\theta\) and \(\tilde{\theta}\) is another, not fully efficient, estimator. The convolution theorem says that, if you rule out stupid exceptions, asymptotically \(\tilde{\theta} = \hat{\theta} + \varepsilon\) where \(\varepsilon\) is pure noise, independent of \(\hat{\theta}.\)

The reason that’s almost obvious is that if it weren’t true, there would be some information about \(\theta\) in \(\tilde{\theta}-\hat{\theta}\), and you could use this information to get a better estimator than \(\hat{\theta}\), which (by assumption) can’t happen. The stupid exceptions are things like the Hodges superefficient estimator that do better at a few values of \(\hat{\theta}\) but much worse at neighbouring values.

References