# Karhunen-Loève expansions

As seen in chaos expansions kind-of, and used in low rank kriging, and assumed in PCA and implied by many basis decompositions etc. Suppose we have a collection $$\{\varphi_n\}$$ of real valued functions on our index space $$T$$, and a collection $$\{\xi_n\}$$ of uncorrelated random variables. Now we define the random process $f(t)=\sum_{n=1}^{\infty} \xi_{n} \varphi_{n}(t).$ We might care about the first two moments of $$f,$$ i.e. $\mathbb{E}\{f(s) f(t)\}=\sum_{n=1}^{\infty} \sigma_{n}^{2} \varphi_{n}(s) \varphi_{n}(t)$ and variance function $\mathbb{E}\left\{f^{2}(t)\right\}=\sum_{n=1}^{\infty} \sigma_{n}^{2} \varphi_{n}^{2}(t)$

Now suppose that we have a stochastic process where the index $$T$$ is a compact domain in $$\mathbb{R}^{N}$$. The corresponding expansion of $$f$$ in the above form is known as the Karhunen-Loève expansion. Suppose that $$f$$ has covariance function $$K$$ and define an operator $$\mathcal{K}$$, taking the space of square integrable functions on $$T$$ to itself, by $(\mathcal{K} \psi)(t)=\int_{T} K(s, t) \psi(s) d s$ Suppose that $$\lambda_{1} \geq \lambda_{2} \geq \ldots$$, and $$\psi_{1}, \psi_{2}, \ldots$$, are, respectively, the (ordered) eigenvalues and normalised eigenfunctions of the operator. That is, the $$\lambda_{n}$$ and $$\psi_{n}$$ solve the integral equation $\int_{T} K(s, t) \psi(s) d s=\lambda \psi(t)$ with the normalisation $\int_{T} \psi_{n}(t) \psi_{m}(t) d t=\left\{\begin{array}{ll} 1 & n=m \\ 0 & n \neq m \end{array}\right.$ These eigenfunctions lead to a natural expansion of $$K$$, known as Mercer’s Theorem, which states that $K(s, t)=\sum_{n=1}^{\infty} \lambda_{n} \psi_{n}(s) \psi_{n}(t)$ where the series converges absolutely and uniformly on $$T \times T$$. The Karhunen-Loève expansion of $$f$$ is obtained by setting $$\varphi_{n}=$$ $$\lambda_{n}^{\frac{1}{2}} \psi_{n},$$ so that $f_{t}=\sum_{n=1}^{\infty} \lambda_{n}^{\frac{1}{2}} \xi_{n} \psi_{n}(t).$ Now, when does such an expansion exist?

Because I look at signals on unbounded domains a lot, the case where a countable basis cannot be assumed to exist seems important.

## Gaussian

An extra-fun case, when this construction induces a Gaussian process because the $$\xi_{n}$$ are Gaussian. Very handy in Gaussian process regression.

### No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.