# Karhunen-Loève expansions

September 16, 2019 — February 1, 2022

convolution
functional analysis
Gaussian
Hilbert space
kernel tricks
metrics
nonparametric
signal processing
statistics
stochastic processes

As seen in chaos expansions kind-of, and used in low rank kriging, and assumed in PCA and implied by many basis decompositions etc.

Suppose we have a collection $$\{\varphi_n\}$$ of real valued functions on our index space $$T$$, and a collection $$\{\xi_n\}$$ of uncorrelated random variables. Now we define the random process $f(t)=\sum_{n=1}^{\infty} \xi_{n} \varphi_{n}(t).$ We might care about the first two moments of $$f,$$ i.e. $\mathbb{E}\{f(s) f(t)\}=\sum_{n=1}^{\infty} \sigma_{n}^{2} \varphi_{n}(s) \varphi_{n}(t)$ and variance function $\mathbb{E}\left\{f^{2}(t)\right\}=\sum_{n=1}^{\infty} \sigma_{n}^{2} \varphi_{n}^{2}(t)$

Now suppose that we have a stochastic process where the index $$T$$ is a compact domain in $$\mathbb{R}^{N}$$. The corresponding expansion of $$f$$ in the above form is known as the Karhunen-Loève expansion. Suppose that $$f$$ has covariance function $$K$$ and define an operator $$\mathcal{K}$$, taking the space of square integrable functions on $$T$$ to itself, by $(\mathcal{K} \psi)(t)=\int_{T} K(s, t) \psi(s) d s$ Suppose that $$\lambda_{1} \geq \lambda_{2} \geq \ldots$$, and $$\psi_{1}, \psi_{2}, \ldots$$, are, respectively, the (ordered) eigenvalues and normalised eigenfunctions of the operator. That is, the $$\lambda_{n}$$ and $$\psi_{n}$$ solve the integral equation $\int_{T} K(s, t) \psi(s) d s=\lambda \psi(t)$ with the normalisation $\int_{T} \psi_{n}(t) \psi_{m}(t) d t=\left\{\begin{array}{ll} 1 & n=m \\ 0 & n \neq m \end{array}\right.$ These eigenfunctions lead to a natural expansion of $$K$$, known as Mercer’s Theorem, which states that $K(s, t)=\sum_{n=1}^{\infty} \lambda_{n} \psi_{n}(s) \psi_{n}(t)$ where the series converges absolutely and uniformly on $$T \times T$$. The Karhunen-Loève expansion of $$f$$ is obtained by setting $$\varphi_{n}=$$ $$\lambda_{n}^{\frac{1}{2}} \psi_{n},$$ so that $f_{t}=\sum_{n=1}^{\infty} \lambda_{n}^{\frac{1}{2}} \xi_{n} \psi_{n}(t).$ Now, when does such an expansion exist?

Because I look at signals on unbounded domains a lot, the case where a countable basis cannot be assumed to exist seems important.

## 1 Gaussian

An extra-fun case, when this construction induces a Gaussian process because the $$\xi_{n}$$ are Gaussian. Very handy in Gaussian process regression.

## 2 References

Adler, Robert J. 2010. The Geometry of Random Fields.
Adler, Robert J., and Taylor. 2007. Random Fields and Geometry. Springer Monographs in Mathematics 115.
Adler, Robert J, Taylor, and Worsley. 2016. Applications of Random Fields and Geometry Draft.
Alexanderian. 2015. arXiv:1509.07526 [Math].
Battey, and Sancetta. 2013. Journal of Multivariate Analysis.
Calatayud Gregori, Chen-Charpentier, Cortés López, et al. 2019. Symmetry.
Ghanem, Roger, and Red-Horse. 2017. In Handbook of Uncertainty Quantification.
Ghanem, Roger, and Spanos. 1990. Journal of Applied Mechanics.
Ghanem, Roger G., and Spanos. 2003. Stochastic Finite Elements: A Spectral Approach.
Gratiet, Marelli, and Sudret. 2016. In Handbook of Uncertainty Quantification.
Horváth, and Kokoszka. 2012. Inference for functional data with applications. Springer series in statistics.
Kim, and Braatz. 2013. International Journal of Control.
Kim, Shen, Nagy, et al. 2013. IEEE Control Systems Magazine.
Lei, Li, Gao, et al. 2018.
Liu, Ray, and Hooker. 2014. arXiv:1411.4681 [Math, Stat].
O’Hagan. 2013. “Polynomial Chaos: A Tutorial and Critique from a Statistician’s Perspective.”
Shang. 2014. AStA Advances in Statistical Analysis.
Wahba. 1990. Spline Models for Observational Data.
Xiu, and Hesthaven. 2005. SIAM Journal on Scientific Computing.
Yaglom. 1961. Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 2: Contributions to Probability Theory.