# Karhunen-Loève expansions

As seen in chaos expansions kind-of, and used in low rank kriging, and assumed in PCA and implied by many basis decompositions etc.

Suppose we have a collection $$\{\varphi_n\}$$ of real valued functions on our index space $$T$$, and a collection $$\{\xi_n\}$$ of uncorrelated random variables. Now we define the random process $f(t)=\sum_{n=1}^{\infty} \xi_{n} \varphi_{n}(t).$ We might care about the first two moments of $$f,$$ i.e. $\mathbb{E}\{f(s) f(t)\}=\sum_{n=1}^{\infty} \sigma_{n}^{2} \varphi_{n}(s) \varphi_{n}(t)$ and variance function $\mathbb{E}\left\{f^{2}(t)\right\}=\sum_{n=1}^{\infty} \sigma_{n}^{2} \varphi_{n}^{2}(t)$

Now suppose that we have a stochastic process where the index $$T$$ is a compact domain in $$\mathbb{R}^{N}$$. The corresponding expansion of $$f$$ in the above form is known as the Karhunen-Loève expansion. Suppose that $$f$$ has covariance function $$K$$ and define an operator $$\mathcal{K}$$, taking the space of square integrable functions on $$T$$ to itself, by $(\mathcal{K} \psi)(t)=\int_{T} K(s, t) \psi(s) d s$ Suppose that $$\lambda_{1} \geq \lambda_{2} \geq \ldots$$, and $$\psi_{1}, \psi_{2}, \ldots$$, are, respectively, the (ordered) eigenvalues and normalised eigenfunctions of the operator. That is, the $$\lambda_{n}$$ and $$\psi_{n}$$ solve the integral equation $\int_{T} K(s, t) \psi(s) d s=\lambda \psi(t)$ with the normalisation $\int_{T} \psi_{n}(t) \psi_{m}(t) d t=\left\{\begin{array}{ll} 1 & n=m \\ 0 & n \neq m \end{array}\right.$ These eigenfunctions lead to a natural expansion of $$K$$, known as Mercer’s Theorem, which states that $K(s, t)=\sum_{n=1}^{\infty} \lambda_{n} \psi_{n}(s) \psi_{n}(t)$ where the series converges absolutely and uniformly on $$T \times T$$. The Karhunen-Loève expansion of $$f$$ is obtained by setting $$\varphi_{n}=$$ $$\lambda_{n}^{\frac{1}{2}} \psi_{n},$$ so that $f_{t}=\sum_{n=1}^{\infty} \lambda_{n}^{\frac{1}{2}} \xi_{n} \psi_{n}(t).$ Now, when does such an expansion exist?

Because I look at signals on unbounded domains a lot, the case where a countable basis cannot be assumed to exist seems important.

## Gaussian

An extra-fun case, when this construction induces a Gaussian process because the $$\xi_{n}$$ are Gaussian. Very handy in Gaussian process regression.

## References

Adler, Robert J. 2010. The Geometry of Random Fields. SIAM ed. Philadelphia: Society for Industrial and Applied Mathematics.
Adler, Robert J., and Jonathan E. Taylor. 2007. Random Fields and Geometry. Springer Monographs in Mathematics 115. New York: Springer.
Adler, Robert J, Jonathan E Taylor, and Keith J Worsley. 2016. Applications of Random Fields and Geometry Draft.
Alexanderian, Alen. 2015. arXiv:1509.07526 [Math], October.
Battey, Heather, and Alessio Sancetta. 2013. Journal of Multivariate Analysis 120 (September): 1–17.
Calatayud Gregori, Julia, Benito M. Chen-Charpentier, Juan Carlos Cortés López, and Marc Jornet Sanz. 2019. Symmetry 11 (1): 43.
Ghanem, Roger G., and Pol D. Spanos. 2003. Stochastic Finite Elements: A Spectral Approach. New York, NY: Courier Corporation.
Ghanem, Roger, and John Red-Horse. 2017. In Handbook of Uncertainty Quantification, edited by Roger Ghanem, David Higdon, and Houman Owhadi, 521–51. Cham: Springer International Publishing.
Ghanem, Roger, and P. D. Spanos. 1990. Journal of Applied Mechanics 57 (1): 197–202.
Gratiet, Loïc Le, Stefano Marelli, and Bruno Sudret. 2016. In Handbook of Uncertainty Quantification, edited by Roger Ghanem, David Higdon, and Houman Owhadi, 1–37. Cham: Springer International Publishing.
Horváth, Lajos, and Piotr Kokoszka. 2012. Inference for functional data with applications. Vol. 200. Springer series in statistics. New York: Springer.
Kim, Kwang-Ki K., and Richard D. Braatz. 2013. International Journal of Control 86 (8): 1324–37.
Kim, Kwang-Ki K., Dongying Erin Shen, Zoltan K. Nagy, and Richard D. Braatz. 2013. IEEE Control Systems Magazine 33 (5): 58–67.
Lei, Huan, Jing Li, Peiyuan Gao, Panos Stinis, and Nathan Baker. 2018. April.
Liu, Chong, Surajit Ray, and Giles Hooker. 2014. arXiv:1411.4681 [Math, Stat], November.
Luo, Wuan. 2006. Phd, California Institute of Technology.
O’Hagan, Anthony. 2013. “Polynomial Chaos: A Tutorial and Critique from a Statistician’s Perspective,” 20.
Shang, Han Lin. 2014. AStA Advances in Statistical Analysis 98 (2): 121–42.
Wahba, Grace. 1990. Spline Models for Observational Data. SIAM.
Xiu, Dongbin. 2010. Numerical Methods for Stochastic Computations: A Spectral Method Approach. USA: Princeton University Press.
Xiu, Dongbin, and Jan S. Hesthaven. 2005. SIAM Journal on Scientific Computing 27 (3): 1118–39.
Yaglom, A. M. 1961. Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 2: Contributions to Probability Theory, January, 593–622.

### No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.