Karhunen-Loève expansions

As seen in chaos expansions kind-of, and used in low rank kriging, and assumed in PCA and implied by many basis decompositions etc.

Suppose we have a collection \(\{\varphi_n\}\) of real valued functions on our index space \(T\), and a collection \(\{\xi_n\}\) of uncorrelated random variables. Now we define the random process \[ f(t)=\sum_{n=1}^{\infty} \xi_{n} \varphi_{n}(t). \] We might care about the first two moments of \(f,\) i.e. \[ \mathbb{E}\{f(s) f(t)\}=\sum_{n=1}^{\infty} \sigma_{n}^{2} \varphi_{n}(s) \varphi_{n}(t) \] and variance function \[ \mathbb{E}\left\{f^{2}(t)\right\}=\sum_{n=1}^{\infty} \sigma_{n}^{2} \varphi_{n}^{2}(t) \]

Now suppose that we have a stochastic process where the index \(T\) is a compact domain in \(\mathbb{R}^{N}\). The corresponding expansion of \(f\) in the above form is known as the Karhunen-Loève expansion. Suppose that \(f\) has covariance function \(K\) and define an operator \(\mathcal{K}\), taking the space of square integrable functions on \(T\) to itself, by \[ (\mathcal{K} \psi)(t)=\int_{T} K(s, t) \psi(s) d s \] Suppose that \(\lambda_{1} \geq \lambda_{2} \geq \ldots\), and \(\psi_{1}, \psi_{2}, \ldots\), are, respectively, the (ordered) eigenvalues and normalised eigenfunctions of the operator. That is, the \(\lambda_{n}\) and \(\psi_{n}\) solve the integral equation \[ \int_{T} K(s, t) \psi(s) d s=\lambda \psi(t) \] with the normalisation \[ \int_{T} \psi_{n}(t) \psi_{m}(t) d t=\left\{\begin{array}{ll} 1 & n=m \\ 0 & n \neq m \end{array}\right. \] These eigenfunctions lead to a natural expansion of \(K\), known as Mercer’s Theorem, which states that \[ K(s, t)=\sum_{n=1}^{\infty} \lambda_{n} \psi_{n}(s) \psi_{n}(t) \] where the series converges absolutely and uniformly on \(T \times T\). The Karhunen-Loève expansion of \(f\) is obtained by setting \(\varphi_{n}=\) \(\lambda_{n}^{\frac{1}{2}} \psi_{n},\) so that \[ f_{t}=\sum_{n=1}^{\infty} \lambda_{n}^{\frac{1}{2}} \xi_{n} \psi_{n}(t). \] Now, when does such an expansion exist?

Because I look at signals on unbounded domains a lot, the case where a countable basis cannot be assumed to exist seems important.


An extra-fun case, when this construction induces a Gaussian process because the \(\xi_{n}\) are Gaussian. Very handy in Gaussian process regression.


Adler, Robert J. 2010. The Geometry of Random Fields. SIAM ed. Philadelphia: Society for Industrial and Applied Mathematics.
Adler, Robert J., and Jonathan E. Taylor. 2007. Random Fields and Geometry. Springer Monographs in Mathematics 115. New York: Springer.
Adler, Robert J, Jonathan E Taylor, and Keith J Worsley. 2016. Applications of Random Fields and Geometry Draft.
Alexanderian, Alen. 2015. A Brief Note on the Karhunen-Loève Expansion.” arXiv:1509.07526 [Math], October.
Battey, Heather, and Alessio Sancetta. 2013. Conditional Estimation for Dependent Functional Data.” Journal of Multivariate Analysis 120 (September): 1–17.
Calatayud Gregori, Julia, Benito M. Chen-Charpentier, Juan Carlos Cortés López, and Marc Jornet Sanz. 2019. Combining Polynomial Chaos Expansions and the Random Variable Transformation Technique to Approximate the Density Function of Stochastic Problems, Including Some Epidemiological Models.” Symmetry 11 (1): 43.
Ghanem, Roger G., and Pol D. Spanos. 2003. Stochastic Finite Elements: A Spectral Approach. New York, NY: Courier Corporation.
Ghanem, Roger, and John Red-Horse. 2017. Polynomial Chaos: Modeling, Estimation, and Approximation.” In Handbook of Uncertainty Quantification, edited by Roger Ghanem, David Higdon, and Houman Owhadi, 521–51. Cham: Springer International Publishing.
Ghanem, Roger, and P. D. Spanos. 1990. Polynomial Chaos in Stochastic Finite Elements.” Journal of Applied Mechanics 57 (1): 197–202.
Gratiet, Loïc Le, Stefano Marelli, and Bruno Sudret. 2016. Metamodel-Based Sensitivity Analysis: Polynomial Chaos Expansions and Gaussian Processes.” In Handbook of Uncertainty Quantification, edited by Roger Ghanem, David Higdon, and Houman Owhadi, 1–37. Cham: Springer International Publishing.
Horváth, Lajos, and Piotr Kokoszka. 2012. Inference for functional data with applications. Vol. 200. Springer series in statistics. New York: Springer.
Kim, Kwang-Ki K., and Richard D. Braatz. 2013. Generalised Polynomial Chaos Expansion Approaches to Approximate Stochastic Model Predictive Control .” International Journal of Control 86 (8): 1324–37.
Kim, Kwang-Ki K., Dongying Erin Shen, Zoltan K. Nagy, and Richard D. Braatz. 2013. Wiener’s Polynomial Chaos for the Analysis and Control of Nonlinear Dynamical Systems with Probabilistic Uncertainties [Historical Perspectives].” IEEE Control Systems Magazine 33 (5): 58–67.
Lei, Huan, Jing Li, Peiyuan Gao, Panos Stinis, and Nathan Baker. 2018. A Data-Driven Framework for Sparsity-Enhanced Surrogates with Arbitrary Mutually Dependent Randomness,” April.
Liu, Chong, Surajit Ray, and Giles Hooker. 2014. Functional Principal Components Analysis of Spatially Correlated Data.” arXiv:1411.4681 [Math, Stat], November.
Luo, Wuan. 2006. Wiener Chaos Expansion and Numerical Solutions of Stochastic Partial Differential Equations.” Phd, California Institute of Technology.
O’Hagan, Anthony. 2013. “Polynomial Chaos: A Tutorial and Critique from a Statistician’s Perspective,” 20.
Shang, Han Lin. 2014. A Survey of Functional Principal Component Analysis.” AStA Advances in Statistical Analysis 98 (2): 121–42.
Wahba, Grace. 1990. Spline Models for Observational Data. SIAM.
Xiu, Dongbin. 2010. Numerical Methods for Stochastic Computations: A Spectral Method Approach. USA: Princeton University Press.
Xiu, Dongbin, and Jan S. Hesthaven. 2005. High-Order Collocation Methods for Differential Equations with Random Inputs.” SIAM Journal on Scientific Computing 27 (3): 1118–39.
Yaglom, A. M. 1961. Second-Order Homogeneous Random Fields.” Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 2: Contributions to Probability Theory, January, 593–622.

No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.