Karhunen-Loève expansions

September 16, 2019 — February 1, 2022

functional analysis
Hilbert space
kernel tricks
signal processing
stochastic processes

As seen in chaos expansions kind-of, and used in low rank kriging, and assumed in PCA and implied by many basis decompositions etc.

Figure 1

Suppose we have a collection \(\{\varphi_n\}\) of real valued functions on our index space \(T\), and a collection \(\{\xi_n\}\) of uncorrelated random variables. Now we define the random process \[ f(t)=\sum_{n=1}^{\infty} \xi_{n} \varphi_{n}(t). \] We might care about the first two moments of \(f,\) i.e. \[ \mathbb{E}\{f(s) f(t)\}=\sum_{n=1}^{\infty} \sigma_{n}^{2} \varphi_{n}(s) \varphi_{n}(t) \] and variance function \[ \mathbb{E}\left\{f^{2}(t)\right\}=\sum_{n=1}^{\infty} \sigma_{n}^{2} \varphi_{n}^{2}(t) \]

Now suppose that we have a stochastic process where the index \(T\) is a compact domain in \(\mathbb{R}^{N}\). The corresponding expansion of \(f\) in the above form is known as the Karhunen-Loève expansion. Suppose that \(f\) has covariance function \(K\) and define an operator \(\mathcal{K}\), taking the space of square integrable functions on \(T\) to itself, by \[ (\mathcal{K} \psi)(t)=\int_{T} K(s, t) \psi(s) d s \] Suppose that \(\lambda_{1} \geq \lambda_{2} \geq \ldots\), and \(\psi_{1}, \psi_{2}, \ldots\), are, respectively, the (ordered) eigenvalues and normalised eigenfunctions of the operator. That is, the \(\lambda_{n}\) and \(\psi_{n}\) solve the integral equation \[ \int_{T} K(s, t) \psi(s) d s=\lambda \psi(t) \] with the normalisation \[ \int_{T} \psi_{n}(t) \psi_{m}(t) d t=\left\{\begin{array}{ll} 1 & n=m \\ 0 & n \neq m \end{array}\right. \] These eigenfunctions lead to a natural expansion of \(K\), known as Mercer’s Theorem, which states that \[ K(s, t)=\sum_{n=1}^{\infty} \lambda_{n} \psi_{n}(s) \psi_{n}(t) \] where the series converges absolutely and uniformly on \(T \times T\). The Karhunen-Loève expansion of \(f\) is obtained by setting \(\varphi_{n}=\) \(\lambda_{n}^{\frac{1}{2}} \psi_{n},\) so that \[ f_{t}=\sum_{n=1}^{\infty} \lambda_{n}^{\frac{1}{2}} \xi_{n} \psi_{n}(t). \] Now, when does such an expansion exist?

Because I look at signals on unbounded domains a lot, the case where a countable basis cannot be assumed to exist seems important.

1 Gaussian

An extra-fun case, when this construction induces a Gaussian process because the \(\xi_{n}\) are Gaussian. Very handy in Gaussian process regression.

2 References

Adler, Robert J. 2010. The Geometry of Random Fields.
Adler, Robert J., and Taylor. 2007. Random Fields and Geometry. Springer Monographs in Mathematics 115.
Adler, Robert J, Taylor, and Worsley. 2016. Applications of Random Fields and Geometry Draft.
Alexanderian. 2015. A Brief Note on the Karhunen-Loève Expansion.” arXiv:1509.07526 [Math].
Battey, and Sancetta. 2013. Conditional Estimation for Dependent Functional Data.” Journal of Multivariate Analysis.
Calatayud Gregori, Chen-Charpentier, Cortés López, et al. 2019. Combining Polynomial Chaos Expansions and the Random Variable Transformation Technique to Approximate the Density Function of Stochastic Problems, Including Some Epidemiological Models.” Symmetry.
Ghanem, Roger, and Red-Horse. 2017. Polynomial Chaos: Modeling, Estimation, and Approximation.” In Handbook of Uncertainty Quantification.
Ghanem, Roger, and Spanos. 1990. Polynomial Chaos in Stochastic Finite Elements.” Journal of Applied Mechanics.
Ghanem, Roger G., and Spanos. 2003. Stochastic Finite Elements: A Spectral Approach.
Gratiet, Marelli, and Sudret. 2016. Metamodel-Based Sensitivity Analysis: Polynomial Chaos Expansions and Gaussian Processes.” In Handbook of Uncertainty Quantification.
Horváth, and Kokoszka. 2012. Inference for functional data with applications. Springer series in statistics.
Kim, and Braatz. 2013. Generalised Polynomial Chaos Expansion Approaches to Approximate Stochastic Model Predictive Control .” International Journal of Control.
Kim, Shen, Nagy, et al. 2013. Wiener’s Polynomial Chaos for the Analysis and Control of Nonlinear Dynamical Systems with Probabilistic Uncertainties [Historical Perspectives].” IEEE Control Systems Magazine.
Lei, Li, Gao, et al. 2018. A Data-Driven Framework for Sparsity-Enhanced Surrogates with Arbitrary Mutually Dependent Randomness.”
Liu, Ray, and Hooker. 2014. Functional Principal Components Analysis of Spatially Correlated Data.” arXiv:1411.4681 [Math, Stat].
Luo. 2006. Wiener Chaos Expansion and Numerical Solutions of Stochastic Partial Differential Equations.”
O’Hagan. 2013. “Polynomial Chaos: A Tutorial and Critique from a Statistician’s Perspective.”
Shang. 2014. A Survey of Functional Principal Component Analysis.” AStA Advances in Statistical Analysis.
Wahba. 1990. Spline Models for Observational Data.
Xiu. 2010. Numerical Methods for Stochastic Computations: A Spectral Method Approach.
Xiu, and Hesthaven. 2005. High-Order Collocation Methods for Differential Equations with Random Inputs.” SIAM Journal on Scientific Computing.
Yaglom. 1961. Second-Order Homogeneous Random Fields.” Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 2: Contributions to Probability Theory.