Warping of stationary stochastic processes

Transforming stationary processes into non-stationary ones by transforming their inputs (Sampson and Guttorp 1992; Genton 2001; Genton and Perrin 2004; Perrin and Senoussi 1999, 2000).

This is of interest in the context of composing kernels to have known desirable properties by known transforms, and also learning (somewhat) arbitrary transforms to attain stationarity.

One might consider instead processes that are stationary upon a manifold.

Stationary reducible kernels

The main idea is to find a new feature space where stationarity (Sampson and Guttorp 1992) or local stationarity (Perrin and Senoussi 1999, 2000; Genton and Perrin 2004) can be achieved.

Genton (2001) summarises:

We say that a nonstationary kernel \(K(\mathbf{x}, \mathbf{z})\) is stationary reducible if there exist a bijective deformation \(\Phi\) such that: \[ K(\mathbf{x}, \mathbf{z})=K_{S}^{*}(\mathbf{\Phi}(\mathbf{x})-\mathbf{\Phi}(\mathbf{z})) \] where \(K_{S}^{*}\) is a stationary kernel.

Classic deformations

MacKay warping

As a function of input

Invented apparently by Gibbs (1998) and generalised in Paciorek and Schervish (2003).

Let \(k_S\) be some stationary kernel on \(\mathbb{R}^D.\) Let \(\Sigma(\mathbf{x})\) be a \(D \times D\) matrix-valued function which is positive definite for all \(\mathbf{x},\) and let \(\Sigma_{i} \triangleq \Sigma\left(\mathbf{x}_{i}\right) .\) ) Then define \[ Q_{i j}=\left(\mathbf{x}_{i}-\mathbf{x}_{j}\right)^{\top}\left(\left(\Sigma_{i}+\Sigma_{j}\right) / 2\right)^{-1}\left(\mathbf{x}_{i}-\mathbf{x}_{j}\right) \] Then \[ k_{\mathrm{NS}}\left(\mathbf{x}_{i}, \mathbf{x}_{j}\right)=2^{D / 2}\left|\Sigma_{i}\right|^{1 / 4}\left|\Sigma_{j}\right|^{1 / 4}\left|\Sigma_{i}+\Sigma_{j}\right|^{-1 / 2} k_{\mathrm{S}}\left(\sqrt{Q_{i j}}\right) \] is a valid non-stationary covariance function.

Homework question: Is this a product of convolutional gaussian processes.

Learning transforms


Anderes, Ethan B., and Michael L. Stein. 2008. β€œEstimating Deformations of Isotropic Gaussian Random Fields on the Plane.” The Annals of Statistics 36 (2): 719–41.
Anderes, Ethan, and Sourav Chatterjee. 2009. β€œConsistent Estimates of Deformed Isotropic Gaussian Random Fields on the Plane.” The Annals of Statistics 37 (5A).
Belkin, Mikhail, Siyuan Ma, and Soumik Mandal. 2018. β€œTo Understand Deep Learning We Need to Understand Kernel Learning.” In International Conference on Machine Learning, 541–49.
Bohn, Bastian, Michael Griebel, and Christian Rieger. 2018. β€œA Representer Theorem for Deep Kernel Learning.” arXiv:1709.10441 [Cs, Math], June.
Damian, Doris, Paul D. Sampson, and Peter Guttorp. 2001. β€œBayesian Estimation of Semi-Parametric Non-Stationary Spatial Covariance Structures.” Environmetrics 12 (2): 161–78.
Feragen, Aasa, and SΓΈren Hauberg. 2016. β€œOpen Problem: Kernel Methods on Manifolds and Metric Spaces. What Is the Probability of a Positive Definite Geodesic Exponential Kernel?” In Conference on Learning Theory, 1647–50. PMLR.
Genton, Marc G. 2001. β€œClasses of Kernels for Machine Learning: A Statistics Perspective.” Journal of Machine Learning Research 2 (December): 299–312.
Genton, Marc G., and Olivier Perrin. 2004. β€œOn a Time Deformation Reducing Nonstationary Stochastic Processes to Local Stationarity.” Journal of Applied Probability 41 (1): 236–49.
Gibbs, M. N. 1998. β€œBayesian Gaussian processes for regression and classification.” Ph.D., University of Cambridge.
Hinton, Geoffrey E, and Ruslan R Salakhutdinov. 2008. β€œUsing Deep Belief Nets to Learn Covariance Kernels for Gaussian Processes.” In Advances in Neural Information Processing Systems 20, edited by J. C. Platt, D. Koller, Y. Singer, and S. T. Roweis, 1249–56. Curran Associates, Inc.
Ikeda, Masahiro, Isao Ishikawa, and Yoshihiro Sawano. 2021. β€œComposition Operators on Reproducing Kernel Hilbert Spaces with Analytic Positive Definite Functions.” arXiv:1911.11992 [Math, Stat], March.
Paciorek, Christopher J., and Mark J. Schervish. 2003. β€œNonstationary Covariance Functions for Gaussian Process Regression.” In Proceedings of the 16th International Conference on Neural Information Processing Systems, 16:273–80. NIPS’03. Cambridge, MA, USA: MIT Press.
Perrin, Olivier, and Rachid Senoussi. 1999. β€œReducing Non-Stationary Stochastic Processes to Stationarity by a Time Deformation.” Statistics & Probability Letters 43 (4): 393–97.
β€”β€”β€”. 2000. β€œReducing Non-Stationary Random Fields to Stationarity and Isotropy Using a Space Deformation.” Statistics & Probability Letters 48 (1): 23–32.
Rasmussen, Carl Edward, and Christopher K. I. Williams. 2006. Gaussian Processes for Machine Learning. Adaptive Computation and Machine Learning. Cambridge, Mass: MIT Press.
Sampson, Paul D., and Peter Guttorp. 1992. β€œNonparametric Estimation of Nonstationary Spatial Covariance Structure.” Journal of the American Statistical Association 87 (417): 108–19.
Schmidt, Alexandra M., and Anthony O’Hagan. 2003. β€œBayesian Inference for Non-Stationary Spatial Covariance Structure via Spatial Deformations.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 65 (3): 743–58.
Shimotsu, Katsumi, and Peter C. B. Phillips. 2004. β€œLocal Whittle Estimation in Nonstationary and Unit Root Cases.” The Annals of Statistics 32 (2): 656–92.
Snoek, Jasper, Kevin Swersky, Rich Zemel, and Ryan Adams. 2014. β€œInput Warping for Bayesian Optimization of Non-Stationary Functions.” In Proceedings of the 31st International Conference on Machine Learning (ICML-14), 1674–82.
Tompkins, Anthony, and Fabio Ramos. 2018. β€œFourier Feature Approximations for Periodic Kernels in Time-Series Modelling.” Proceedings of the AAAI Conference on Artificial Intelligence 32 (1).
Vu, Quan, Andrew Zammit-Mangion, and Noel Cressie. 2020. β€œModeling Nonstationary and Asymmetric Multivariate Spatial Covariances via Deformations,” April.
Wilson, Andrew Gordon, Zhiting Hu, Ruslan Salakhutdinov, and Eric P. Xing. 2016. β€œDeep Kernel Learning.” In Artificial Intelligence and Statistics, 370–78. PMLR.
Zammit-Mangion, Andrew, Tin Lok James Ng, Quan Vu, and Maurizio Filippone. 2021. β€œDeep Compositional Spatial Models.” Journal of the American Statistical Association 0 (0): 1–22.

No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.