# Random fields as stochastic differential equations

Precision vs covariance, fight!

October 12, 2020 — March 1, 2021

\[ \renewcommand{\var}{\operatorname{Var}} \renewcommand{\dd}{\mathrm{d}} \renewcommand{\pd}{\partial} \renewcommand{\sinc}{\operatorname{sinc}} \]

The representation of certain random fields, especially Gaussian random fields as stochastic differential equations. This is the engine that makes filtering Gaussian processes go, and is also a natural framing for probabilistic spectral analysis.

I do not have much to say right now about this, but I am using it so watch this space.

## 1 Creating a stationary Markov SDE with desired covariance

The Gauss-Markov Random Field approach.

Warning: I’m taking crib notes for myself here, so I lazily switch between signal processing filter terminology and probabilist terminology. I assume Bochner’s and Yaglom’s Theorems as comprehensible methods for analysing covariance kernels.

Let’s start with stationary kernels. We consider an SDE \(f: \mathbb{R}\to\mathbb{R}\) at stationarity. We will let its driving noise be some Wiener process. We are concerned with deriving the parameters of the SDE such that it has a given stationary covariance function \(k\).

If there are no zeros in the spectral density, then there are no poles in the inverse transfer function, and we can model it with an all-pole SDE. This includes all the classic Matérn functions. This is covered in J. Hartikainen and Särkkä (2010), and Lindgren, Rue, and Lindström (2011). Worked examples starting from a discrete time formulation are given in a tutorial introduction Grigorievskiy and Karhunen (2016).

More generally, (quasi-)periodic covariances have zeros and we need to find a full rational function approximation. Särkkä, Solin, and Hartikainen (2013) introduces one such method. Bolin and Lindgren (2011) explores a slightly different class

Solin and Särkkä (2014) has a fancier method employing resonators a.k.a. filter banks, to address a concern of Steven Reece et al. (2014) that atomic spectral peaks in the Fourier transform are not well approximated by rational functions.

Bolin and Lindgren (2011) consider a general class of realisable systems, given by \[ \mathcal{L}_{1} X(\mathbf{s})=\mathcal{L}_{2} \mathcal{W}(\mathbf{s}) \] for some linear operators \(\mathcal{L}_{1}\) and \(\mathcal{L}_{2} .\)

In the case that \(\mathcal{L}_{1}\) and \(\mathcal{L}_{2}\) commute, this may be put in hierarchical form: \[\begin{aligned} \mathcal{L}_{1} X_{0}(\mathbf{s})&=\mathcal{W}(\mathbf{s})\\ X(\mathbf{s})&=\mathcal{L}_{2} X_{0}(\mathbf{s}). \end{aligned}\]

They explain

\(X(\mathbf{s})\) is simply \(\mathcal{L}_{2}\) applied to the solution one would get to if \(\mathcal{L}_{2}\) was the identity operator.

They call this a *nested* PDE, although AFAICT you could also say *ARMA*. They are particularly interested in equations of this form: \[
\left(\kappa^{2}-\Delta\right)^{\alpha / 2} X(\mathbf{s})=\left(b+\mathbf{B}^{\top} \nabla\right) \mathcal{W}(\mathbf{s})
\]

The SPDE generating this class of models is \[ \left(\prod_{i=1}^{n_{1}}\left(\kappa^{2}-\Delta\right)^{\alpha_{i} / 2}\right) X(\mathbf{s})=\left(\prod_{i=1}^{n_{2}}\left(b_{i}+\mathbf{B}_{i}^{\top} \nabla\right)\right) \mathcal{W}(\mathbf{s}) \]

They show that spectral density for such an \(X(\mathbf{s})\) is given by \[ S(\mathbf{k})=\frac{\phi^{2}}{(2 \pi)^{d}} \frac{\prod_{j=1}^{n_{2}}\left(b_{j}^{2}+\mathbf{k}^{\top} \mathbf{B}_{j} \mathbf{B}_{j}^{\top} \mathbf{k}\right)}{\prod_{j=1}^{n_{1}}\left(\kappa_{j}^{2}+\|\mathbf{k}\|^{2}\right)^{\alpha_{j}}}. \]

## 2 Convolution representations

See stochastic convolution or pragmatically, assume Gaussianity and see Gaussian convolution processes.

## 3 Covariance representation

Suppose there is a linear SDE on domain \(\mathbb{R}^d\) whose measure has the desired covariance structure, and ignore all questions of existence and convergence for now. We define terms of the driving noise \(\varepsilon\) and a linear differential operator \(\mathcal{L}\) such that \[ \mathcal{L}f(\mathbf{x})=\varepsilon(\mathbf{x}). \]

Assume there is a Green’s function for the PDE, i.e. that for any \(\mathbf{s} \in\mathbb{R}^d\) we may find a function \(G_\mathbf{s}(\mathbf{x})\) such that \[ \mathcal{L}G_\mathbf{s}(\mathbf{x})=\delta_\mathbf{s}(\mathbf{x}). \]

The solutions of the SDE, ignoring a whole bunch of existence stuff, are then given by the convolution of these Green’s functions with the driving noise, i.e. \(f(\mathbf{x}_p) \overset{\text{sorta}}{=}\int G_\mathbf{s}(\mathbf{x}_p)\varepsilon(\mathbf{s}) d \mathbf{s}.\) We use this to find the covariance of the solutions in terms of inner products of these fundamental solutions. \[\begin{align*} k(\mathbf{x}_p, \mathbf{x}_q) &=\mathbb{E}[f(\mathbf{x}_p)f(\mathbf{x}_q)] \\ &=\mathbb{E}\left[\int G_\mathbf{s}(\mathbf{x}_p)\varepsilon(\mathbf{s}) d \mathbf{s} \int G_\mathbf{t}(\mathbf{x}_q)\varepsilon(\mathbf{t}) d \mathbf{t} \right] \\ &=\mathbb{E}\left[\iint G_\mathbf{s}(\mathbf{x}_p) G_\mathbf{t}(\mathbf{x}_q) \varepsilon(\mathbf{s}) \varepsilon(\mathbf{t}) d \mathbf{t} d \mathbf{s} \right] \\ &=\iint G_\mathbf{s}(\mathbf{x}_p) G_\mathbf{t}(\mathbf{x}_q) \mathbb{E}[\varepsilon(\mathbf{s}) \varepsilon(\mathbf{t})] d \mathbf{t} d \mathbf{s} \\ &=\iint G_\mathbf{s}(\mathbf{x}_p) G_\mathbf{t}(\mathbf{x}_q) \sigma^2_\varepsilon \delta_\mathbf{s} (\mathbf{t}) d \mathbf{t} d \mathbf{s} &\text{ whiteness}\\ &=\sigma^2_\varepsilon \int G_\mathbf{s}(\mathbf{x}_p) G_\mathbf{s}(\mathbf{x}_q) d \mathbf{s}\\ &=\sigma^2_\varepsilon \langle G_\cdot(\mathbf{x}_p), G_\cdot(\mathbf{x}_q)\rangle \end{align*}\]

After that, the question is, given a Green’s function can you produce a linear operator that realises it?

For example, the arc-cosine kernel of order \(1\) corresponding to the ReLU is \[\begin{align*} k(\mathbf{x}_p, \mathbf{x}_q) &= \frac{\sigma_\varepsilon^2 \Vert \mathbf{x}_p \Vert \Vert \mathbf{x}_q \Vert }{2\pi} \Big( \sin |\theta| + \big(\pi - |\theta| \big) \cos\theta \Big) \end{align*}\] so for Green’s functions inducing this to exist we would want \[\begin{align} \int G_\mathbf{s}(\mathbf{x}_p) G_\mathbf{s}(\mathbf{x}_q) d \mathbf{s} &=\frac{\Vert \mathbf{x}_p \Vert \Vert \mathbf{x}_q \Vert }{2\pi} \Big( \sin |\theta| + \big(\pi - |\theta| \big) \cos\theta \Big) \end{align}\] For this to work we would need \(G_\mathbf{s}(\mathbf{x})\propto\Vert \mathbf{x} \Vert.\)

## 4 Input measures

Warning: this is just a dump of some notes from a paper I was writing; It does not make much sense RN. The essential idea I want to get at is considering different enveloping strategies for the SDE; Enveloping the input noise, for example

Suppose \(\mathbf{x}_p, \mathbf{x}_q \in \mathbb{R}^d\). The kernel satisfies \[\begin{aligned} k(\mathbf{x}_p, \mathbf{x}_q) = \sum_{j=1}^d \frac{\partial k}{\partial x_{pj}} x_{pj}. \end{aligned}\] Let \(f\) denote the Gaussian process with covariance function \(k\) and let \(\mathcal{F}_{\mu}[f]\) denote the Fourier transform of \(f\) with respect to the finite measure \(\mu\). Let the Fourier transform of \(\mu\) be denoted \(\mathcal{F}[\mu](\mathbf{\omega})=\int e^{-i \mathbf{\omega}^\top \mathbf{x}} \, \mu(\dd\mathbf{x})\), so that \(\mathcal{F}_{\mu}[f]=\mathcal{F}[f(x)\partial_x \mu(x)]=\mathcal{F}[f(x)]\ast\mathcal{F} [ \mu].\)

We have \[\begin{aligned} \mathbb{E} | \mathcal{F}_{\mu}[f](\mathbf{\omega})|^2 &= \iint \mathbb{E}\big[ f(\mathbf{x}_p) f(\mathbf{x}_q) \big]\, e^{-i\mathbf{\omega}^\top(\mathbf{x}_p - \mathbf{x}_q)} \mu(\dd\mathbf{x}_p) \mu(\dd\mathbf{x}_q) \\ &= \iint k(\mathbf{x}_p, \mathbf{x}_q) \, e^{-i\mathbf{\omega}^\top(\mathbf{x}_p - \mathbf{x}_q)}\,\mu(\dd\mathbf{x}_p) \mu(\dd\mathbf{x}_q) \\ &= \iint \sum_{j=1}^d \frac{\partial k}{\partial x_{pj}} x_{pj} e^{-i \mathbf{\omega}^\top \mathbf{x}_p} \mu(\dd\mathbf{x}_p) \, e^{i \mathbf{\omega}^\top \mathbf{x}_q} \,\mu(\dd\mathbf{x}_q) \\ &= \int \sum_{j=1}^d i \frac{\partial}{\partial \omega_j} \Bigg( \int \frac{\partial k}{\partial x_{pj}} e^{-i \mathbf{\omega}^\top \mathbf{x}_p} \, \mu(\dd\mathbf{x}_p)\Bigg) e^{i \mathbf{\omega}^\top \mathbf{x}_q} \mu(\dd\mathbf{x}_q)\\ &= \mathcal{F}_{\mu}^{\mathbf{x}_q} \left[ \sum_{j=1}^d i \frac{\partial}{\partial \omega_j} \Bigg( \int \frac{\partial k}{\partial x_{pj}} e^{-i \mathbf{\omega}^\top \mathbf{x}_p} \, \mu(\dd\mathbf{x}_p)\Bigg) \right]\\ &= \mathcal{F}_{\mu}^{\mathbf{x}_q} \left[ \sum_{j=1}^d i \frac{\partial}{\partial \omega_j} \Bigg( \mathcal{F}_{\mu}^{\mathbf{x}_p}\left[ \frac{\partial k}{\partial x_{pj}} \right]\Bigg) \right].\end{aligned}\] Then \[\begin{aligned} \mathbb{E} | \mathcal{F}_{\mu}[f](\mathbf{\omega})|^2 &= \int \sum_{j=1}^d i \frac{\partial}{\partial \omega_j} \Bigg( \int \frac{\partial k}{\partial x_{pj}} e^{-i \mathbf{\omega}^\top \mathbf{x}_p} \, \mu(\dd\mathbf{x}_p) \, \Bigg) e^{i \mathbf{\omega}^\top \mathbf{x}_q}\mu(\dd\mathbf{x}_q)\\ &= -\int \sum_{j=1}^d \frac{\partial}{\partial \omega_j} \Bigg( \omega_j \int k(\mathbf{x}_p, \mathbf{x}_q) e^{-i \mathbf{\omega}^\top \mathbf{x}_p} \, \mu(\dd\mathbf{x}_p) \, \Bigg)e^{i \mathbf{\omega}^\top \mathbf{x}_q}\mu(\dd\mathbf{x}_q) \\ &= -\sum_{j=1}^d \int \Bigg( \int k(\mathbf{x}_p, \mathbf{x}_q) e^{-i \mathbf{\omega}^\top \mathbf{x}_p} \, \mu(\dd\mathbf{x}_p) \, \Bigg) e^{i \mathbf{\omega}^\top \mathbf{x}_q}\mu(\dd\mathbf{x}_q)\\ &\phantom{{}={}}-\int \Bigg( \omega_j \frac{\partial}{\partial \omega_j} \int k(\mathbf{x}_p, \mathbf{x}_q) e^{-i \mathbf{\omega}^\top \mathbf{x}_p} \, \mu(\dd\mathbf{x}_p) \, \Bigg) e^{i \mathbf{\omega}^\top \mathbf{x}_q}\mu(\dd\mathbf{x}_q) \\ (d+1)\mathbb{E} | \mathcal{F}_{\mu}[f](\mathbf{\omega})|^2 &= -\int \Bigg( \omega_j \frac{\partial}{\partial \omega_j} \int k(\mathbf{x}_p, \mathbf{x}_q) e^{-i \mathbf{\omega}^\top \mathbf{x}_p} \, \mu(\dd\mathbf{x}_p) \, \Bigg) e^{i \mathbf{\omega}^\top \mathbf{x}_q}\mu(\dd\mathbf{x}_q) \end{aligned}\]

### 4.1 \(\mu\) is a hypercube

We assume that \(\mu\) is invariant with respect to permutation of coordinates. If we aren’t being silly, that means a Cartesian product of intervals \(I\), \(\mu(A):=\operatorname{Leb}(A\cap I^d).\) Let us go with \(I=[-1,1].\) Then \[\begin{aligned} \mathcal{F}[\mu](\mathbf{\omega}) &=\prod_{j=1}^d \sinc \left( \frac{\omega_j}{4\pi}\right)\\ &=\prod_{j=1}^d \frac{\sin (\omega_j/2)}{\omega}\end{aligned}\] Also \[\begin{aligned} \sinc'x &=\frac{\cos \pi x - \sinc x}{x}.\end{aligned}\]

### 4.2 \(\mu\) is the unit sphere

TBD.

### 4.3 \(\mu\) is an isotropic Gaussian

Suppose \(\mu\) is an isotropic Gaussian of variance \(I\sigma^2\) so that \(\dd \mu(\mathbf{x})=(2\pi)^{-d/2}\sigma^{-d}e^{-\sigma^2\mathbf{x}^\top\mathbf{x}/2}\) and \(\mathcal{F}[\mu]=e^{-\sigma^2\mathbf{\omega}^\top\mathbf{\omega}/2}=(2\pi)^{-d/2}\sigma^{-d}\dd \mu(\mathbf{\omega}).\)

## 5 Without stationarity via Green’s functions

Suppose our SDE may be specified in terms of a Gaussian white driving noise with variance \(\sigma_w^2\) and an impulse response function/Green’s function, \(g\) such that \[\begin{aligned} f(x):=\int g(\mathbf{u},\mathbf{x})\dd w(\mathbf{u}).\end{aligned}\] We know that the kernel is an inner product kernel and therefore invariant to rotation about \(\mathbf{0},\) i.e. for orthogonal \(Q\), \(k(Q\mathbf{x}_p, Q\mathbf{x}_q)=k(\mathbf{x}_p, \mathbf{x}_q).\) It follows that \(g(Q\mathbf{u}, Q\mathbf{x})=g(\mathbf{u}, \mathbf{x}).\) In fact, we may write each in dot-product form, i.e. \(k(\mathbf{x}_p, \mathbf{x}_q)=k(\mathbf{x}_p\cdot \mathbf{x}_q)\) and \(g(\mathbf{u}, \mathbf{x})=g(\mathbf{u}\cdot \mathbf{x}).\) The kernel satisfies \[\begin{aligned} k(\mathbf{x}_p, \mathbf{x}_q) &= \mathbb{E}\left[\int g(\mathbf{u},\mathbf{x}_p)\dd w(\mathbf{u})\int g(\mathbf{v},\mathbf{x}_q)\dd w(\mathbf{v})\right]\\ &= \mathbb{E}\left[\iint g(\mathbf{u},\mathbf{x}_p) g(\mathbf{v},\mathbf{x}_q)\dd w(\mathbf{u})\dd w(\mathbf{v})\right]\\ &= \iint g(\mathbf{u},\mathbf{x}_p) g(\mathbf{v},\mathbf{x}_q) \sigma_w^2\delta(\mathbf{u},\mathbf{v})\dd \mathbf{v}\dd \mathbf{u}\\ &= \sigma_w^2\int g(\mathbf{u},\mathbf{x}_p) g(\mathbf{u},\mathbf{x}_q) \dd\mathbf{u}\end{aligned}\] Up to a scaling factor, the Green’s function is simply the covariance kernel under the assumption that the driving noise is white.

Recalling \(k(\mathbf{x}_p, \mathbf{x}_q) = \mathbb{E}\big[ \psi(\mathbf{W}^\top \mathbf{x}_q) \psi(\mathbf{W}^\top \mathbf{x}_p) \big]= \mathbb{E}\big[ \psi(Z_p) \psi(Z_q) \big]\) the Green’s function thus must satisfy \[\begin{aligned} \sigma_w^2\int g(\mathbf{u}\cdot\mathbf{x}_p) g(\mathbf{u}\cdot \mathbf{x}_q) \dd\mathbf{u} &= \mathbb{E}\big[ \psi(\mathbf{W}^\top \mathbf{x}_q) \psi(\mathbf{W}^\top \mathbf{x}_p) \big]\end{aligned}\] Now we need to see how this works for individual kernels.

## 6 References

*IEEE Transactions on Automatic Control*.

*IEEE Transactions on Pattern Analysis and Machine Intelligence*.

*Mathematical System Theory: The Influence of R. E. Kalman*.

*WIREs Computational Statistics*.

*Minimal Factorization of Matrix and Operator Functions*. Operator Theory, Advances and Applications, v. 1.

*arXiv:2002.07928 [Physics, Stat]*.

*Scandinavian Journal of Statistics*.

*Journal of Computational and Graphical Statistics*.

*The Annals of Applied Statistics*.

*arXiv:2006.10160 [Cs, Stat]*.

*arXiv:1802.08167 [Stat]*.

*MLSP*.

*SIAM Journal on Control*.

*arXiv:2105.04504 [Cs, Stat]*.

*IEEE Transactions on Information Theory*.

*IEEE Transactions on Information Theory*.

*Communications in Mathematics and Statistics*.

*1975 IEEE Conference on Decision and Control Including the 14th Symposium on Adaptive Processes*.

*IEEE Transactions on Automatic Control*.

*2016 International Joint Conference on Neural Networks (IJCNN)*.

*arXiv:1610.08035 [Stat]*.

*2010 IEEE International Workshop on Machine Learning for Signal Processing*.

*Proceedings of the Twenty-Seventh Conference on Uncertainty in Artificial Intelligence*. UAI’11.

*Proceedings of the 29th International Coference on International Conference on Machine Learning*. ICML’12.

*Environmental and Ecological Statistics*.

*Quantitative Methods for Current Environmental Issues*.

*arXiv:1906.00286 [Stat]*.

*Pattern Recognition Letters*.

*WIREs Computational Statistics*.

*The Annals of Mathematical Statistics*.

*IEEE Transactions on Information Theory*.

*1971 IEEE Conference on Decision and Control*.

*IEEE Transactions on Information Theory*.

*IEEE Transactions on Information Theory*.

*IEEE Transactions on Automatic Control*.

*IEEE Transactions on Automatic Control*.

*IEEE Transactions on Information Theory*.

*IEEE Transactions on Information Theory*.

*2016 IEEE 26th International Workshop on Machine Learning for Signal Processing (MLSP)*.

*Statistical Modelling*.

*arXiv:2111.01084 [Stat]*.

*Journal of Statistical Software*.

*Journal of the Royal Statistical Society: Series B (Statistical Methodology)*.

*IEEE Transactions on Information Theory*.

*1975 IEEE Conference on Decision and Control Including the 14th Symposium on Adaptive Processes*.

*WIREs Computational Statistics*.

*Pacific Journal of Mathematics*.

*arXiv:math/0701323*.

*arXiv.org*.

*The Journal of Machine Learning Research*.

*2010 13th International Conference on Information Fusion*.

*Scandinavian Journal of Statistics*.

*Artificial Neural Networks and Machine Learning – ICANN 2011*. Lecture Notes in Computer Science.

*IEEE Transactions on Automatic Control*.

*2014 IEEE International Workshop on Machine Learning for Signal Processing (MLSP)*.

*Applied Stochastic Differential Equations*. Institute of Mathematical Statistics Textbooks 10.

*IEEE Signal Processing Magazine*.

*arXiv:1703.02112 [Stat]*.

*IEEE Transactions on Information Theory*.

*IEEE Transactions on Information Theory*.

*Journal of Statistical Software*.

*Journal of the Royal Statistical Society: Series B (Statistical Methodology)*.

*Physical Review E*.

*Artificial Intelligence and Statistics*.

*Statistics and Computing*.

*Proceedings of the AAAI Conference on Artificial Intelligence*.

*The Annals of Statistics*.

*1974 IEEE Conference on Decision and Control Including the 13th Symposium on Adaptive Processes*.

*Bulletin of the International Statistical Institute*.

*ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*.

*Proceedings of the 37th International Conference on Machine Learning*.

*Journal of Machine Learning Research*.

*Biometrika*.

*Correlation Theory of Stationary and Related Random Functions. Volume II: Supplementary Notes and References*. Springer Series in Statistics.

*Correlation Theory of Stationary and Related Random Functions Volume I*.

*An Introduction to the Theory of Stationary Random Functions*.