Reproducing kernels satisfying physical equations

1 Gaussian Processes and Their Derivatives

An important property of Gaussian processes is that linear transformations of GPs remain GPs under broad circumstances. Specifically, the derivative of a GP is also a GP, provided the covariance function is sufficiently smooth. If $f$ is a GP with mean function $m(x)$ and covariance function $k(x,x^{\prime })$, then its derivative $f^{\prime }$ is a GP with mean $m^{\prime }(x)$ and covariance $k^{\prime }(x,x^{\prime })$, where:

$$k^{\prime }(x,x^{\prime })=\frac{{\partial }^{2}k(x,x^{\prime })}{\partial x\partial x^{\prime }}$$

This property allows us to incorporate differential operators into the GP framework, enabling us to encode PDE constraints directly into the kernel.

2 Latent Force Models

Latent force models are one of the earlier methods for integrating differential equations into GP regression (M. Álvarez, Luengo, and Lawrence 2009; M. A. Álvarez, Luengo, and Lawrence 2013; Moss et al. 2022). In LFMs, the idea is to model the unknown latent forces driving a system using GPs. These latent forces are then connected to the observed data through differential equations.

e.g. Consider a system governed by a linear ordinary differential equation (ODE):

$$\frac{df(t)}{dt}+af(t)=u(t)$$

Here, $f(t)$ is the observed function, $a$ is a known constant, and $u(t)$ is an unknown latent function modelled as a GP. By placing a GP prior on $u(t)$, we induce a GP prior on $f(t)$ that inherently satisfies the ODE.

The function $f(t)$ resides in the Sobolev space $W^{1,2}([0,T])$, which consists of functions whose first derivative is square-integrable over the interval ([0, T]).

3 Divergence-Free and Curl-Free Kernels

Some fun tricks are of special relevance to fluids; e.g. for kernels which imply divergence-free or curl-free fields, especially on the surface of a sphere (Narcowich, Ward, and Wright 2007; E. J. Fuselier, Shankar, and Wright 2016; E. J. Fuselier and Wright 2009).

E. Fuselier (2008) says:

Constructing divergence-free and curl-free matrix-valued RBFs is fairly simple. If $\varphi$ is a scalar-valued function consider

$$\begin{array}{rl}{\mathrm{\Phi }}_{\text{div }}& :=(-\mathrm{\Delta }I+\mathrm{\nabla }{\mathrm{\nabla }}^T)\varphi ,\\ {\mathrm{\Phi }}_{curl}& :=-\mathrm{\nabla }{\mathrm{\nabla }}^T\varphi .\end{array}$$

If $\varphi$ is an RBF, then these functions can be used to produce divergence-free and curl-free interpolants, respectively. We note that these are not radial functions, but because they are usually generated by an RBF $\varphi$, they are still commonly called “matrix-valued RBFs”.

AFAICT there is nothing RBF-specific; I think it works for any stationary kernel. Do we even need stationarity?

The functions produced by these kernels seem to reside in specific Sobolev spaces that respect the divergence-free or curl-free conditions. For instance, divergence-free vector fields in ${\mathbb{R}}^3$ belong to the space:

$${\mathbf{H}}_{\text{div}}=\{\mathbf{f}\in [L^2(\mathrm{\Omega }){]}^3:\mathrm{\nabla }\cdot \mathbf{f}=0\}$$

4 Linearly-constrained, Operator-Valued Kernels

Operator-valued kernels extend the concept of scalar kernels to vector or function outputs. They are particularly handy when the physical constraints can be expressed as linear operators acting on functions (Lange-Hegermann 2018, 2021)

Consider a linear operator $\mathcal{L}$ acting on a function $f$. An operator-valued kernel $K(x,x^{\prime })$ can be designed such that:

$${\mathcal{L}}_{x}K(x,x^{\prime })=\delta (x-x^{\prime })$$

where $\delta$ is the Dirac delta function. This approach ensures that functions drawn from the associated RKHS satisfy $\mathcal{L}f=0$.

5 Physics-Informed Gaussian Processes

Physics-informed Gaussian processes (PIGPs) incorporate physical laws by modifying the GP’s prior or likelihood to enforce PDE constraints (Raissi and Karniadakis 2018). This can be done by penalizing deviations from the PDE in the loss function or by directly incorporating the differential operators into the kernel.

For a function $f(x)$ that should satisfy $\mathcal{L}f(x)=0$, the GP prior can be adjusted such that:

$$\text{Cov}(\mathcal{L}f(x),\mathcal{L}f(x^{\prime }))=k_{\mathcal{L}}(x,x^{\prime })$$

where $k_{\mathcal{L}}$ is a covariance function constructed to reflect the operator $\mathcal{L}$.

More at (Perdikaris et al. 2017; Raissi and Karniadakis 2018; Raissi, Perdikaris, and Karniadakis 2017a, 2017b, 2018)

6 Implicit

Not quite sure what to call it, but Kian Ming A. Chai introduced us to Mora et al. (2024), which seems to be an interesting variant. Keyword match to Brouard, Szafranski, and D’Alché-Buc (2016).

7 References