# Machine learning for partial differential equations

$$\newcommand{\solop}{\mathcal{G}^{\dagger}}$$

Using statistical or machine learning approaches to solve PDEs, and maybe even to perform inference through them. There are many approaches to ML learning of PDEs and I will document on an ad hoc basis as I need them. No claim is made to completeness.

TODO: To reduce proliferation of unclear symbols by introducing a specific example; which neural nets represent operators, which represent specific functions, between which spaces etc.

TODO: Harmonise the notation used in this section with subsections below; right now they match the papers’ notation but not each other.

TODO: should the intro section actually be filed under PDEs?

TODO: introduce a consistent notation for coordinate space, output spaces, and function space?

TODO: this is mostly Eulerian fluid flow models right now. Can we mention Lagrangian models at least?

## Background

Suppose we have a PDE defined over some input domain, which we presume is a time dimension and some number of spatial dimensions. The PDE is specified by some differential operator $$\mathcal{D}$$ and some forcing or boundary condition $$u\in \mathscr{U},$$ as $\mathcal{D}[f]=u.$ These functions will map from some coordinate space $$C$$ to some output space $$O$$. At the moment we consider only compact sets of positive Lebesgue measure, $$C\subseteq\mathbb{R}^{d_C}$$ and $$O\subseteq\mathbb{R}^{d_O}.$$ The first coordinate of the input space often has the special interpretation as time $$t\in \mathbb{R}$$ and the subsequent coordinates are then spatial coordinate $$x\in D\subseteq \mathbb{R}^{d_{D}}$$ where $$d_{D}=d_{C}-1.$$ Sometimes we make this explicit by writing the time coordinate separately as $$f(t,x).$$ A common case, concretely, is $$C=\mathbb{R} \times \mathbb{R}^2=\mathbb{R} \times D$$ and $$O=\mathbb{R}.$$ For each time $$t\in \mathbb{R}$$ we assume the instantaneous solution $$f(t, \cdot)$$ to be an element of some Banach space $$f\in \mathscr{A}$$ of functions $$f(t, \cdot): D\to O.$$ The overall solutions $$f: C\to O$$ have their own Banach space $$\mathscr{F}$$. More particularly, we might consider solutions a restricted time domain $$t\in [0,T]$$ and some spatial domain $$D\subseteq \mathbb{R}^2$$ where a solution is a function $$f$$ that maps $$[0,T] \times D \to \mathbb{R}.$$ This would naturally model, say, a 2D height-field evolving over time.

We have thrown the term Banach space about without making it clear which one we mean. There are usually some implied smoothness properties and of course we would want to include some kind of metric to fully specify these spaces, but we gloss over that for now.

We have introduced one operator, the defining operator $$\mathcal{D}$$ . Another that we think about a lot is the PDE propagator or forward operator $$\mathcal{P}_s,$$ which produces a representation of the entire solution surface at some future moment, given current and boundary conditions. $\mathcal{P}_s[f(t, \cdot)]=f( t+s, \cdot).$ We might also discuss a solution operator $\solop:\begin{array}{l}\mathscr{U}\to\mathscr{F}\\ u\mapsto f\end{array}$ such that $\mathcal{D}\left[\solop[u]\right]=u.$

Handling all these weird, and presumably infinite-dimensional, function spaces $$\mathscr{A},\mathscr{U},\mathscr{F},\dots$$ on a finite computer requires use to introduce a notion of discretisation. We need to find some finite-dimensional representations of these functions so that they can be computed in a finite machine. PDE solvers use various tricks to do that, and each one is its own research field. Finite difference approximations treat all the solutions as values on a grid, effectively approximating $$\mathscr{F}$$ with some new space of functions $$\mathbb{Z}^2 \times \mathbb{Z} \to \mathbb{R},$$ or, if you’d like, in terms of “bar chart” basis functions. Finite element methods define the PDE over a more complicated indexing system of compactly-supported basis functions which form a mesh. Particle systems approximate PDEs with moving particle who define their own adaptive basis. If there is some other natural (preferably orthogonal) basis of functions on the solution surface we might use those, for example with the right structure the eigenfunctions of the defining operator might give us such a basis. Fourier bases are famous in this case.

A classic for neural nets is to learn a finite-difference approximation of the PDE on a grid of values and treat it as a convnet regression, and indeed the dynamical treatment of neural nets is based on that. For various practical reasons I would like to avoid requiring a grid on my input values as much as possible. For one thing, grid systems are memory intensive and need expensive GPUs. For another, it is hard to integrate observations at multiple resolutions into a gridded data system. For a third, the research field of image prediction is too crowded for easy publications. Thus, that will not be treated further.

A grid-free approach is graph networks that learn a topology and interaction system. This seems to naturally map on to PDEs of the kind that we usually solve by particle systems, e.g. fluid dynamics with immiscible substances. Nothing wrong with this idea per se, but it does not seem to be the most compelling approach to me for my domain of spatiotemporal prediction where we already know the topology and can avoid all the complicated bits of graph networks. So this I will also ignore for now.

There are a few options. For an overview of many other techniques see Physics-based Deep Learning by Philipp Holl, Maximilian Mueller, Patrick Schnell, Felix Trost, Nils Thuerey, Kiwon Um . Also, see Brunton and Kutz, Data-Driven Science and Engineering. covers related material; both go farther thank mere PDEs and consider general general scientific settings. Also, the eeminar series by the authors of that latter book is a moving feast of the latest results in this area.

Here we look in depth mainly at two important ones.

One approach learns a network $$\hat{f}\in \mathscr{F}, \hat{f}: C \to O$$ such that $$\hat{f}\approx f$$ . This is the annoyingly-named implicit representation trick. Another approach is used in networks like Li, Kovachki, Azizzadenesheli, Liu, Bhattacharya, et al. (2020b) which learn the forward operator $$\mathcal{P}_1: \mathscr{A}\to\mathscr{A}.$$ When the papers mentioned talk about operator learning, this is the operator that they seem to mean per default.

Physics-informed approximation of dynamics

This entire idea might seem weird if you are used to typical ML research. Unlike the usual neural network setting, we start by not trying to solve a statistical inference problem, where we have to learn an unknown prediction function from data, but we have a partially or completely known function (PDE solver) that we are trying to approximate with a more convenient substitute (a neural approximation to that PDE solver).

That approximant is not necessarily exciting as a PDE solver, in itself. Probably we could have implemented the reference PDE solver on the GPU, or tweaked it a little, and got a faster PDE solver. Identifying when we have a non-trivial speed benefit from training a Neuyral net to do a thing is a whole project in itself.

However, I would like it if the reference solvers were easier to differentiate through, and to construct posteriors with - what you might call tomography, or inverse problems. But note that we still do not need to use ML methods to day that. In fact, if I already know the PDE operator and am implementing it in any case, I could avoid the learning step and simply implement the PDE using an off-the-shelf differentiable solver, which would allow us to perform this inference.

Nonetheless, we might wish to learn to approximate a PDE, for whatever reason. Perhaps we do not know the governing equations precisely, or something like that. In my case it is that am required to match an industry-standard black-box solver that is not flexible, which is a common reason. YMMV.

There are several approaches to learning the dynamics of a PDE solver for given parameters.

## Neural operator

Learning to predict the next step given this step. Think image-to-image regression. A whole topic in itself. See Neural operators.

## The PINN lineage

This body of literature encompasses both DeepONet (‘operator learning’) and PINN (‘physics informed neural nets’) approaches. Distinctions TBD.

See PINNs.

## Weak formulation

TODO: should this be filed with PINNs?

A different network topology using the implicit representation trick is explored in Zang et al. (2020) and extended to inverse problems in G. Bao et al. (2020), They discuss this in terms of a weak formulation of a PDE.

🏗️ Terminology warning: I have not yet harmonised the terminology of this section with the rest of the page.

We start with the example second-order elliptic1 PDE with on domain $$\Omega \subset \mathbb{R}^{d}$$ given $\mathcal{D}[u]-f:=-\sum_{i=1}^{d} \partial_{i}\left(\sum_{j=1}^{d} a_{i j} \partial_{j} u\right)+\sum_{i=1}^{d} b_{i} \partial_{i} u+c u-f=0$ where $$a_{i j}, b_{i}, c: \Omega \rightarrow \mathbb{R}$$ for $$i, j \in[d] \triangleq\{1, \ldots, d\}, f: \Omega \rightarrow \mathbb{R}$$ and $$g: \partial \Omega \rightarrow \mathbb{R}$$ are all given. We start by assuming Dirichlet boundary conditions, $$u(x)-g(x)=0,$$ although this is rapidly generalised.

By multiplying both sides by a test function $$\varphi \in H_{0}^{1}(\Omega ; \mathbb{R})$$ and integrating by parts: $\left\{\begin{array}{l}\langle\mathcal{D}[u], \varphi\rangle \triangleq \int_{\Omega}\left(\sum_{j=1}^{d} \sum_{i=1}^{d} a_{i j} \partial_{j} u \partial_{i} \varphi+\sum_{i=1}^{d} b_{i} \varphi \partial_{i} u+c u \varphi-f \varphi\right) \mathrm{d} x=0 \\ \mathcal{B}[u]=0, \quad \text { on } \partial \Omega\end{array}\right.$

The clever insight is that this inspires an adversarial problem to find the weak solutions, by considering the $$L^2$$ operator norm of $$\mathcal{D}[u](\varphi) \triangleq\langle\mathcal{D}[u], \varphi\rangle$$. Then the operator norm of $$\mathcal{D}[u]$$ is defined $\|\mathcal{D}[u]\|_{o p} \triangleq \max \left\{\langle\mathcal{D}[u], \varphi\rangle /\|\varphi\|_{2} \mid \varphi \in H_{0}^{1}, \varphi \neq 0\right\}.$ Therefore, $$u$$ is a weak solution of the PDE if and only if $$\|\mathcal{D}[u]\|_{o p}=0$$ and the boundary condition $$\mathscr{B}[u]=0$$ is satisfied on $$\delta \Omega$$. As $$\|\mathcal{D}[u]\|_{o p} \geq 0$$, we know that a weak solution $$u$$ thus solves the following two equivalent problems in observation: $\min _{u \in H^{1}}\|\mathcal{D}[u]\|_{o p}^{2} \Longleftrightarrow \min _{u \in H^{1}} \max _{\varphi \in H_{0}^{1}}|\langle\mathcal{D}[u], \varphi\rangle|^{2} /\|\varphi\|_{2}^{2}.$

Specifically the solutions $$u_{\theta}: \mathbb{R}^{d} \rightarrow \mathbb{R}$$ are realized as a deep neural network with parameter $$\theta$$ to be learned, such that $$\mathscr{S}\left[u_{\theta}\right]$$ minimizes the (estimated) operator norm. The test function $$\varphi$$, is a deep adversarial network with parameter $$\eta$$, which adversarially challenges $$u_{\theta}$$ by maximizing $$\left\langle\mathcal{D}\left[u_{\theta}\right], \varphi_{\eta}\right\rangle/\left\|\varphi_{\eta}\right\|_{2}$$ for every given $$u_{\theta}$$.

To train the deep neural network $$u_{\theta}$$ and the adversarial network $$\varphi_{\eta}$$ we construct appropriate loss functions $$u_{\theta}$$ and $$\varphi_{\eta}$$. Since logarithm function is monotone and strictly increasing, we can for convenience formulate the objective of $$u_{\theta}$$ and $$\varphi_{\eta}$$ in the interior of $$\Omega$$ as $L_{\text {int }}(\theta, \eta) \triangleq \log \left|\left\langle\mathcal{D}\left[u_{\theta}\right], \varphi_{\eta}\right\rangle\right|^{2}-\log \left\|\varphi_{\eta}\right\|_{2}^{2}.$ In addition, the weak solution $$u_{\theta}$$ must also satisfy the boundary condition $$\mathscr{B}[u]=0$$ on $$\delta \Omega$$ which we fill in as above, calling it $$L_{\text {bdry }}(\theta).$$ The total adversarial objective function is the weighted sum of the two objectives for which we seek for a saddle point that solves the minimax problem: $\min _{o} \max L(\theta, \eta), \text{ where } L(\theta, \eta) \triangleq L_{\text {int }}(\theta, \eta)+\alpha L_{\text {bdry }}(\theta).$ $$\alpha$$ might seem arbitrary; apparently it is useful as a tuning parameter.

This is a very elegant idea, although the implicit representation thing is still a problem for my use cases.

## Neural operator

Learning to predict the next step given this step. Think image-to-image regression. A whole topic in itself. See Neural operators.

TBD

## DeepONet

From the people who brought you PINN, above, comes the paper of Lu, Jin, and Karniadakis (2020). The setup is related, but AFAICT differs in a few ways in that

1. we don’t (necessarily?) use the derivative information at the sensor locations
2. we learn an operator mapping initial/latent conditions to output functions
3. we decompose the input function space into a basis and them sample randomly from the bases in order to span (in some sense) the input space at training time

The authors argue they have found a good topology for a network that does this

A DeepONet consists of two sub-networks, one for encoding the input function at a fixed number of sensors $$x_i, i = 1, \dots, m$$ (branch net), and another for encoding the locations for the output functions (trunk net).

This addresses some problems with generalisation that make the PINN setup seem unsatisfactory; in particular we can change the inputs, or project arbitrary inputs forward.

The boundary conditions and input points appear to stay fixed though, and inference of the unknowns is still vexed.

🏗️

## GAN approaches

One approach I am less familiar with advocates for conditional GAN models to simulate conditional latent distributions. I’m curious about these but they look more computationally expensive and specific than I need at the moment, so I’m filing for later .

A recent examples from fluid-flow dynamics has particularly beautiful animations attached:

F. Sigrist, Künsch, and Stahel (2015b) finds a nice spectral representation of certain classes of stochastic PDE. These are extended in Liu, Yeo, and Lu (2020) to non-stationary operators. By being less generic, these come out with computationally convenient spectral representations.

## Inverse problems

Tomography through PDEs.

## As implicit representations

Many of these PDE methods effectively use the “implicit representation” trick, i.e. they produce networks that map from input coordinates to values of solutions at those coordinates. This means we share some interesting tools with those networks, such as position encodings. TBD.

## Differentiable solvers

Suppose we are keen to devise yet another method that will do clever things to augment PDE solvers with ML somehow. To that end it would be nice to have a PDE solver that was not a completely black box but which we could interrogate for useful gradients. Obviously all PDE solvers use gradient information, but only some of them expose that to us as users; e.g. MODFLOW will give me a solution filed but not the gradients of the field that were used to calculate that gradient. In ML toolkits accessing this information is easy.

OTOH, there is a lot of sophisticatd work done by PDE solvers that is hard for ML toolkits to recreate. That is why PDE solvers are a thing.

Tools which combine both worlds, PDE solutions and ML optimisations, do exist; there are adjoint method systems for mainstream PDE solvers just as there are PDE solvers for ML frameworks. Let us list some of the options under differentiable PDE solvers.

Fits here?

## Incoming

• boschresearch/torchphysics

TorchPhysics is a Python library of (mesh-free) deep learning methods to solve differential equations. You can use TorchPhysics e.g. to

• solve ordinary and partial differential equations
• train a neural network to approximate solutions for different parameters
• solve inverse problems and interpolate external data

## NVIDIA

NVIDIA’s MODULUS (formerly SimNet) needs filing .

They are implementing many popular algorithms, but with a comically clunky distribution system and onerous licensing. Not currently recommended.

## References

Alexanderian, Alen. 2021. arXiv:2005.12998 [Math], January.
Alexanderian, Alen, Noemi Petra, Georg Stadler, and Omar Ghattas. 2016. SIAM Journal on Scientific Computing 38 (1): A243–72.
Altmann, Robert, Patrick Henning, and Daniel Peterseim. 2021. Acta Numerica 30 (May): 1–86.
Arora, Sanjeev, Rong Ge, Tengyu Ma, and Ankur Moitra. 2015. In Proceedings of The 28th Conference on Learning Theory, 40:113–49. Paris, France: PMLR.
Atkinson, Steven, Waad Subber, and Liping Wang. 2019. “Data-Driven Discovery of Free-Form Governing Differential Equations.” In, 7.
Bao, Gang, Xiaojing Ye, Yaohua Zang, and Haomin Zhou. 2020. Inverse Problems 36 (11): 115003.
Bao, Tianshu, Shengyu Chen, Taylor T. Johnson, Peyman Givi, Shervin Sammak, and Xiaowei Jia. 2022. In Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence, 118–28. PMLR.
Bar-Sinai, Yohai, Stephan Hoyer, Jason Hickey, and Michael P. Brenner. 2019. Proceedings of the National Academy of Sciences 116 (31): 15344–49.
Basir, Shamsulhaq, and Inanc Senocak. n.d. In AIAA SCITECH 2022 Forum. American Institute of Aeronautics and Astronautics.
Beck, Christian, Weinan E, and Arnulf Jentzen. 2019. Journal of Nonlinear Science 29 (4): 1563–1619.
Bezgin, Deniz A., Aaron B. Buhendwa, and Nikolaus A. Adams. 2022. arXiv:2203.13760 [Physics], March.
Bhattacharya, Kaushik, Bamdad Hosseini, Nikola B. Kovachki, and Andrew M. Stuart. 2020. arXiv:2005.03180 [Cs, Math, Stat], May.
Blechschmidt, Jan, and Oliver G. Ernst. 2021. GAMM-Mitteilungen 44 (2): e202100006.
Bottero, Luca, Francesco Calisto, Giovanni Graziano, Valerio Pagliarino, Martina Scauda, Sara Tiengo, and Simone Azeglio. 2020. December.
Brandstetter, Johannes, Daniel Worrall, and Max Welling. 2022. arXiv:2202.03376 [Cs, Math], March.
Brehmer, Johann, Kyle Cranmer, Siddharth Mishra-Sharma, Felix Kling, and Gilles Louppe. 2019. “Mining Gold: Improving Simulation-Based Inference with Latent Information.” In, 7.
Brunton, Steven L., and Jose Nathan Kutz. 2019. Data-Driven Science and Engineering: Machine Learning, Dynamical Systems, and Control. Cambridge: Cambridge University Press.
Chu, Mengyu, Nils Thuerey, Hans-Peter Seidel, Christian Theobalt, and Rhaleb Zayer. 2021. ACM Transactions on Graphics 40 (4): 1–13.
Cockayne, Jon, and Andrew B. Duncan. 2020. September.
Cranmer, Miles D, Rui Xu, Peter Battaglia, and Shirley Ho. 2019. “Learning Symbolic Physics with Graph Networks.” In Machine Learning and the Physical Sciences Workshop at the 33rd Conference on Neural Information Processing Systems (NeurIPS), 6.
Dandekar, Raj, Karen Chung, Vaibhav Dixit, Mohamed Tarek, Aslan Garcia-Valadez, Krishna Vishal Vemula, and Chris Rackauckas. 2021. arXiv:2012.07244 [Cs], March.
Daw, Arka, Jie Bu, Sifan Wang, Paris Perdikaris, and Anuj Karpatne. 2022. arXiv.
Di Giovanni, Francesco, James Rowbottom, Benjamin P. Chamberlain, Thomas Markovich, and Michael M. Bronstein. 2022. arXiv.
Duffin, Connor, Edward Cripps, Thomas Stemler, and Mark Girolami. 2021. Proceedings of the National Academy of Sciences 118 (2).
E, Weinan. 2021. Notices of the American Mathematical Society 68 (04): 1.
E, Weinan, Jiequn Han, and Arnulf Jentzen. 2017. Communications in Mathematics and Statistics 5 (4): 349–80.
———. 2020. arXiv:2008.13333 [Cs, Math], September.
E, Weinan, and Bing Yu. 2018. Communications in Mathematics and Statistics 6 (1): 1–12.
Eigel, Martin, Reinhold Schneider, Philipp Trunschke, and Sebastian Wolf. 2019. Advances in Computational Mathematics 45 (5-6): 2503–32.
Fan, Yuwei, Cindy Orozco Bohorquez, and Lexing Ying. 2019. Journal of Computational Physics 384 (May): 1–15.
Finzi, Marc, Roberto Bondesan, and Max Welling. 2020. arXiv:2010.10876 [Cs], October.
Freeman, C Daniel, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. 2021. arXiv Preprint arXiv:2106.13281.
Frerix, Thomas, Dmitrii Kochkov, Jamie A. Smith, Daniel Cremers, Michael P. Brenner, and Stephan Hoyer. 2021. arXiv.
Gan, Chuang, Jeremy Schwartz, Seth Alter, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, et al. 2020. arXiv Preprint arXiv:2007.04954.
Ghattas, Omar, and Karen Willcox. 2021. Acta Numerica 30 (May): 445–554.
Girolami, Mark, Eky Febrianto, Ge Yin, and Fehmi Cirak. 2021. Computer Methods in Applied Mechanics and Engineering 375 (March): 113533.
Goswami, Somdatta, Aniruddha Bora, Yue Yu, and George Em Karniadakis. 2022. July.
Granas, Andrzej, and James Dugundji. 2003. Fixed Point Theory. Springer Monographs in Mathematics. New York, NY: Springer New York.
Guibas, John, Morteza Mardani, Zongyi Li, Andrew Tao, Anima Anandkumar, and Bryan Catanzaro. 2021. November.
Gulian, Mamikon, Ari Frankel, and Laura Swiler. 2020. arXiv:2012.11857 [Cs, Math, Stat], December.
Guo, Mengwu, and Jan S. Hesthaven. 2019. Computer Methods in Applied Mechanics and Engineering 345 (March): 75–99.
Han, Jiequn, Arnulf Jentzen, and Weinan E. 2018. Proceedings of the National Academy of Sciences 115 (34): 8505–10.
Hennigh, Oliver, Susheela Narasimhan, Mohammad Amin Nabian, Akshay Subramaniam, Kaustubh Tangsali, Max Rietmann, Jose del Aguila Ferrandis, Wonmin Byeon, Zhiwei Fang, and Sanjay Choudhry. 2020. arXiv:2012.07938 [Physics], December.
Hoffimann, Júlio, Maciel Zortea, Breno de Carvalho, and Bianca Zadrozny. 2021. Frontiers in Applied Mathematics and Statistics 7.
Holl, Philipp, Vladlen Koltun, Kiwon Um, and Nils Thuerey. 2020. In NeurIPS Workshop.
Holl, Philipp, Nils Thuerey, and Vladlen Koltun. 2020. In ICLR, 5.
Hu, Yuanming, Tzu-Mao Li, Luke Anderson, Jonathan Ragan-Kelley, and Frédo Durand. 2019. ACM Transactions on Graphics 38 (6): 1–16.
Huang, Zizhou, Teseo Schneider, Minchen Li, Chenfanfu Jiang, Denis Zorin, and Daniele Panozzo. 2021. arXiv:2112.05309 [Cs], December.
Innes, Mike, Alan Edelman, Keno Fischer, Chris Rackauckas, Elliot Saba, Viral B. Shah, and Will Tebbutt. 2019. arXiv.
Jiang, Chiyu Max, Soheil Esmaeilzadeh, Kamyar Azizzadenesheli, Karthik Kashinath, Mustafa Mustafa, Hamdi A. Tchelepi, Philip Marcus, Prabhat, and Anima Anandkumar. 2020. May.
Jo, Hyeontae, Hwijae Son, Hyung Ju Hwang, and Eun Heui Kim. 2020. Networks & Heterogeneous Media 15 (2): 247.
Kadri, Hachem, Emmanuel Duflos, Philippe Preux, Stéphane Canu, Alain Rakotomamonjy, and Julien Audiffren. 2016. The Journal of Machine Learning Research 17 (1): 613–66.
Karniadakis, George Em, Ioannis G. Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. 2021. Nature Reviews Physics 3 (6): 422–40.
Kasim, M. F., D. Watson-Parris, L. Deaconu, S. Oliver, P. Hatfield, D. H. Froula, G. Gregori, et al. 2020. arXiv:2001.08055 [Physics, Stat], January.
Kasim, Muhammad, J Topp-Mugglestone, P Hatﬁeld, D H Froula, G Gregori, M Jarvis, E Viezzer, and Sam Vinko. 2019. “A Million Times Speed up in Parameters Retrieval with Deep Learning.” In, 5.
Kharazmi, E., Z. Zhang, and G. E. Karniadakis. 2019. arXiv:1912.00873 [Physics, Stat], November.
Khodayi-Mehr, Reza, and Michael M. Zavlanos. 2019. arXiv:1912.07443 [Physics, Stat], December.
Kochkov, Dmitrii, Alvaro Sanchez-Gonzalez, Jamie Smith, Tobias Pfaff, Peter Battaglia, and Michael P Brenner. 2020. “Learning Latent FIeld Dynamics of PDEs.” In, 7.
Kochkov, Dmitrii, Jamie A. Smith, Ayya Alieva, Qing Wang, Michael P. Brenner, and Stephan Hoyer. 2021. Proceedings of the National Academy of Sciences 118 (21).
Kononenko, O., and I. Kononenko. 2018. arXiv:1801.07337 [Physics], March.
Kovachki, Nikola, Samuel Lanthaler, and Siddhartha Mishra. 2021. arXiv:2107.07562 [Cs, Math], July.
Kovachki, Nikola, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. 2021. In arXiv:2108.08481 [Cs, Math].
Krämer, Nicholas, Nathanael Bosch, Jonathan Schmidt, and Philipp Hennig. 2021. arXiv.
Krishnapriyan, Aditi, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. 2021. In Advances in Neural Information Processing Systems, 34:26548–60. Curran Associates, Inc.
Lagaris, I.E., A. Likas, and D.I. Fotiadis. 1998. IEEE Transactions on Neural Networks 9 (5): 987–1000.
Lei, Huan, Jing Li, Peiyuan Gao, Panos Stinis, and Nathan Baker. 2018. April.
Li, Zongyi, Daniel Zhengyu Huang, Burigede Liu, and Anima Anandkumar. 2022. arXiv.
Li, Zongyi, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. 2020a. In. arXiv.
———. 2020b. arXiv:2010.08895 [Cs, Math], October.
Li, Zongyi, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Andrew Stuart, Kaushik Bhattacharya, and Anima Anandkumar. 2020. In Advances in Neural Information Processing Systems. Vol. 33.
Li, Zongyi, Hongkai Zheng, Nikola Borislavov Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Andrew Stuart, Kamyar Azizzadenesheli, and Anima Anandkumar. 2021. November.
Lian, Heng. 2007. Canadian Journal of Statistics 35 (4): 597–606.
Liao, Yulei, and Pingbing Ming. 2021.
Lienen, Marten, and Stephan Günnemann. 2021. In International Conference on Learning Representations.
Liu, Xiao, Kyongmin Yeo, and Siyuan Lu. 2020. Journal of the American Statistical Association 0 (0): 1–18.
Long, Da, Zheng Wang, Aditi Krishnapriyan, Robert Kirby, Shandian Zhe, and Michael Mahoney. 2022. arXiv.
Long, Zichao, Yiping Lu, Xianzhong Ma, and Bin Dong. 2018. In Proceedings of the 35th International Conference on Machine Learning, 3208–16. PMLR.
Lu, Lu, Pengzhan Jin, and George Em Karniadakis. 2020. arXiv:1910.03193 [Cs, Stat], April.
Lu, Lu, Xuhui Meng, Zhiping Mao, and George Em Karniadakis. 2021. SIAM Review 63 (1): 208–28.
Ma, Yingbo, Shashi Gowda, Ranjan Anantharaman, Chris Laughman, Viral Shah, and Chris Rackauckas. 2021. March.
Magnani, Emilia, Nicholas Krämer, Runa Eschenhagen, Lorenzo Rosasco, and Philipp Hennig. 2022. arXiv.
Meng, Xuhui, Hessam Babaee, and George Em Karniadakis. 2021. Journal of Computational Physics 438 (August): 110361.
Mitusch, Sebastian K., Simon W. Funke, and Jørgen S. Dokken. 2019. Journal of Open Source Software 4 (38): 1292.
Mowlavi, Saviz, and Saleh Nabi. 2021. arXiv:2111.09880 [Physics], November.
Müller, Johannes, and Marius Zeinhofer. 2020. arXiv.
Nabian, Mohammad Amin, and Hadi Meidani. 2019. Probabilistic Engineering Mechanics 57 (July): 14–25.
Naumann, Uwe. 2011. The Art of Differentiating Computer Programs: An Introduction to Algorithmic Differentiation. Society for Industrial and Applied Mathematics.
Négiar, Geoffrey, Michael W. Mahoney, and Aditi S. Krishnapriyan. 2022. arXiv.
O’Hagan, Anthony. 2013. “Polynomial Chaos: A Tutorial and Critique from a Statistician’s Perspective,” 20.
Oladyshkin, S., and W. Nowak. 2012. Reliability Engineering & System Safety 106 (October): 179–90.
Otness, Karl, Arvi Gjoka, Joan Bruna, Daniele Panozzo, Benjamin Peherstorfer, Teseo Schneider, and Denis Zorin. 2021. In.
Pathak, Jaideep, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, et al. 2022. February, 28.
Perdikaris, Paris, Daniele Venturi, and George Em Karniadakis. 2016. SIAM Journal on Scientific Computing 38 (4): B521–38.
Perdikaris, P., D. Venturi, J. O. Royset, and G. E. Karniadakis. 2015. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences 471 (2179): 20150018.
Pestourie, Raphaël, Youssef Mroueh, Chris Rackauckas, Payel Das, and Steven G. Johnson. 2022. arXiv.
Pestourie, Raphaël, Youssef Mroueh, Christopher Vincent Rackauckas, Payel Das, and Steven Glenn Johnson. 2021. In.
Poli, Michael, Stefano Massaroli, Federico Berto, Jinkyoo Park, Tri Dao, Christopher Re, and Stefano Ermon. 2022. In.
Qian, Elizabeth, Boris Kramer, Benjamin Peherstorfer, and Karen Willcox. 2020. Physica D: Nonlinear Phenomena 406 (May): 132401.
Rackauckas, Chris, Alan Edelman, Keno Fischer, Mike Innes, Elliot Saba, Viral B Shah, and Will Tebbutt. 2020. MIT Web Domain, 6.
Rackauckas, Christopher. 2019. The Winnower.
Raissi, Maziar, Paris Perdikaris, and George Em Karniadakis. 2017a. November.
Raissi, Maziar, P. Perdikaris, and George Em Karniadakis. 2019. Journal of Computational Physics 378 (February): 686–707.
Ramsundar, Bharath, Dilip Krishnamurthy, and Venkatasubramanian Viswanathan. 2021. arXiv:2109.07573 [Physics], September.
Ray, Deep, Orazio Pinti, and Assad A. Oberai. 2023.
Rezende, Danilo J, Sébastien Racanière, Irina Higgins, and Peter Toth. 2019. “Equivariant Hamiltonian Flows.” In Machine Learning and the Physical Sciences Workshop at the 33rd Conference on Neural Information Processing Systems (NeurIPS), 6.
Rodriguez-Torrado, Ruben, Pablo Ruiz, Luis Cueto-Felgueroso, Michael Cerny Green, Tyler Friesen, Sebastien Matringe, and Julian Togelius. 2022. Scientific Reports 12 (1): 7557.
Saha, Akash, and Palaniappan Balamurugan. 2020. In Advances in Neural Information Processing Systems. Vol. 33.
Sarkar, Soumalya, and Michael Joly. 2019. In NeurIPS, 5.
Schnell, Patrick, Philipp Holl, and Nils Thuerey. 2022. arXiv:2203.10131 [Physics], March.
Shankar, Varun, Gavin D Portwood, Arvind T Mohan, Peetak P Mitra, Christopher Rackauckas, Lucas A Wilson, David P Schmidt, and Venkatasubramanian Viswanathan. 2020. “Learning Non-Linear Spatio-Temporal Dynamics with Convolutional Neural ODEs.” In Third Workshop on Machine Learning and the Physical Sciences (NeurIPS 2020).
Shi, Zheng, Nur Sila Gulgec, Albert S. Berahas, Shamim N. Pakzad, and Martin Takáč. 2020. arXiv.
Sigrist, Fabio Roman Albert. 2013. Application/pdf. ETH Zurich.
Sigrist, Fabio, Hans R. Künsch, and Werner A. Stahel. 2015a. Application/pdf. Journal of Statistical Software 63 (14).
———. 2015b. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 77 (1): 3–33.
Silvester, Steven, Anthony Tanbakuchi, Paul Müller, Juan Nunez-Iglesias, Mark Harfouche, Almar Klein, Matt McCormick, et al. 2020. Zenodo.
Sirignano, Justin, and Konstantinos Spiliopoulos. 2018. Journal of Computational Physics 375 (December): 1339–64.
Solin, Arno, and Simo Särkkä. 2020. Statistics and Computing 30 (2): 419–46.
Stachenfeld, Kimberly, Drummond B. Fielding, Dmitrii Kochkov, Miles Cranmer, Tobias Pfaff, Jonathan Godwin, Can Cui, Shirley Ho, Peter Battaglia, and Alvaro Sanchez-Gonzalez. 2022. arXiv.
Sulam, Jeremias, Aviad Aberdam, Amir Beck, and Michael Elad. 2020. IEEE Transactions on Pattern Analysis and Machine Intelligence 42 (8): 1968–80.
Tait, Daniel J., and Theodoros Damoulas. 2020. arXiv:2006.15641 [Cs, Stat], June.
Takamoto, Makoto, Timothy Praditia, Raphael Leiteritz, Dan MacKinlay, Francesco Alesiani, Dirk Pflüger, and Mathias Niepert. 2022.
Tartakovsky, Alexandre M., Carlos Ortiz Marrero, Paris Perdikaris, Guzel D. Tartakovsky, and David Barajas-Solano. 2018. August.
Thuerey, Nils, Philipp Holl, Maximilian Mueller, Patrick Schnell, Felix Trost, and Kiwon Um. 2021. Physics-Based Deep Learning. WWW.
Torrado, Ruben Rodriguez, Pablo Ruiz, Luis Cueto-Felgueroso, Michael Cerny Green, Tyler Friesen, Sebastien Matringe, and Julian Togelius. 2021. December.
Um, Kiwon, Robert Brand, Yun Fei, Philipp Holl, and Nils Thuerey. 2021. arXiv:2007.00016 [Physics], January.
Um, Kiwon, and Philipp Holl. 2021. “Differentiable Physics for Improving the Accuracy of Iterative PDE-Solvers with Neural Networks.” In, 5.
Vadyala, Shashank Reddy, Sai Nethra Betgeri, and Naga Parameshwari Betgeri. 2022. Array 13 (March): 100110.
Wacker, Philipp. 2017. arXiv:1701.07989 [Math], April.
Wang, Chulin, Eloisa Bentivegna, Wang Zhou, Levente J Klein, and Bruce Elmegreen. 2020. “Physics-Informed Neural Network Super Resolution for Advection-Diffusion Models.” In, 9.
Wang, Rui, Karthik Kashinath, Mustafa Mustafa, Adrian Albert, and Rose Yu. 2020. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 1457–66. KDD ’20. New York, NY, USA: Association for Computing Machinery.
Wang, Sifan, Xinling Yu, and Paris Perdikaris. 2020. July.
Wen, Gege, Zongyi Li, Kamyar Azizzadenesheli, Anima Anandkumar, and Sally M. Benson. 2022. Advances in Water Resources 163 (May): 104180.
Xu, Kailai, and Eric Darve. 2019. arXiv.
———. 2020. In arXiv:2011.11955 [Cs, Math].
Yang, Liu, Xuhui Meng, and George Em Karniadakis. 2021. Journal of Computational Physics 425 (January): 109913.
Yang, Liu, Dongkun Zhang, and George Em Karniadakis. 2020. SIAM Journal on Scientific Computing 42 (1): A292–317.
Zammit-Mangion, Andrew, Michael Bertolacci, Jenny Fisher, Ann Stavert, Matthew L. Rigby, Yi Cao, and Noel Cressie. 2021. Geoscientific Model Development Discussions, July, 1–51.
Zang, Yaohua, Gang Bao, Xiaojing Ye, and Haomin Zhou. 2020. Journal of Computational Physics 411 (June): 109409.
Zeng, Qi, Spencer H. Bryngelson, and Florian Schäfer. 2022. arXiv.
Zhang, Dongkun, Ling Guo, and George Em Karniadakis. 2020. SIAM Journal on Scientific Computing 42 (2): A639–65.
Zhang, Dongkun, Lu Lu, Ling Guo, and George Em Karniadakis. 2019. Journal of Computational Physics 397 (November): 108850.
Zhi, Weiming, Tin Lai, Lionel Ott, Edwin V. Bonilla, and Fabio Ramos. 2022. In International Conference on Machine Learning, 27060–74. PMLR.
Zubov, Kirill, Zoe McCarthy, Yingbo Ma, Francesco Calisto, Valerio Pagliarino, Simone Azeglio, Luca Bottero, et al. 2021. arXiv.

1. For five internet points, can you explain to me why it must be elliptic?↩︎

### 1 comment

As a geophysicist working with 3D and lots of data with coupled PDEs, a fast solver is nice, but often intractably slow. Even with modern solvers. Even with GPU. Replacing the solver with a NN approximant is potentially much faster, even if the speed is merely amortized. That has so many benefits for real-world modeling work.