Foundation models for partial differential equations

November 14, 2024 — November 14, 2024

calculus
dynamical systems
geometry
Hilbert space
how do science
Lévy processes
machine learning
neural nets
PDEs
physics
regression
sciml
SDEs
signal processing
statistics
statmech
stochastic processes
surrogate
time series
uncertainty
Figure 1

\(\newcommand{\solop}{\mathcal{G}^{\dagger}}\)

Using statistical or machine learning approaches to solve PDEs via foundation models.

1 Architecture

2 Representations

Do we still use tokens?

3 What inverse problems can we handle this way?

This is the key challenge for foundation models, IMO: conditioning on the observations we have, and solving for what produced them. I can think of ways we might do this, but the implementations I have observed in the wild are not terribly persuasive. TBC

4 References

Bodnar, Bruinsma, Lucic, et al. 2024. “Aurora: A Foundation Model of the Atmosphere.”
Duraisamy, Iaccarino, and Xiao. 2019. “Turbulence Modeling in the Age of Data.” Annual Review of Fluid Mechanics.
Guibas, Mardani, Li, et al. 2021. “Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers.”
Herde, Raonić, Rohner, et al. 2024. “Poseidon: Efficient Foundation Models for PDEs.”
Hoffimann, Zortea, de Carvalho, et al. 2021. “Geostatistical Learning: Challenges and Opportunities.” Frontiers in Applied Mathematics and Statistics.
Mialon, Garrido, Lawrence, et al. 2024. “Self-Supervised Learning with Lie Symmetries for Partial Differential Equations.”
Xu, Gupta, Cheng, et al. 2024. “Specialized Foundation Models Struggle to Beat Supervised Baselines.”