Foundation models for partial differential equations

November 14, 2024 — November 14, 2024

calculus
dynamical systems
geometry
Hilbert space
how do science
Lévy processes
machine learning
neural nets
PDEs
physics
regression
sciml
SDEs
signal processing
statistics
statmech
stochastic processes
surrogate
time series
uncertainty

Suspiciously similar content

Figure 1

\(\newcommand{\solop}{\mathcal{G}^{\dagger}}\)

Using statistical or machine learning approaches to solve PDEs via foundation models.

1 Architecture

Are they always transformer-like?

2 Representations

Do we still use tokens?

3 What inverse problems can we handle this way?

This is the key challenge for foundation models, IMO: conditioning on the observations we have, and solving for what produced them. I can think of ways we might do this, but the implementations I have observed in the wild are not terribly persuasive. TBC

4 Incoming

5 References

Alkin, Fürst, Schmid, et al. 2024. “Universal Physics Transformers: A Framework For Efficiently Scaling Neural Operators.” In Advances in Neural Information Processing Systems.
Barman, Caron, Sullivan, et al. 2025. “Large Physics Models: Towards a Collaborative Approach with Large Language Models and Foundation Models.”
Bodnar, Bruinsma, Lucic, et al. 2024. “Aurora: A Foundation Model of the Atmosphere.”
Duraisamy, Iaccarino, and Xiao. 2019. “Turbulence Modeling in the Age of Data.” Annual Review of Fluid Mechanics.
Guibas, Mardani, Li, et al. 2021. “Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers.”
Herde, Raonić, Rohner, et al. 2024. “Poseidon: Efficient Foundation Models for PDEs.”
Hoffimann, Zortea, de Carvalho, et al. 2021. “Geostatistical Learning: Challenges and Opportunities.” Frontiers in Applied Mathematics and Statistics.
Mialon, Garrido, Lawrence, et al. 2024. “Self-Supervised Learning with Lie Symmetries for Partial Differential Equations.”
Xu, Gupta, Cheng, et al. 2024. “Specialized Foundation Models Struggle to Beat Supervised Baselines.”