Foundation models for partial differential equations
November 14, 2024 — November 14, 2024
Using statistical or machine learning approaches to solve PDEs via foundation models.
1 Architecture
Are they always transformer-like?
2 Representations
Do we still use tokens?
3 What inverse problems can we handle this way?
This is the key challenge for foundation models, IMO: conditioning on the observations we have, and solving for what produced them. I can think of ways we might do this, but the implementations I have observed in the wild are not terribly persuasive. TBC
4 Incoming
5 References
Alkin, Fürst, Schmid, et al. 2024. “Universal Physics Transformers: A Framework For Efficiently Scaling Neural Operators.” In Advances in Neural Information Processing Systems.
Barman, Caron, Sullivan, et al. 2025. “Large Physics Models: Towards a Collaborative Approach with Large Language Models and Foundation Models.”
Bodnar, Bruinsma, Lucic, et al. 2024. “Aurora: A Foundation Model of the Atmosphere.”
Duraisamy, Iaccarino, and Xiao. 2019. “Turbulence Modeling in the Age of Data.” Annual Review of Fluid Mechanics.
Guibas, Mardani, Li, et al. 2021. “Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers.”
Herde, Raonić, Rohner, et al. 2024. “Poseidon: Efficient Foundation Models for PDEs.”
Hoffimann, Zortea, de Carvalho, et al. 2021. “Geostatistical Learning: Challenges and Opportunities.” Frontiers in Applied Mathematics and Statistics.
Mialon, Garrido, Lawrence, et al. 2024. “Self-Supervised Learning with Lie Symmetries for Partial Differential Equations.”
Xu, Gupta, Cheng, et al. 2024. “Specialized Foundation Models Struggle to Beat Supervised Baselines.”