System identification in continuous time

Learning in continuous ODEs, SDEs and CDEs



Learning the parameters of a dynamical system in continuous time. I am imagining here that we are thinking about a parametric setting. If we want to learn some non-parametric approximation to dynamics

Introductory reading

Rackauckas et al. (2018) and and even some tutorial implementations by the indefatigable Chris Rackauckas, and a whole MIT course. Chris Rackauckas’ lecture notes christen this area “scientific machine learning”.

Learning stochastic partial differential equations where a whole random field evolves in time is something of interest to me; see spatiotemporal nets and spatiotemporal dynamics for more on that theme.

In PDEs

See differentiable PDE solvers for now.

General SDEs

With sparse SDEs

For least-squares system identification see sparse stochastic processes.

Controlled differential equations

TBD

Method of adjoints

A trick in differentiation which happens to be useful in differentiating likelihood (or other functions) of time evolving systems e.g. Errico (1997).

For now, see the method of adjoints in the autodiff notebook.

Python

Diffrax

Diffrax is a JAX-based library providing numerical differential equation solvers.

Features include:

  • ODE/SDE/CDE (ordinary/stochastic/controlled) solvers
  • lots of different solvers (including Tsit5, Dopri8, symplectic solvers, implicit solvers)
  • vmappable everything (including the region of integration)
  • using a PyTree as the state
  • dense solutions
  • multiple adjoint methods for backpropagation
  • support for neural differential equations.

From a technical point of view, the internal structure of the library is pretty cool — all kinds of equations (ODEs, SDEs, CDEs) are solved in a unified way (rather than being treated separately), producing a small tightly-written library.

torchdyn (docs).

Julia

Chris Rauckackas is a veritable wizard with this stuff; read his blog.

Here is a tour of fun tricks with stochastic PDEs. There is a lot of tooling for this; DiffEqOperators … does something. DiffEqFlux (EZ neural ODEs works with Flux and claims to make neural SDE simple.

+1 for Julia here.

References

Batz, Philipp, Andreas Ruttor, and Manfred Opper. 2017. Approximate Bayes Learning of Stochastic Differential Equations.” arXiv:1702.05390 [Physics, Stat], February.
Baydin, Atilim Gunes, and Barak A. Pearlmutter. 2014. Automatic Differentiation of Algorithms for Machine Learning.” arXiv:1404.7456 [Cs, Stat], April.
Beck, Christian, Weinan E, and Arnulf Jentzen. 2019. Machine Learning Approximation Algorithms for High-Dimensional Fully Nonlinear Partial Differential Equations and Second-Order Backward Stochastic Differential Equations.” Journal of Nonlinear Science 29 (4): 1563–1619.
Chang, Bo, Minmin Chen, Eldad Haber, and Ed H. Chi. 2019. AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks.” In Proceedings of ICLR.
Chen, Tian Qi, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. 2018. Neural Ordinary Differential Equations.” In Advances in Neural Information Processing Systems 31, edited by S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, 6572–83. Curran Associates, Inc.
Dandekar, Raj, Karen Chung, Vaibhav Dixit, Mohamed Tarek, Aslan Garcia-Valadez, Krishna Vishal Vemula, and Chris Rackauckas. 2021. Bayesian Neural Ordinary Differential Equations.” arXiv:2012.07244 [Cs], March.
Delft, Anne van, and Michael Eichler. 2016. Locally Stationary Functional Time Series.” arXiv:1602.05125 [Math, Stat], February.
Errico, Ronald M. 1997. What Is an Adjoint Model? Bulletin of the American Meteorological Society 78 (11): 2577–92.
Gierjatowicz, Patryk, Marc Sabate-Vidales, David Šiška, Lukasz Szpruch, and Žan Žurič. 2020. Robust Pricing and Hedging via Neural SDEs.” arXiv:2007.04154 [Cs, q-Fin, Stat], July.
Grathwohl, Will, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. 2018. FFJORD: Free-Form Continuous Dynamics for Scalable Reversible Generative Models.” arXiv:1810.01367 [Cs, Stat], October.
Gu, Albert, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. 2021. Combining Recurrent, Convolutional, and Continuous-Time Models with Linear State-Space Layers.” arXiv:2110.13985 [Cs], October.
Jia, Junteng, and Austin R Benson. 2019. Neural Jump Stochastic Differential Equations.” In Advances in Neural Information Processing Systems 32, edited by H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alché-Buc, E. Fox, and R. Garnett, 9847–58. Curran Associates, Inc.
Kelly, Jacob, Jesse Bettencourt, Matthew James Johnson, and David Duvenaud. 2020. Learning Differential Equations That Are Easy to Solve.” In.
Li, Xuechen, Ting-Kam Leonard Wong, Ricky T. Q. Chen, and David Duvenaud. 2020. Scalable Gradients for Stochastic Differential Equations.” In International Conference on Artificial Intelligence and Statistics, 3870–82. PMLR.
Ljung, Lennart. 2010. Perspectives on System Identification.” Annual Reviews in Control 34 (1): 1–12.
Lu, Peter Y., Joan Ariño, and Marin Soljačić. 2021. Discovering Sparse Interpretable Dynamics from Partial Observations.” arXiv:2107.10879 [Physics], July.
Marelli, D. 2007. A Functional Analysis Approach to Subband System Approximation and Identification.” IEEE Transactions on Signal Processing 55 (2): 493–506.
Massaroli, Stefano, Michael Poli, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama. 2020. Dissecting Neural ODEs.” In arXiv:2002.08071 [Cs, Stat].
Morrill, James, Patrick Kidger, Cristopher Salvi, James Foster, and Terry Lyons. 2020. “Neural CDEs for Long Time Series via the Log-ODE Method.” In, 5.
Nabian, Mohammad Amin, and Hadi Meidani. 2019. A Deep Learning Solution Approach for High-Dimensional Random Differential Equations.” Probabilistic Engineering Mechanics 57 (July): 14–25.
Pham, Tung, and Victor Panaretos. 2016. Methodology and Convergence Rates for Functional Time Series Regression.” arXiv:1612.07197 [Math, Stat], December.
Pillonetto, Gianluigi. 2016. The Interplay Between System Identification and Machine Learning.” arXiv:1612.09158 [Cs, Stat], December.
Rackauckas, Christopher. 2019. The Essential Tools of Scientific Machine Learning (Scientific ML).” The Winnower, August.
Rackauckas, Christopher, Yingbo Ma, Vaibhav Dixit, Xingjian Guo, Mike Innes, Jarrett Revels, Joakim Nyberg, and Vijay Ivaturi. 2018. A Comparison of Automatic Differentiation and Continuous Sensitivity Analysis for Derivatives of Differential Equation Solutions.” arXiv:1812.01892 [Cs], December.
Rackauckas, Christopher, Yingbo Ma, Julius Martensen, Collin Warner, Kirill Zubov, Rohit Supekar, Dominic Skinner, Ali Ramadhan, and Alan Edelman. 2020. Universal Differential Equations for Scientific Machine Learning.” arXiv:2001.04385 [Cs, Math, q-Bio, Stat], August.
Ramsundar, Bharath, Dilip Krishnamurthy, and Venkatasubramanian Viswanathan. 2021. Differentiable Physics: A Position Piece.” arXiv:2109.07573 [Physics], September.
Särkkä, Simo. 2011. Linear Operators and Stochastic Partial Differential Equations in Gaussian Process Regression.” In Artificial Neural Networks and Machine Learning – ICANN 2011, edited by Timo Honkela, Włodzisław Duch, Mark Girolami, and Samuel Kaski, 6792:151–58. Lecture Notes in Computer Science. Berlin, Heidelberg: Springer.
Särkkä, Simo, and Arno Solin. 2019. Applied Stochastic Differential Equations. Institute of Mathematical Statistics Textbooks 10. Cambridge ; New York, NY: Cambridge University Press.
Solin, Arno. 2016. Stochastic Differential Equation Methods for Spatio-Temporal Gaussian Process Regression.” Aalto University.
Um, Kiwon, Robert Brand, Yun Fei, Philipp Holl, and Nils Thuerey. 2021. Solver-in-the-Loop: Learning from Differentiable Physics to Interact with Iterative PDE-Solvers.” arXiv:2007.00016 [Physics], January.
Um, Kiwon, and Philipp Holl. 2021. “Differentiable Physics for Improving the Accuracy of Iterative PDE-Solvers with Neural Networks.” In, 5.
Unser, Michael A., and Pouya Tafti. 2014. An Introduction to Sparse Stochastic Processes. New York: Cambridge University Press.
Werbos, Paul J. 1988. Generalization of Backpropagation with Application to a Recurrent Gas Market Model.” Neural Networks 1 (4): 339–56.
Yıldız, Çağatay, Markus Heinonen, and Harri Lähdesmäki. 2019. ODE\(^2\)VAE: Deep Generative Second Order ODEs with Bayesian Neural Networks.” arXiv:1905.10994 [Cs, Stat], October.
Yoshida, Nakahiro. 2022. Quasi-Likelihood Analysis and Its Applications.” Statistical Inference for Stochastic Processes 25 (1): 43–60.
Zhang, Dongkun, Ling Guo, and George Em Karniadakis. 2020. Learning in Modal Space: Solving Time-Dependent Stochastic PDEs Using Physics-Informed Neural Networks.” SIAM Journal on Scientific Computing 42 (2): A639–65.

No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.