System identification in continuous time

Learning in continuous ODEs, SDEs and CDEs

August 1, 2016 — November 30, 2023

calculus
dynamical systems
Hilbert space
Lévy processes
machine learning
neural nets
regression
sciml
SDEs
signal processing
sparser than thou
statistics
stochastic processes
time series

Learning the parameters of a dynamical system in continuous time. Gets extra tricky for stochastic systems.

1 Recursive estimation

See recursive identification for generic theory of learning a Markovian under the distribution shift induced by a moving unobserved state vector.

See the writings of the indefatigable Chris Rackauckas , plus his tutorial implementations and whole whole MIT course. Chris Rackauckas’ lecture notes christen this area “scientific machine learning”, which is a confusing and ambiguous choice, but let us run with it.

TBD.

4 In PDEs

Learning stochastic partial differential equations where a whole random field evolves in time is something of interest to me; see spatiotemporal nets and spatiotemporal dynamics for more on that theme.

See differentiable PDE solvers for now.

5 With sparse SDEs

For least-squares system identification see sparse stochastic processes.

TBD

9 Method of adjoints

A trick in differentiation which happens to be useful in differentiating likelihood (or other functions) of time evolving systems e.g. Errico (1997). If the system is deterministic this is not too bad. It is complicated in stoah

For now, see the method of adjoints in the autodiff notebook.

10 Tooling

10.1 Python

Diffrax

Diffrax is a JAX-based library providing numerical differential equation solvers.

Features include:

• ODE/SDE/CDE (ordinary/stochastic/controlled) solvers
• lots of different solvers (including Tsit5, Dopri8, symplectic solvers, implicit solvers)
• vmappable everything (including the region of integration)
• using a PyTree as the state
• dense solutions
• multiple adjoint methods for backpropagation
• support for neural differential equations.

From a technical point of view, the internal structure of the library is pretty cool — all kinds of equations (ODEs, SDEs, CDEs) are solved in a unified way (rather than being treated separately), producing a small tightly-written library.

torchdyn (docs).

10.2 Julia

Chris Rauckackas is a veritable wizard with this stuff; read his blog.

Here is a tour of fun tricks with stochastic PDEs. There is a lot of tooling for this; DiffEqOperators … does something. DiffEqFlux (EZ neural ODEs works with Flux and claims to make neural SDE simple.

+1 for Julia here.

12 References

Anderson. 1982. Stochastic Processes and Their Applications.
Batz, Ruttor, and Opper. 2017. arXiv:1702.05390 [Physics, Stat].
Baydin, and Pearlmutter. 2014. arXiv:1404.7456 [Cs, Stat].
Beck, E, and Jentzen. 2019. Journal of Nonlinear Science.
Chang, Chen, Haber, et al. 2019. In Proceedings of ICLR.
Chen, Rubanova, Bettencourt, et al. 2018. In Advances in Neural Information Processing Systems 31.
Choromanski, Davis, Likhosherstov, et al. 2020. In Advances in Neural Information Processing Systems.
Dandekar, Chung, Dixit, et al. 2021. arXiv:2012.07244 [Cs].
Errico. 1997. Bulletin of the American Meteorological Society.
Gholami, Keutzer, and Biros. 2019. arXiv:1902.10298 [Cs].
Gierjatowicz, Sabate-Vidales, Šiška, et al. 2020. arXiv:2007.04154 [Cs, q-Fin, Stat].
Grathwohl, Chen, Bettencourt, et al. 2018. arXiv:1810.01367 [Cs, Stat].
Gu, Johnson, Goel, et al. 2021. In Advances in Neural Information Processing Systems.
Hirsh, Barajas-Solano, and Kutz. 2022. Royal Society Open Science.
Jia, and Benson. 2019. In Advances in Neural Information Processing Systems 32.
Kelly, Bettencourt, Johnson, et al. 2020. In.
Kidger, Morrill, Foster, et al. 2020. arXiv:2005.08926 [Cs, Stat].
Li, Yuhong, Cai, Zhang, et al. 2022.
Li, Yang, and Duan. 2021. arXiv:2107.10127 [Math, Stat].
Li, Xuechen, Wong, Chen, et al. 2020. In International Conference on Artificial Intelligence and Statistics.
Ljung. 2010. Annual Reviews in Control.
Lu, Ariño, and Soljačić. 2021. arXiv:2107.10879 [Physics].
Malartic, Farchi, and Bocquet. 2021. arXiv:2107.11253 [Nlin, Physics:physics, Stat].
Marelli. 2007. IEEE Transactions on Signal Processing.
Massaroli, Poli, Park, et al. 2020. In arXiv:2002.08071 [Cs, Stat].
Morrill, Kidger, Salvi, et al. 2020. “Neural CDEs for Long Time Series via the Log-ODE Method.” In.
Nabian, and Meidani. 2019. Probabilistic Engineering Mechanics.
Nguyen, and Malinsky. 2020. “Exploration and Implementation of Neural Ordinary Diﬀerential Equations.”
Pham, and Panaretos. 2016. arXiv:1612.07197 [Math, Stat].
Pillonetto. 2016. arXiv:1612.09158 [Cs, Stat].
Rackauckas. 2019.
Rackauckas, Ma, Dixit, et al. 2018. arXiv:1812.01892 [Cs].
Rackauckas, Ma, Martensen, et al. 2020. arXiv.org.
Ramsundar, Krishnamurthy, and Viswanathan. 2021. arXiv:2109.07573 [Physics].
Särkkä. 2011. In Artificial Neural Networks and Machine Learning – ICANN 2011. Lecture Notes in Computer Science.
Särkkä, and Solin. 2019. Applied Stochastic Differential Equations. Institute of Mathematical Statistics Textbooks 10.
Schmidt, Krämer, and Hennig. 2021. arXiv:2103.10153 [Cs, Stat].
Song, Durkan, Murray, et al. 2021. In Advances in Neural Information Processing Systems.
Song, Sohl-Dickstein, Kingma, et al. 2022. In.
Um, Brand, Fei, et al. 2021. arXiv:2007.00016 [Physics].
Um, and Holl. 2021. “Differentiable Physics for Improving the Accuracy of Iterative PDE-Solvers with Neural Networks.” In.
Unser, and Tafti. 2014. An Introduction to Sparse Stochastic Processes.
van Delft, and Eichler. 2016. arXiv:1602.05125 [Math, Stat].
Wedig. 1984. Nuclear Engineering and Design.
Werbos. 1988. Neural Networks.
Yıldız, Heinonen, and Lähdesmäki. 2019. arXiv:1905.10994 [Cs, Stat].
Yoshida. 2022. Statistical Inference for Stochastic Processes.
Zhang, Guo, and Karniadakis. 2020. SIAM Journal on Scientific Computing.