# Neural learning dynamical systems

August 13, 2018 — May 23, 2023

Learning to approximate differential equations and other interpreatble physical dynamics with neural nets. Related: Analysing a neural net itself *as* a dynamical system, which is not quite the same but crosses over, or learning general recurrent dynamics. Variational state filters. Where the parameters are meaningful, not just weights, we tend to think about system identification.

A deterministic version of this problem is what e.g. the famous Vector Institute Neural ODE paper (T. Q. Chen et al. 2018) did. Author Duvenaud argues that in some ways the hype ran away with the Neural ODE paper, and credits CasADI with the innovations.

There are various laypersons’ introductions/ tutorials in this area, including the simple and practical magical take in julia. See also the CASADI example.

Learning an ODE in particular a purely deterministic process, feels unsatisfying; We want a model which encodes responses,and effects to interactions. It is not ideal to have time series models which need to encode everything in an initial state.

Also, we would prefer models to be stochastic. Learnable *SDEs* are probably what we want. I’m particularly interested on jump ODE regression.

Homework: Duvenaud again, tweeting some explanatory animations.

Note connection to reparameterization tricks, in that neural ODEs give you cheap differentiable reparameterizations.

Gu et al. (2021) unifies neural ODEs with RNNs.

## 1 Questions

How do you do ensemble training for posterior predictives in NODEs? How do you guarantee stability in the learned dynamics?

## 2 Recursive estimation

See recursive identification for generic theory of learning under the distribution shift induced by a moving parameter vector.

## 3 S4

Interesting package of tools from Christopher Ré’s lab, at the intersection of recurrent networks and linear feedback systems. See HazyResearch/state-spaces: Sequence Modeling with Structured State Spaces. I find these aesthetically satisfying, because I spent 2 years of my PhD trying to solve the same problem, and failed. These folks did a better job, so I find it slightly validating that the idea was not stupid.

## 4 Incoming

- google-research/torchsde: Differentiable SDE solvers with GPU support and efficient sensitivity analysis. (Kidger et al. 2021; X. Li et al. 2020)
- Patrick Kidger’s thesis is the current canonical textbook on ODE learning (Kidger 2022).
- Corenflos et al. (2021) describe an optimal transport method
- Campbell et al. (2021) describes variational inference that factors out the unknown parameters.

## 5 References

*Mathematical Programming Computation*.

*Acta Numerica*.

*Proceedings of the National Academy of Sciences*.

*arXiv:1812.05916 [Math, q-Fin, Stat]*.

*PLOS ONE*.

*Proceedings of ICLR*.

*PRoceedings of ICLR*.

*Advances in Neural Information Processing Systems*.

*Nature Computational Science*.

*Advances in Neural Information Processing Systems 31*.

*Advances in Neural Information Processing Systems*.

*arXiv:2102.07850 [Cs, Stat]*.

*Advances in Neural Information Processing Systems*.

*Advances in Neural Information Processing Systems*.

*arXiv:1904.01681 [Cs, Stat]*.

*Communications in Mathematics and Statistics*.

*Notices of the American Mathematical Society*.

*Scandinavian Journal of Statistics*.

*arXiv:1807.01083 [Cs, Math]*.

*ICML*.

*Advances in Neural Information Processing Systems*.

*arXiv:1807.01613 [Cs, Stat]*.

*arXiv:1902.10298 [Cs]*.

*Advances in Neural Information Processing Systems*.

*arXiv:2007.04154 [Cs, q-Fin, Stat]*.

*Physical Review Research*.

*arXiv:1810.01367 [Cs, Stat]*.

*Advances in Neural Information Processing Systems*.

*arXiv:1805.08034 [Cs, Math]*.

*Proceedings of the National Academy of Sciences*.

*IMA Note*.

*arXiv:2006.04439 [Cs, Stat]*.

*Nature Machine Intelligence*.

*PRoceedings of ICLR*.

*Advances in Neural Information Processing Systems*.

*arXiv:1812.04300 [Math, Stat]*.

*Advances in Neural Information Processing Systems 32*.

*Advances in Neural Information Processing Systems*.

*Proceedings of the 38th International Conference on Machine Learning*.

*Proceedings of the 38th International Conference on Machine Learning*.

*arXiv:2005.08926 [Cs, Stat]*.

*Machine Learning and the Physical Sciences Workshop at the 33rd Conference on Neural Information Processing Systems (NeurIPS)*.

*Advances in Neural Information Processing Systems*.

*Physical Review. X*.

*arXiv:1612.06212 [Cs]*.

*Advances in Neural Information Processing Systems*.

*International Conference on Artificial Intelligence and Statistics*.

*Advances in Neural Information Processing Systems*.

*Advances in Neural Information Processing Systems*.

*arXiv:1910.03193 [Cs, Stat]*.

*Advances in Neural Information Processing Systems*.

*Nature Communications*.

*arXiv:2003.08063 [Cs, Math, Stat]*.

*arXiv:2002.08071 [Cs, Stat]*.

*PMLR*.

*arXiv:2109.00173 [Cs, Stat]*.

*arXiv:1904.12933 [Quant-Ph, Stat]*.

*arXiv:1905.10437 [Cs, Stat]*.

*Bulletin of the American Mathematical Society*.

*Workshop on Bayesian Deep LEarning*.

*International Conference on Artificial Intelligence and Statistics*.

*Advances in Neural Information Processing Systems*.

*arXiv:2009.09346 [Cs]*.

*arXiv:1812.01892 [Cs]*.

*arXiv.org*.

*ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*.

*arXiv:1905.12090 [Cs, Stat]*.

*Journal of Mathematical Imaging and Vision*.

*arXiv:1910.09349 [Cs, Stat]*.

*Proceedings of the 37th International Conference on Machine Learning*.

*arXiv:2103.10153 [Cs, Stat]*.

*arXiv:2012.08405 [Cs, Eess]*.

*CoRR*.

*arXiv:1906.10264 [Cs, Stat]*.

*bioRxiv*.

*Physics-Based Deep Learning*.

*Proceedings of the Web Conference 2021*.

*arXiv:1905.09883 [Cs, Stat]*.

*PMLR*.

*arXiv:1805.08349 [Cond-Mat, Stat]*.

*SIAM Journal on Scientific Computing*.

*arXiv:1905.10994 [Cs, Stat]*.

*Spatial Statistics*.

*arXiv:1907.12998 [Cs, Stat]*.

*International Conference on Machine Learning*.