Learning stochastic differential equations with neural nets.
Related: Analysing a neural net itself *as* a dynamical system, which is not quite the same but crosses over.
Variational state filters.

A deterministic version of this problem is what e.g. the famous Vector Institute Neural ODE paper (Chen et al. 2018) did. Author Duvenaud argues that in some ways the hype ran away with the Neural ODE paper, and credits CasADI with the innovations.

There are various laypersons’ introductions/ tutorials in this area, including the simple and practical magical take in julia. See also the CASADI example.

Learning an ODE in particular a purely deterministic process, feels unsatisfying; We want a model which encodes responses,and effects to interactions. It is not ideal to have time series models which need to encode everything in an initial state.

Also, we would prefer models to be stochastic.
Learnable *SDEs* are probably what we want.
I’m particularly interested on
jump ODE regression.

Homework: Duvenaud again, tweeting some explanatory animations.

Turtorials on this are available. (Rackauckas et al. 2018; Niu, Horesh, and Chuang 2019), and even some tutorial implementations by the indefatigable Chris Rackauckas, and a whole MIT course. Chris Rackauckas’ lecture notes christen this development “scientific machine learning”.

Learning stochastic partial differential equations where a whole random field evolves in time is something of interest to me; see spatiotemporal nets.

Note connection to reparameterization tricks, in that neural ODEs give you cheap differentiable reparameterizations.

## Questions

How do you do ensemble training for posterior predictives in NODEs? How do you guarantee stability in the learned dynamics?

## References

*Mathematical Programming Computation*11 (1): 1–36.

*Acta Numerica*28 (May): 1–174.

*Proceedings of the National Academy of Sciences*111 (52): 18507–12.

*arXiv:1812.05916 [Math, q-Fin, Stat]*, January.

*Proceedings of ICLR*.

*PRoceedings of ICLR*.

*Advances in Neural Information Processing Systems 31*, edited by S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, 6572–83. Curran Associates, Inc.

*Advances in Neural Information Processing Systems*. Vol. 33.

*Advances in Neural Information Processing Systems*. Vol. 33.

*arXiv:1904.01681 [Cs, Stat]*, April.

*Communications in Mathematics and Statistics*5 (1): 1–11.

*arXiv:1807.01083 [Cs, Math]*, July.

*Scandinavian Journal of Statistics*n/a (n/a).

*ICML*, 14.

*Advances in Neural Information Processing Systems*. Vol. 33.

*arXiv:1807.01613 [Cs, Stat]*, July, 10.

*arXiv:1902.10298 [Cs]*, February.

*Advances in Neural Information Processing Systems*. Vol. 33.

*arXiv:2007.04154 [Cs, q-Fin, Stat]*, July.

*arXiv:1810.01367 [Cs, Stat]*, October.

*arXiv:2110.13985 [Cs]*, October.

*arXiv:1805.08034 [Cs, Math]*, May.

*Proceedings of the National Academy of Sciences*115 (34): 8505–10.

*IMA Note*.

*arXiv:2006.04439 [Cs, Stat]*, December.

*PRoceedings of ICLR*.

*Advances in Neural Information Processing Systems*. Vol. 33.

*arXiv:1812.04300 [Math, Stat]*, December.

*Advances in Neural Information Processing Systems 32*, edited by H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alché-Buc, E. Fox, and R. Garnett, 9847–58. Curran Associates, Inc.

*Advances in Neural Information Processing Systems*. Vol. 33.

*Advances In Neural Information Processing Systems*, 6.

*arXiv:2005.08926 [Cs, Stat]*, November.

*Advances in Neural Information Processing Systems*, 9.

*arXiv:2007.14823 [Cond-Mat, Physics:nlin, q-Bio]*.

*Advances in Neural Information Processing Systems*. Vol. 33.

*International Conference on Artificial Intelligence and Statistics*, 3870–82. PMLR.

*Advances in Neural Information Processing Systems*. Vol. 33.

*arXiv:1906.08324 [Cs, Stat]*, June.

*arXiv:1910.03193 [Cs, Stat]*, April.

*Advances in Neural Information Processing Systems*. Vol. 33.

*arXiv:2003.08063 [Cs, Math, Stat]*, March.

*arXiv:2002.08071 [Cs, Stat]*.

*PMLR*, 2401–9.

*arXiv:2109.00173 [Cs, Stat]*, August.

*arXiv:1904.12933 [Quant-Ph, Stat]*, April.

*arXiv:1905.10437 [Cs, Stat]*, February.

*Bulletin of the American Mathematical Society*80 (3): 503–5.

*Workshop on Bayesian Deep LEarning*, 7.

*International Conference on Artificial Intelligence and Statistics*, 1126–36. PMLR.

*Advances in Neural Information Processing Systems*. Vol. 33.

*arXiv:2009.09346 [Cs]*, September.

*The Winnower*, August.

*arXiv:1812.01892 [Cs]*, December.

*arXiv:2001.04385 [Cs, Math, q-Bio, Stat]*, August.

*ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 3905–9.

*arXiv:1905.12090 [Cs, Stat]*, May.

*arXiv:1804.04272 [Cs, Math, Stat]*, April.

*arXiv:1910.09349 [Cs, Stat]*, March.

*arXiv:2002.09405 [Physics, Stat]*.

*arXiv:2012.08405 [Cs, Eess]*, December.

*CoRR*abs/2006.09313.

*arXiv:1906.10264 [Cs, Stat]*, June.

*bioRxiv*, February, 272005.

*Physics-Based Deep Learning*. WWW.

*Proceedings of the Web Conference 2021*, 730–42. Ljubljana Slovenia: ACM.

*arXiv:1905.09883 [Cs, Stat]*, October.

*PMLR*, 3570–78.

*arXiv:1805.08349 [Cond-Mat, Stat]*, October.

*SIAM Journal on Scientific Computing*42 (1): A292–317.

*arXiv:1905.10994 [Cs, Stat]*, October.

*Spatial Statistics*37 (June): 100408.

*arXiv:1907.12998 [Cs, Stat]*, February.

## No comments yet. Why not leave one?