\[\renewcommand{\var}{\operatorname{Var}}
\renewcommand{\cov}{\operatorname{Cov}}
\renewcommand{\dd}{\mathrm{d}}
\renewcommand{\bb}[1]{\mathbb{#1}}
\renewcommand{\vv}[1]{\boldsymbol{#1}}
\renewcommand{\rv}[1]{\mathsf{#1}}
\renewcommand{\vrv}[1]{\vv{\rv{#1}}}
\renewcommand{\disteq}{\stackrel{d}{=}}
\renewcommand{\gvn}{\mid}
\renewcommand{\Ex}{\mathbb{E}}
\renewcommand{\Pr}{\mathbb{P}}
\renewcommand{\one}{\unicode{x1D7D9}}\]
Training neural networks by ensemble Kalman updates instead of SGD. Arises naturally from the dynamical perspective on neural networks. TBD.
Claudia Schilling’s filter (Schillings and Stuart 2017) is an elegant variant of the ensemble Kalman Filter which looks somehow more general than the original but also simpler and may be applicable. Haber, Lucka, and Ruthotto (2018) use it to train neural nets (!) and show a rather beautiful connection to stochastic gradient descent in section 3.2.
References
Chada, Iglesias, Roininen, et al. 2018.
“Parameterizations for Ensemble Kalman Inversion.” Inverse Problems.
Chen, Chong, Dou, Chen, et al. 2022.
“A Novel Neural Network Training Framework with Data Assimilation.” The Journal of Supercomputing.
Chen, Yuming, Sanz-Alonso, and Willett. 2022.
“Autodifferentiable Ensemble Kalman Filters.” SIAM Journal on Mathematics of Data Science.
Dunbar, Duncan, Stuart, et al. 2022.
“Ensemble Inference Methods for Models With Noisy and Expensive Likelihoods.” SIAM Journal on Applied Dynamical Systems.
Haykin, ed. 2001.
Kalman Filtering and Neural Networks. Adaptive and Learning Systems for Signal Processing, Communications, and Control.
Huang, Schneider, and Stuart. 2022.
“Iterated Kalman Methodology for Inverse Problems.” Journal of Computational Physics.
Iglesias, Law, and Stuart. 2013.
“Ensemble Kalman Methods for Inverse Problems.” Inverse Problems.
Schillings, and Stuart. 2017.
“Analysis of the Ensemble Kalman Filter for Inverse Problems.” SIAM Journal on Numerical Analysis.
Taghvaei, and Mehta. 2021.
“An Optimal Transport Formulation of the Ensemble Kalman Filter.” IEEE Transactions on Automatic Control.