Method of Adjoints for differentiating through ODEs

September 15, 2017 — May 15, 2023

Bayes
dynamical systems
linear algebra
probability
signal processing
state space models
statistics
time series
Figure 1

Constructing a backward (P)DE which effectively gives us the gradients of the forward (P)DE. A trick in automatic differentiation which happens to be useful in differentiating likelihood (or other functions) of time-evolving systems. This is an active area of research (Kidger, Chen, and Lyons 2021; Kidger et al. 2020; Li et al. 2020; Rackauckas et al. 2018; Stapor, Fröhlich, and Hasenauer 2018; Cao et al. 2003), but also old and well-studied [Errico (1997);

Nick McGreivy, The adjoint method for PDE-constrained optimization: the conclusions of my 9-month struggle to understand the word adjoint. I’ll be talking about how automatic differentiation (AD) can help us better understand the adjoint method, and vice versa.

1 References

Cao, Li, Petzold, et al. 2003. “Adjoint Sensitivity Analysis for Differential-Algebraic Equations: The Adjoint DAE System and Its Numerical Solution.” SIAM Journal on Scientific Computing.
Carpenter, Hoffman, Brubaker, et al. 2015. “The Stan Math Library: Reverse-Mode Automatic Differentiation in C++.” arXiv Preprint arXiv:1509.07164.
Errico. 1997. “What Is an Adjoint Model?” Bulletin of the American Meteorological Society.
Gahungu, Lanyon, Álvarez, et al. 2022. “Adjoint-Aided Inference of Gaussian Process Driven Differential Equations.” In.
Giles. 2008. “Collected Matrix Derivative Results for Forward and Reverse Mode Algorithmic Differentiation.” In Advances in Automatic Differentiation.
Innes. 2018. “Don’t Unroll Adjoint: Differentiating SSA-Form Programs.” arXiv:1810.07951 [Cs].
Ionescu, Vantzos, and Sminchisescu. 2016. “Training Deep Networks with Structured Layers by Matrix Backpropagation.”
Johnson. 2012. “Notes on Adjoint Methods for 18.335.”
Kavvadias, Papoutsis-Kiachagias, and Giannakoglou. 2015. “On the Proper Treatment of Grid Sensitivities in Continuous Adjoint Methods for Shape Optimization.” Journal of Computational Physics.
Kidger, Chen, and Lyons. 2021. “‘Hey, That’s Not an ODE’: Faster ODE Adjoints via Seminorms.” In Proceedings of the 38th International Conference on Machine Learning.
Kidger, Morrill, Foster, et al. 2020. “Neural Controlled Differential Equations for Irregular Time Series.” arXiv:2005.08926 [Cs, Stat].
Li, Wong, Chen, et al. 2020. “Scalable Gradients for Stochastic Differential Equations.” In International Conference on Artificial Intelligence and Statistics.
Margossian, Vehtari, Simpson, et al. 2020. “Hamiltonian Monte Carlo Using an Adjoint-Differentiated Laplace Approximation: Bayesian Inference for Latent Gaussian Models and Beyond.” arXiv:2004.12550 [Stat].
Mitusch, Funke, and Dokken. 2019. “Dolfin-Adjoint 2018.1: Automated Adjoints for FEniCS and Firedrake.” Journal of Open Source Software.
Papoutsis-Kiachagias, Evangelos. 2013. “Adjoint Methods for Turbulent Flows, Applied to Shape or Topology Optimization and Robust Design.”
Papoutsis-Kiachagias, E. M., and Giannakoglou. 2016. “Continuous Adjoint Methods for Turbulent Flows, Applied to Shape and Topology Optimization: Industrial Applications.” Archives of Computational Methods in Engineering.
Papoutsis-Kiachagias, E. M., Magoulas, Mueller, et al. 2015. “Noise Reduction in Car Aerodynamics Using a Surrogate Objective Function and the Continuous Adjoint Method with Wall Functions.” Computers & Fluids.
Rackauckas, Ma, Dixit, et al. 2018. “A Comparison of Automatic Differentiation and Continuous Sensitivity Analysis for Derivatives of Differential Equation Solutions.” arXiv:1812.01892 [Cs].
Stapor, Fröhlich, and Hasenauer. 2018. “Optimization and Uncertainty Analysis of ODE Models Using 2nd Order Adjoint Sensitivity Analysis.” bioRxiv.