Neural nets with implicit layers


A unifying framework for various networks, including neural ODEs, where we our layers are not simple forward operations but evaluate some optimisation problem in themselves.

For some info see the NeurIPS 2020 tutorial, Deep Implicit Layers - Neural ODEs, Deep Equilibirum Models, and Beyond, created by Zico Kolter, David Duvenaud, and Matt Johnson.

NB this is different to the implicit representation method.

Differentiable Convex Optimization Layers introduces cvxpylayers:

Optimization layers add domain-specific knowledge or learnable hard constraints to machine learning models. Many of these layers solve convex and constrained optimization problems of the form

\[ \begin{array}{rl} x^{\star}(\theta)=\operatorname{argmin}_{x} & f(x ; \theta) \\ \text { subject to } g(x ; \theta) & \leq 0 \\ h(x ; \theta) & =0 \end{array} \]

with parameters θ, objective f, and constraint functions g,h and do end-to-end learning through them with respect to θ.

In this tutorial we introduce our new library cvxpylayers for easily creating differentiable new convex optimization layers. This lets you express your layer with the CVXPY domain specific language as usual and then export the CVXPY object to an efficient batched and differentiable layer with a single line of code. This project turns every convex optimization problem expressed in CVXPY into a differentiable layer.

References

Amos, Brandon, and J. Zico Kolter. 2017. OptNet: Differentiable Optimization as a Layer in Neural Networks,” March. https://arxiv.org/abs/1703.00443v4.
Amos, Brandon, Ivan Dario Jimenez Rodriguez, Jacob Sacks, Byron Boots, and J. Zico Kolter. 2018. “Differentiable MPC for End-to-End Planning and Control,” October. https://arxiv.org/abs/1810.13400v3.
Andersson, Joel A. E., Joris Gillis, Greg Horn, James B. Rawlings, and Moritz Diehl. 2019. CasADi: A Software Framework for Nonlinear Optimization and Optimal Control.” Mathematical Programming Computation 11 (1): 1–36. https://doi.org/10.1007/s12532-018-0139-4.
Arora, Sanjeev, Rong Ge, Tengyu Ma, and Ankur Moitra. 2015. “Simple, Efficient, and Neural Algorithms for Sparse Coding.” In Proceedings of The 28th Conference on Learning Theory, 40:113–49. Paris, France: PMLR. http://proceedings.mlr.press/v40/Arora15.html.
Bai, Shaojie, J Zico Kolter, and Vladlen Koltun. n.d. “Deep Equilibrium Models,” 12.
Bai, Shaojie, Vladlen Koltun, and J. Zico Kolter. 2020. “Multiscale Deep Equilibrium Models.” In Advances in Neural Information Processing Systems. Vol. 33. https://proceedings.neurips.cc//paper/2020/hash/3812f9a59b634c2a9c574610eaba5bed-Abstract.html.
Barratt, Shane. 2018. “On the Differentiability of the Solution to Convex Optimization Problems,” April. https://arxiv.org/abs/1804.05098v3.
Border, KC. 2019. “Notes on the Implicit Function Theorem.” http://www.its.caltech.edu/~kcborder/Notes/IFT.pdf.
Djolonga, Josip, and Andreas Krause. n.d. “Differentiable Learning of Submodular Models,” 11.
Domke, Justin. 2012. “Generic Methods for Optimization-Based Modeling.” In International Conference on Artificial Intelligence and Statistics, 318–26. http://machinelearning.wustl.edu/mlpapers/paper_files/AISTATS2012_Domke12.pdf.
Donti, Priya L., Brandon Amos, and J. Zico Kolter. 2017. “Task-Based End-to-End Model Learning in Stochastic Optimization,” March. https://arxiv.org/abs/1703.04529v4.
G.Krantz, Steven, and Harold R.Parks. 2002. The Implicit Function Theorem. Springer.
Gould, Stephen, Basura Fernando, Anoop Cherian, Peter Anderson, Rodrigo Santa Cruz, and Edison Guo. 2016. “On Differentiating Parameterized Argmin and Argmax Problems with Application to Bi-Level Optimization,” July. https://arxiv.org/abs/1607.05447v2.
Gould, Stephen, Richard Hartley, and Dylan Campbell. 2019. “Deep Declarative Networks: A New Hope,” September. https://arxiv.org/abs/1909.04866v2.
Haber, Eldad, and Lars Ruthotto. 2018. “Stable Architectures for Deep Neural Networks.” Inverse Problems 34 (1): 014004. https://doi.org/10.1088/1361-6420/aa9a90.
Landry, Benoit, Joseph Lorenzetti, Zachary Manchester, and Marco Pavone. 2019. “Bilevel Optimization for Planning Through Contact: A Semidirect Method,” June. https://arxiv.org/abs/1906.04292v2.
Lee, Kwonjoon, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. 2019. “Meta-Learning with Differentiable Convex Optimization,” April. https://arxiv.org/abs/1904.03758v2.
Mena, Gonzalo, David Belanger, Scott Linderman, and Jasper Snoek. 2018. “Learning Latent Permutations with Gumbel-Sinkhorn Networks,” February. https://arxiv.org/abs/1802.08665v1.
Poli, Michael, Stefano Massaroli, Atsushi Yamashita, Hajime Asama, and Jinkyoo Park. 2020. “Hypersolvers: Toward Fast Continuous-Depth Models.” In Advances in Neural Information Processing Systems. Vol. 33. https://proceedings.neurips.cc//paper/2020/hash/f1686b4badcf28d33ed632036c7ab0b8-Abstract.html.
Rajeswaran, Aravind, Chelsea Finn, Sham Kakade, and Sergey Levine. 2019. “Meta-Learning with Implicit Gradients,” September. https://arxiv.org/abs/1909.04630v1.
Sulam, Jeremias, Aviad Aberdam, Amir Beck, and Michael Elad. 2020. “On Multi-Layer Basis Pursuit, Efficient Algorithms and Convolutional Neural Networks.” IEEE Transactions on Pattern Analysis and Machine Intelligence 42 (8): 1968–80. https://doi.org/10.1109/TPAMI.2019.2904255.
Wang, Po-Wei, Priya L. Donti, Bryan Wilder, and Zico Kolter. 2019. SATNet: Bridging Deep Learning and Logical Reasoning Using a Differentiable Satisfiability Solver,” May. https://arxiv.org/abs/1905.12149v1.