A unifying framework for various networks, including neural ODEs, where our layers are not simple forward operations but who exacluation is represented as some optimisation problem.

For some info see the NeurIPS 2020 tutorial, Deep Implicit Layers - Neural ODEs, Deep Equilibirum Models, and Beyond, by Zico Kolter, David Duvenaud, and Matt Johnson.

NB: This is different to the implicit *representation* method.
Since implicit layers and implicit representation layers also occur in the same problems (such as ML PDES this terminological confusion will haunt us.

## Optimization layers

Differentiable Convex Optimization Layers introduces cvxpylayers:

Optimization layers add domain-specific knowledge or learnable hard constraints to machine learning models.. Many of these layers solve

convexandconstrainedoptimization problems of the form\[ \begin{array}{rl} x^{\star}(\theta)=\operatorname{argmin}_{x} & f(x ; \theta) \\ \text { subject to } g(x ; \theta) & \leq 0 \\ h(x ; \theta) & =0 \end{array} \]

with parameters θ, objective f, and constraint functions g,h and do end-to-end learning through them with respect to θ.

In this tutorial we introduce our new library cvxpylayers for easily creating differentiable new convex optimization layers. This lets you express your layer with the CVXPY domain specific language as usual and then export the CVXPY object to an efficient batched and differentiable layer with a single line of code. This project turns

everyconvex optimization problem expressed in CVXPY into a differentiable layer.

## Deep declarative networks

A different terminology, although AFAICT closely related technology, is used Stephen Gould in Gould, Hartley, and Campbell (2019), under the banner of Deep Declarative Networks. Fun applications he highlight: robust losses in pooling layers, projection onto shapes, convex programming and warping, matching problems, (relaxed) graph alignment, noisy point-cloud surface reconstruction… (I am sitting in his seminar as I write this.) They have example code (pytorch). He relates some minimax-like optimisations to “Stackelberg games” which are an optimisation problem embedded in game theory.

That provokes certain other ideas: Learning basis decomposition, hyperparameter optimisation… That last one Stephen relates to this one by discussing both problems as “bi-level optimisation problems”.

## Deep equilibrium networks

Related: Deep equilibrium networks (Bai, Kolter, and Koltun 2019; Bai, Koltun, and Kolter 2020). In this one we assume that the network has a single layer which is iterated, and then solve for a fixed point of that iterated layer; this turns out to be memory efficient and in fact powerful (you need to scale that magic layer up, but not so very much.)

Example code: locuslab/deq

## References

*Advances In Neural Information Processing Systems*. http://arxiv.org/abs/1910.12430.

*Mathematical Programming Computation*11 (1): 1–36. https://doi.org/10.1007/s12532-018-0139-4.

*Proceedings of The 28th Conference on Learning Theory*, 40:113–49. Paris, France: PMLR. http://proceedings.mlr.press/v40/Arora15.html.

*Advances in Neural Information Processing Systems*, 32:12. https://openreview.net/forum?id=S1eS4NBgLS.

*Advances in Neural Information Processing Systems*. Vol. 33. http://arxiv.org/abs/2006.08656.

*Proceedings of the 31st International Conference on Neural Information Processing Systems*, 1014–24. NIPS’17. Red Hook, NY, USA: Curran Associates Inc. https://proceedings.neurips.cc/paper/2017/file/192fc044e74dffea144f9ac5dc9f3395-Paper.pdf.

*International Conference on Artificial Intelligence and Statistics*, 318–26. http://machinelearning.wustl.edu/mlpapers/paper_files/AISTATS2012_Domke12.pdf.

*The Implicit Function Theorem*. Springer.

*Proceedings of the 27th International Conference on Machine Learning (ICML-10)*, 399–406. http://yann.lecun.com/exdb/publis/pdf/gregor-icml-10.pdf.

*Inverse Problems*34 (1): 014004. https://doi.org/10.1088/1361-6420/aa9a90.

*Advances in Neural Information Processing Systems*. Vol. 33. https://proceedings.neurips.cc//paper/2020/hash/f1686b4badcf28d33ed632036c7ab0b8-Abstract.html.

*IEEE Transactions on Pattern Analysis and Machine Intelligence*42 (8): 1968–80. https://doi.org/10.1109/TPAMI.2019.2904255.

## No comments yet. Why not leave one?