Fun tricks in non-convex optimisation

2014-10-04 — 2022-07-14

Wherein the role of initialization and symmetries in steering final optima is examined, and phase retrieval is treated while a double-toboggan illustration is employed to show dependence on basins of attraction.

functional analysis
optimization
statmech
Figure 1: In non-convex optimisations our ultimate destination depends upon the starting point.

1 With symmetries

Zhang, Qu, and Wright (2022)

2 In phase retrieval

See phase retrieval.

3 References

Azizian, Iutzeler, Malick, et al. 2025. “The Global Convergence Time of Stochastic Gradient Descent in Non-Convex Landscapes: Sharp Estimates via Large Deviations.”
Choromanska, Henaff, Mathieu, et al. 2015. “The Loss Surfaces of Multilayer Networks.” In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics.
Jain, and Kar. 2017. Non-Convex Optimization for Machine Learning.
Soltanolkotabi, Javanmard, and Lee. 2019. “Theoretical Insights Into the Optimization Landscape of Over-Parameterized Shallow Neural Networks.” IEEE Transactions on Information Theory.
Wright, and Ma. 2022. High-dimensional data analysis with low-dimensional models: Principles, computation, and applications.
Zhang, Qu, and Wright. 2022. “From Symmetry to Geometry: Tractable Nonconvex Problems.”