Crawling through alien landscapes in the fog, looking for mountain peaks.

I’m mostly interested in continuous optimisation, but, you know, combinatorial optimisation is a whole thing.

A vast topic, with many sub-topics. I have neither the time nor the expertise to construct a detailed map of these As Moritz Hardt observes (and this is just in the convex context),

It’s easy to spend a semester of convex optimization on various guises of gradient descent alone. Simply pick one of the following variants and work through the specifics of the analysis: conjugate, accelerated, projected, conditional, mirrored, stochastic, coordinate, online. This is to name a few. You may also choose various pairs of attributes such as “accelerated coordinate” descent. Many triples are also valid such as “online stochastic mirror” descent. An expert unlike me would know exactly which triples are admissible. You get extra credit when you use “subgradient” instead of “gradient”. This is really only the beginning of optimization and it might already seem confusing.

When I was even younger and yet more foolish I decided the divide was between online optimization and offline optimization, which in hindsight is neither a clear nor useful taxonomy for the problems facing me. Now there are more tightly topical pages, such as gradient descent, and 2nd order methods, surrogate optimisation, constrained optimisation, and I shall create additional such as circumstances demand.

TODO: insert brief taxonomy here.

🏗 Diagram.

See Zeyuan Allen-Zhu and Elad Hazan on their teaching strategy which also gives a split into 16 different areas:

The following dilemma is encountered by many of my friends when teaching basic optimization: which variant/proof of gradient descent should one start with? Of course, one needs to decide on which depth of convex analysis one should dive into, and decide on issues such as “should I define strong-convexity?”, “discuss smoothness?”, “Nesterov acceleration?”, etc.

[…] If one wishes to go into more depth, usually in convex optimization courses, one covers the full spectrum of different smoothness/ strong-convexity/ acceleration/ stochasticity regimes, each with a separate analysis (a total of 16 possible configurations!)

This year I’ve tried something different in COS511 @ Princeton, which turns out also to have research significance. We’ve covered basic GD for well-conditioned functions, i.e. smooth and strongly-convex functions, and then extended these result by reduction to all other cases! A (simplified) outline of this teaching strategy is given in chapter 2 of Introduction to Online Convex Optimization.

Classical Strong-Convexity and Smoothness Reductions:

Given any optimization algorithm A for the well-conditioned case (i.e., strongly convex and smooth case), we can derive an algorithm for smooth but not strongly functions as follows.

Given a non-strongly convex but smooth objective \(f\), define a objective by \(f_1(x)=f(x)+e\|x\|^2\).

It is straightforward to see that \(f_1\) differs from \(f\) by at most ϵ times a distance factor, and in addition it is ϵ-strongly convex. Thus, one can apply A to minimize \(f_1\) and get a solution which is not too far from the optimal solution for \(f\) itself. This simplistic reduction yields an almost optimal rate, up to logarithmic factors.

Keywords: Complimentary slackness theorem, High or very high dimensional methods, approximate method, Lagrange multipliers, primal and dual problems, fixed point methods, gradient, subgradient, proximal gradient, optimal control problems, convexity, sparsity, ways to avoid wrecking finding the extrema of perfectly simple little 10000-parameter functions before everyone observes that I am a fool in the guise of a mathematician but everyone is not there because I wandered off the optimal path hours ago, and now I am alone and lost in a valley of lower-case Greek letters.

See also geometry of fitness landscapes, expectation maximisation, matrix factorisations, discrete optimisation, nature-inspired “meta-heuristic” optimisation.

## Brief intro material

- Luca Tevisan, Posts on online optimisation.
- Zeyuan ALLEN-ZHU: Recent Advances in Stochastic Convex and Non-Convex Optimization. Clear, has good pointers.
- Basic but enlightening, John Nash’s graphical explanation of R’s optimization
- Martin Jaggi’s Optimization in two hours
- Celebrated union of optimisation and economics, market complexity

## Textbooks

Whole free textbooks online. Mostly convex.

- K. Madsen, H.B. Nielsen, O. Tingleff, Methods for Non-linear Least Squares Problems is super simple for least-squares type optimisations
- Aharon Ben-Tal and Arkadi Nemirovski’s lectures on modern convex optimization
- Arkadi Nemirovski, Interior point polynomial time methods in convex programming
- Boyd and Vandenberghe’s influential Convex Optimization
- Bubeck, S. (2014).
Convex Optimization: Algorithms and Complexity.
*arXiv:1405.4980 [cs, Math, Stat]*. based on Bubeck’s course notes - Elad Hazan’s Introduction to Online Convex Optimization.

## To file

- Elad Hazan and Satyan Kale’s tutorial on online convex optimisation.
- Elad Hazan’s Introduction to Online Convex Optimization.
- Suvrit Sra’s eye-bleeding ugly but pertinent introduction to this stuff
- Francis Bach’s slides on practical ML SGD.

## Alternating Direction Method of Multipliers

Dunno. It’s everywhere, though. (S. Boyd 2010)

In this review, we argue that the alternating direction method of multipliers is well suited to distributed convex optimization, and in particular to large-scale problems arising in statistics, machine learning, and related areas. The method was developed in the 1970s, with roots in the 1950s, and is equivalent or closely related to many other algorithms, such as dual decomposition, the method of multipliers, Douglas—Rachford splitting, Spingarn’s method of partial inverses, Dykstra’s alternating projections, Bregman iterative algorithms for \(\ell_1\) problems, proximal methods, and others. After briefly surveying the theory and history of the algorithm, we discuss applications to a wide variety of statistical and machine learning problems of recent interest, including the lasso, sparse logistic regression, basis pursuit, covariance selection, support vector machines, and many others. We also discuss general distributed optimization, extensions to the nonconvex setting, and efficient implementation, including some details on distributed MPI and Hadoop Map Reduce implementations.

## Optimisation on manifolds

See Nicolas Boumen’s introductory blog post.

Optimization on manifolds is about solving problems of the form

\[\mathrm{minimize}_{x\in\mathcal{M}} f(x),\]

where \(\mathcal{M}\) is a nice, known manifold. By “nice”, I mean a smooth, finite-dimensional Riemannian manifold.

Practical examples include the following (and all possible products of these):

- Euclidean spaces
- The sphere (set of vectors or matrices with unit Euclidean norm)
- The Stiefel manifold (set of orthonormal matrices)
- The Grassmann manifold (set of linear subspaces of a given dimension; this is a quotient space)
- The rotation group (set of orthogonal matrices with determinant +1)
- The manifold of fixed-rank matrices
- The same, further restricted to positive semidefinite matrices
- The cone of (strictly) positive definite matrices
- …
Conceptually, the key point is to think of optimization on manifolds as unconstrained optimization: we do not think of \(\mathcal{M}\) as being embedded in a Euclidean space. Rather, we think of \(\mathcal{M}\) as being “the only thing that exists,” and we strive for intrinsic methods. Besides making for elegant theory, it also makes it clear how to handle abstract spaces numerically (such as the Grassmann manifold for example); and it gives algorithms the “right” invariances (computations do not depend on an arbitrarily chosen representation of the manifold).

There are at least two reasons why this class of problems is getting much attention lately. First, it is because optimization problems over the aforementioned sets (mostly matrix sets) come up pervasively in applications, and at some point it became clear that the intrinsic viewpoint leads to better algorithms, as compared to general-purpose constrained optimization methods (where \(\mathcal{M}\) is considered as being inside a Euclidean space \(\mathcal{E}\), and algorithms move in \(\mathcal{E}\), while penalizing distance to \(\mathcal{M}\)). The second is that, as I will argue momentarily, Riemannian manifolds are “the right setting” to talk about unconstrained optimization. And indeed, there is a beautiful book by [Absil, Sepulchre, Mahony], called

Optimization algorithms on matrix manifolds(freely available), that shows how the classical methods for unconstrained optimization (gradient descent, Newton, trust-regions, conjugate gradients…) carry over seamlessly to the more general Riemannian framework.

## Gradient-free optimization

Not all the methods described here use gradient information, but it’s frequently assumed to be something you can access easily. It’s worth considering which objectives you can optimize easily

But not all objectives are easily differentiable, even when parameters are
continuous.
For example, if you are not getting your measurement from a mathematical model,
but *from a physical experiment* you can’t differentiate it since reality
itself is usually not analytically differentiable.
In this latter case, you are getting close to a question of online experiment
design, as in ANOVA, and a further constraint that
your function evaluations are possibly stupendously expensive.
See Bayesian optimisation for
one approach to this i the context of experiment design.

In general situations like this we use gradient-free methods, such as simulated annealing or numerical gradient etc.

### “Meta-heuristic” methods

Biologically-inspired or arbitrary. Evolutionary algorithms, particle swarm optimisation, ant colony optimisation, harmony search. A lot of the tricks from these are adopted into mainstream stochastic methods. Some not.

See biometic algorithms for the care and husbandry of such as those.

### Annealing and Monte Carlo optimisation methods

Simulated annealing: Constructing a process to yield maximally-likely estimates for the parameters. This has a statistical mechanics justification that makes it attractive to physicists; But it’s generally useful. You don’t necessarily need a gradient here, just the ability to evaluate something interpretable as a “likelihood ratio”. Long story. I don’t yet cover this at Monte Carlo methods but I should.

Elad Hazan The two cultures of optimization:

The standard curriculum in high school math includes elementary functional analysis, and methods for finding the minima, maxima and saddle points of a single dimensional function. When moving to high dimensions, this becomes beyond the reach of your typical high-school student: mathematical optimization theory spans a multitude of involved techniques in virtually all areas of computer science and mathematics.

Iterative methods, in particular, are the most successful algorithms for large-scale optimization and the most widely used in machine learning. Of these, most popular are first-order gradient-based methods due to their very low per-iteration complexity.

However, way before these became prominent, physicists needed to solve large scale optimization problems, since the time of the Manhattan project at Los Alamos. The problems that they faced looked very different, essentially simulation of physical experiments, as were the solutions they developed. The Metropolis algorithm is the basis for randomized optimization methods and Markov Chain Monte Carlo algorithms. […]

In our recent paper (Abernethy and Hazan 2016), we show that for convex optimization, the heat path and central path for IPM for a particular barrier function (called the entropic barrier, following the terminology of the recent excellent work of Bubeck and Eldan) are identical! Thus, in some precise sense, the two cultures of optimization have been studied the same object in disguise and using different techniques.

### Expectation maximization

## Parallel

Classic, basic SGD takes walks through the data set example-wise or feature-wise — but this doesn’t work in parallel, so you tend to go for mini-batch gradient descent so that you can at least vectorize. Apparently you can make SGD work in “true” parallel across communication-constrained cores, but I don’t yet understand how.

## Implementations

Specialised optimisation software.

See also statistical software, and gradient descent

GENO (Soeren Laue, Mitterreiter, and Giesen 2019; Sören Laue, Blacher, and Giesen 2022)

GENO provides optimization solvers for everyone. You can enter your optimization problem in an easy-to-read modeling language in the code editor below. Python code is then generated automatically that can solve this class of optimization problems on the CPU or on the GPU. The automatically generated solvers are often as fast as handwritten, specialized solvers…

The GENO solver combines an Augmented Lagrangian approach with a limited memory quasi-Newton method (L-BFGS-B) that can handle also bound constraints on the variables. Quasi-Newton methods are very efficient for problems involving thousands of optimization variables. The GENO solver is then instantiated by the automatically generated methods for computing function values and gradients that are provided by this website to solve the specified class of optimization problems. This approach is very well suited for optimization problems originating from classical machine learning problems.

Looks useful for an interesting class of semidefinite programming problems.

ensmallen (Bhardwaj et al. 2021)

We present ensmallen, a fast and flexible C++ library for mathematical optimization of arbitrary user-supplied functions, which can be applied to many machine learning problems. Several types of optimizations are supported, including differentiable, separable, constrained, and categorical objective functions. The library provides many pre-built optimizers (including numerous variants of SGD and Quasi-Newton optimizers) as well as a flexible framework for implementing new optimizers and objective functions. Implementation of a new optimizer requires only one method and a new objective function requires typically one or two C++ functions. This can aid in the quick implementation and prototyping of new machine learning algorithms. Due to the use of C++ template metaprogramming, ensmallen is able to support compiler optimizations that provide fast runtimes. Empirical comparisons show that ensmallen is able to outperform other optimization frameworks (like Julia and SciPy), sometimes by large margins. The library is distributed under the BSD license and is ready for use in production environments.

SPORCO a Python package for solving optimisation problems with sparsity-inducing regularisation. These consist primarily of sparse coding and dictionary learning problems, including convolutional sparse coding and dictionary learning, but there is also support for other problems such as Total Variation regularisation and Robust PCA. In the current version, all of the optimisation algorithms are based on the Alternating Direction Method of Multipliers (ADMM).

scipy.optimise.minimize: The python default. Includes many different algorithms than can do whatever you want. Failure modes are opaque, online-only and they don’t support warm-restarts, which is a thing for me, but a good starting point unless you have reason to prefer others. (i.e. if all your data does not fit in RAM, don’t bother.)

SPAMS (SPArse Modeling Software) is an optimization toolbox for solving various sparse estimation problems. Dictionary learning and matrix factorization (NMF, sparse PCA, …) Solving sparse decomposition problems with LARS, coordinate descent, OMP, SOMP, proximal methods Solving structured sparse decomposition problems (\(ell_1/ell_2,\) \(\ell_1/\ell_\infty,\) sparse group lasso, tree-structured regularization structured sparsity with overlapping groups,…). It is developped by Julien Mairal, with the collaboration of Francis Bach, Jean Ponce, Guillermo Sapiro, Rodolphe Jenatton and Guillaume Obozinski. It is coded in C++ with a Matlab interface. Recently, interfaces for R and Python have been developed by Jean-Paul Chieze (INRIA), and archetypal analysis was written by Yuansi Chen (UC Berkeley).

…is a user friendly interface to several conic and integer programming solvers, very much like YALMIP or CVX under MATLAB.

The main motivation for PICOS is to have the possibility to enter an optimization problem as a high level model, and to be able to solve it with several different solvers. Multidimensional and matrix variables are handled in a natural fashion, which makes it painless to formulate a SDP or a SOCP. This is very useful for educational purposes, and to quickly implement some models and test their validity on simple examples.

also maintains a list of other solvers.

Manifold optimisation implementations (for e.g. learning on manifolds)

… is a free software package for convex optimization based on the Python programming language. It can be used with the interactive Python interpreter, on the command line by executing Python scripts, or integrated in other software via Python extension modules. Its main purpose is to make the development of software for convex optimization applications straightforward by building on Python’s extensive standard library and on the strengths of Python as a high-level programming language. […]

efficient Python classes for dense and sparse matrices (real and complex), with Python indexing and slicing and overloaded operations for matrix arithmetic

an interface to most of the double-precision real and complex BLAS

an interface to LAPACK routines for solving linear equations and least-squares problems, matrix factorisations (LU, Cholesky, LDLT and QR), symmetric eigenvalue and singular value decomposition, and Schur factorization

an interface to the fast Fourier transform routines from FFTW

interfaces to the sparse LU and Cholesky solvers from UMFPACK and CHOLMOD

routines for linear, second-order cone, and semidefinite programming problems

routines for nonlinear convex optimization

interfaces to the linear programming solver in GLPK, the semidefinite programming solver in DSDP5, and the linear, quadratic and second-order cone programming solvers in MOSEK

a modeling tool for specifying convex piecewise-linear optimization problems.

seems to reinvent half of numpy and scipy. Also seems to be used by the all the other python packages.

Pyomo is a Python-based open-source software package that supports a diverse set of optimization capabilities for formulating, solving, and analyzing optimization models.

A core capability of Pyomo is modeling structured optimization applications. Pyomo can be used to define general symbolic problems, create specific problem instances, and solve these instances using commercial and open-source solvers. Pyomo’s modeling objects are embedded within a full-featured high-level programming language providing a rich set of supporting libraries, which distinguishes Pyomo from other algebraic modeling languages like AMPL, AIMMS and GAMS.…

Pyomo was formerly released as the Coopr software library.

…is a Python-embedded modeling language for convex optimization problems. It allows you to express your problem in a natural way that follows the math, rather than in the restrictive standard form required by solvers.

So it’s a DSL for convex constraint programming. Can be extended heuristically to nonconvex constraints by…

… is a package for modeling and solving problems with convex objectives and decision variables from a nonconvex set. This package provides heuristic such as NC-ADMM (a variation of alternating direction method of multipliers for nonconvex problems) and relax-round-polish, which can be viewed as a majorization-minimization algorithm. The solver methods provided and the syntax for constructing problems are discussed in our associated paper.

… is a free/open-source library for nonlinear optimization, providing a common interface for a number of different free optimization routines available online as well as original implementations of various other algorithms. Its features include:

Callable from C, C++, Fortran, Matlab or GNU Octave, Python, GNU Guile, Julia, GNU R, Lua, and OCaml.

A common interface for many different algorithms—try a different algorithm just by changing one parameter.

Support for large-scale optimization (some algorithms scalable to millions of parameters and thousands of constraints)…

Algorithms using function values only (derivative-free) and also algorithms exploiting user-supplied gradients.

…(pronounced tee-fox) provides a set of Matlab templates, or building blocks, that can be used to construct efficient, customized solvers for a variety of convex models, including in particular those employed in sparse recovery applications. It was conceived and written by Stephen Becker, Emmanuel J. Candès and Michael Grant.

stan is famous for Monte Carlo sampling, but also does deterministic optimisation using automatic differentiation. this is a luxurious “full service” option, although with limited scope for customisation; Curious how it performs in very high dimensions, as L-BFGS does not scale forever.

Optimization algorithms:

Limited-memory BFGS (Stan’s default optimization algorithm)

BFGS

Laplace’s method for classical standard error estimates and approximate Bayesian posteriors

Optim.jl is a generic optimizer for julia

JuMP.jl is a domain-specific modeling language for mathematical optimization embedded in Julia. It currently supports a number of open-source and commercial solvers (Bonmin, Cbc, Clp, Couenne, CPLEX, ECOS, FICO Xpress, GLPK, Gurobi, Ipopt, KNITRO, MOSEK, NLopt, SCS, BARON) for a variety of problem classes, including linear programming, (mixed) integer programming, second-order conic programming, semidefinite programming, and nonlinear programming.

NLsolve.jl solves systems of nonlinear equations. […]

The package is also able to solve mixed complementarity problems, which are similar to systems of nonlinear equations, except that the equality to zero is allowed to become an inequality if some boundary condition is satisfied. See further below for a formal definition and the related commands.

Since there is some overlap between optimizers and nonlinear solvers, this package borrows some ideas from the Optim package, and depends on it for linesearch algorithms.

Many of these solvers optionally use commercial backends such as Mosek.

## Non-convex

## To file

### Miscellaneous optimisation techniques suggested on Linkedin

The whole world of exotic specialized optimisers. See, e.g. Nuit Blanche name-dropping Bregmann iteration, alternating method, augmented Lagrangian…

### Primal/dual problems

🏗

### Majorization-minorization

🏗

### Difference-of-Convex-objectives

When your objective function is not convex but you can represent it in terms of convex functions, somehow or other, use DC optimisation. (Gasso, Rakotomamonjy, and Canu 2009) (I don’t think this guarantees you a global optimum, but rather faster convergence to a local one)

## References

*arXiv Preprint arXiv:1603.04467*.

*International Conference on Machine Learning*, 2520–28. PMLR.

*Nature*435 (7043): 759–64.

*Journal of Machine Learning Research*15 (1): 1111–33.

*arXiv:1602.03943 [Cs, Stat]*, February.

*Advances in Neural Information Processing Systems 29*, edited by D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, 1606–14. Curran Associates, Inc.

*arXiv:1709.05069 [Cs]*.

*arXiv:1309.3117 [Cs, Math]*, September.

*arXiv:1306.2119 [Cs, Math, Stat]*, 773–81.

*Optimization With Sparsity-Inducing Penalties*. Foundations and Trends(r) in Machine Learning 1.0. Now Publishers Inc.

*Advances in Neural Information Processing Systems (NIPS)*, –. Spain.

*Neural Computation*4 (2): 141–66.

*SIAM Journal on Imaging Sciences*2 (1): 183–202.

*SIAM Journal on Optimization*15 (3): 780–804.

*Mathematical Programming*149 (1-2): 301–27.

*Journal of Machine Learning Research*10 (December): 1737–54.

*arXiv:1607.01981 [Cs, Stat]*, July.

*Electronic Journal of Statistics*9 (2): 2058–75.

*Proceedings of Neuro-Nîmes 91*. Nimes, France: EC2.

*Online Learning and Neural Networks*, edited by David Saad, 17:142. Cambridge, UK: Cambridge University Press.

*Proceedings of the 19th International Conference on Computational Statistics (COMPSTAT’2010)*, 177–86. Paris, France: Springer.

*Neural Networks: Tricks of the Trade*, 421–36. Lecture Notes in Computer Science. Springer, Berlin, Heidelberg.

*Advances in Neural Information Processing Systems*, edited by J.C. Platt, D. Koller, Y. Singer, and S. Roweis, 20:161–68. NIPS Foundation (http://books.nips.cc).

*arXiv:1606.04838 [Cs, Math, Stat]*, June.

*Advances in Neural Information Processing Systems 16*, edited by Sebastian Thrun, Lawrence Saul, and Bernhard Schölkopf. Cambridge, MA: MIT Press.

*Journal of Machine Learning Research*15: 1455–59.

*Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers*. Vol. 3. Now Publishers Inc.

*Convex Optimization*. Cambridge, UK ; New York: Cambridge University Press.

*Convex Optimization: Algorithms and Complexity*. Vol. 8. Foundations and Trends in Machine Learning. Now Publishers.

*arXiv:1412.1587 [Cs, Math]*, December.

*IEEE Signal Processing Magazine*31 (5): 32–43.

*IEEE International Conference on Acoustics, Speech and Signal Processing, 2008. ICASSP 2008*, 3869–72.

*Mathematical Programming*134 (1): 71–99.

*Mathematical Programming*143 (1-2): 371–83.

*Advances In Neural Information Processing Systems*.

*Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics*, 192–204.

*arXiv:0809.0660 [Stat]*, September.

*Inverse Problems*24 (6): 065014.

*arXiv:1704.04752 [Math, Stat]*, April.

*Advances in Neural Information Processing Systems 27*, 2933–41. Curran Associates, Inc.

*Advances in Neural Information Processing Systems 27*.

*Handbooks in Operations Research and Management Science*, 1:1–72. Optimization. Elsevier.

*Numerical Methods for Unconstrained Optimization and Nonlinear Equations*. Vol. 16. Classics in Applied Mathematics. Society for Industrial and Applied Mathematics.

*Acta Numerica*7 (January): 51–150.

*Journal of Machine Learning Research*12 (Jul): 2121–59.

*The Computer Journal*6 (2): 163–68.

*Progress in Aerospace Sciences*45 (1–3): 50–79.

*SIAM Journal on Scientific Computing*34 (3): A1380–1405.

*Computational Statistics & Data Analysis*, Nonlinear Methods and Data Mining, 38 (4): 367–78.

*The Annals of Applied Statistics*1 (2): 302–32.

*Journal of Statistical Software*33 (1): 1–22.

*Algebra, Topology, Diﬀerential Calculus, and Optimization Theory For Computer Science and Machine Learning*.

*IEEE Transactions on Signal Processing*57 (12): 4686–98.

*SIAM Journal on Optimization*23 (4): 2341–68.

*arXiv:1310.3787 [Math]*, October.

*Advances in Automatic Differentiation*, edited by Christian H. Bischof, H. Martin Bücker, Paul Hovland, Uwe Naumann, and Jean Utke, 64:35–44. Berlin, Heidelberg: Springer Berlin Heidelberg.

*Journal of the Society for Industrial and Applied Mathematics Series A Control*3 (1): 147–51.

*arXiv:1501.04979 [Cs, Math]*, January.

*Optimization Stories: 21st International Symposium on Mathematical Programming, Berlin, August 19 - 24, 2012*. Documenta Mathematica, 2012 : Extra vol. Bielefeld.

*Mathematical Programming*152 (1-2): 75–112.

*Advances in Neural Information Processing Systems 28*, edited by C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, 1594–1602. Curran Associates, Inc.

*Stochastic Processes and Their Applications*2 (4): 359–70.

*arXiv Preprint arXiv:1506.07677*.

*Advances in Neural Information Processing Systems*, 781–89. Curran Associates, Inc.

*Journal of Machine Learning Research*, 427–35.

*IEEE Transactions on Signal Processing*62 (4): 868–82.

*Signal Processing*125 (August): 274–89.

*Proceeding of ICLR*.

*University of California at Berkeley, Computer Science Division*.

*The Annals of Statistics*31 (2): 391–406.

*Advances in Neural Information Processing Systems 21*, edited by D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, 905–12. Curran Associates, Inc.

*Advances in Neural Information Processing Systems*. Vol. 32. Curran Associates, Inc.

*Proceedings of the AAAI Conference on Artificial Intelligence*, 36:7300–7308.

*arXiv:1611.04831 [Cs, Math, Stat]*, November.

*Advances in Neural Information Processing Systems 29*, edited by D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, 4988–96. Curran Associates, Inc.

*arXiv:1610.00960 [Math, Stat]*.

*arXiv:1503.08316 [Cs]*, March.

*arXiv Preprint arXiv:1512.04039*.

*arXiv:1703.10622 [Cs, Stat]*, March.

*SIAM Journal on Optimization*25 (2): 829–55.

*Advances in Neural Information Processing Systems*, 2283–91.

*International Conference on Machine Learning*, 783–91.

*Sparse Modeling for Image and Vision Processing*. Vol. 8.

*Proceedings of the 27th International Conference on International Conference on Machine Learning*, 735–42. ICML’10. USA: Omnipress.

*Proceedings of the 28th International Conference on International Conference on Machine Learning*, 1033–40. ICML’11. USA: Omnipress.

*Neural Networks: Tricks of the Trade*, 479–535. Lecture Notes in Computer Science. Springer.

*IEEE Signal Processing Magazine*27 (3): 50–61.

*Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining*, 1222–30. KDD ’13. New York, NY, USA: ACM.

*arXiv:1607.06534 [Stat]*, July.

*arXiv:2009.11243 [Cs, Stat]*, September.

*arXiv:1605.09774 [Cs, Math, Stat]*, May.

*Interpretable Machine Learning*.

*SIAM Journal on Optimization*22 (2): 341–62.

*Mathematical Programming*112 (1): 159–81.

*Mathematical Programming*140 (1): 125–61.

*Introductory Lectures on Convex Optimization*. Vol. 87. Applied Optimization. Boston, MA: Springer US.

*arXiv:1705.07261 [Cs, Math, Stat]*, May.

*Numerical Optimization*. 2nd ed. Springer Series in Operations Research and Financial Engineering. New York: Springer-Verlag.

*Proximal Algorithms*. Vol. 1.

*arXiv:1702.00317 [Cs, Math, Stat]*, February.

*Journal of Machine Learning Research*17 (53): 1–38.

*SIAM Journal on Control and Optimization*30 (4): 838–55.

*Statistical Science*12 (4): 279–300.

*Progress in Aerospace Sciences*41 (1): 1–28.

*PMLR*, 1603:314–23.

*The Annals of Mathematical Statistics*22 (3): 400–407.

*Optimizing Methods in Statistics*, edited by Jagdish S. Rustagi, 233–57. Academic Press.

*The Annals of Statistics*35 (3): 1012–30.

*arXiv:1609.04747 [Cs]*, September.

*The Annals of Statistics*13 (1): 236–45.

*arXiv:1412.6615 [Cs, Stat]*, December.

*Advances in Neural Information Processing Systems 29*, edited by D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, 901–1. Curran Associates, Inc.

*University of British Columbia, Technical Report TR-2009*19.

*Neural Computation*14 (7): 1723–38.

*Journal of Machine Learning Research*12 (July): 1865–92.

*arXiv:1610.00040 [Math, Stat]*, September.

*Journal of Statistical Software*39 (5).

*arXiv:1512.04011 [Cs]*, December.

*IEEE Transactions on Automatic Control*45 (10): 1839–53.

*arXiv:1601.02712 [Cs, Math, Stat]*, January.

*arXiv:1906.06821 [Cs, Math, Stat]*, October.

*Statistical Science*12 (4): 296–98.

*Journal of Machine Learning Research*17 (137): 1–5.

*Proceedings of the 23rd International Conference on Machine Learning*.

*Annual Review of Statistics and Its Application*1 (1): 233–53.

*arXiv:1509.03616 [Math]*, September.

*Proceedings of the National Academy of Sciences*113 (47): E7351–58.

*Microsoft Research*, July.

*IEEE Transactions on Signal Processing*57 (7): 2479–93.

*Optimization for Data Analysis*. New York: Cambridge University Press.

*The Annals of Applied Statistics*2 (1): 224–44.

*arXiv:1107.2490 [Cs]*, July.

*arXiv:1502.03032 [Cs, Math, Stat]*, February.

*Computational Optimization and Applications*48 (2): 273–307.

*arXiv:1511.03766 [Cs]*, November.

*arXiv:1701.00481 [Stat]*, January.

*Proceedings of the Twentieth International Conference on International Conference on Machine Learning*, 928–35. ICML’03. Washington, DC, USA: AAAI Press.

*Advances in Neural Information Processing Systems 23*, edited by J. D. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R. S. Zemel, and A. Culotta, 2595–2603. Curran Associates, Inc.

## No comments yet. Why not leave one?