(Weighted) least squares fits

A classic. Surprisingly deep.

A few non-comprehensive notes to approximating by the arbitrary-but-convenient expedient of minimising the sum of the squares of the deviances.

As used in many many problems. e.g. lasso regression.

  • Nonlinear least squares with ceres-solver:

    Ceres Solve is an open source C++ library for modeling and solving large, complicated optimization problems. It can be used to solve Non-linear Least Squares problems with bounds constraints and general unconstrained optimization problems. It is a mature, feature rich, and performant library that has been used in production at Google since 2010.

  • Boyd and Vandenberghe’s Julia Companion to their Introduction to Applied Linear Algebra: Vectors, Matrices, and Least Squares is a solid introduction to both linear algebra and Julia, focussing especially on least-squares problems.

in mechanism design

TBD. See quadratic voting and public goods provision (Buterin, Hitzig, and Weyl 2019).


Bagge Carlson, Fredrik. 2018. β€œMachine Learning and System Identification for Estimation in Physical Systems.” Thesis/docmono, Lund University.
Bellec, Pierre C., Guillaume LecuΓ©, and Alexandre B. Tsybakov. 2017. β€œTowards the Study of Least Squares Estimators with Convex Penalty.” arXiv:1701.09120 [Math, Stat], January.
Buterin, Vitalik, ZoΓ« Hitzig, and E. Glen Weyl. 2019. β€œA Flexible Design for Funding Public Goods.” Management Science 65 (11): 5171–87.
Chartrand, R., and Wotao Yin. 2008. β€œIteratively Reweighted Algorithms for Compressive Sensing.” In IEEE International Conference on Acoustics, Speech and Signal Processing, 2008. ICASSP 2008, 3869–72.
Chatla, Suneel Babu, and Galit Shmueli. 2016. β€œModeling Big Count Data: An IRLS Framework for CMP Regression and GAM.” arXiv:1610.08244 [Stat], October.
Chen, Xiaojun, Dongdong Ge, Zizhuo Wang, and Yinyu Ye. 2012. β€œComplexity of Unconstrained L_2-L_p.” Mathematical Programming 143 (1-2): 371–83.
Flammarion, Nicolas, and Francis Bach. 2017. β€œStochastic Composite Least-Squares Regression with Convergence Rate O(1/n).” arXiv:1702.06429 [Math, Stat], February.
Friedman, Jerome H. 2002. β€œStochastic Gradient Boosting.” Computational Statistics & Data Analysis, Nonlinear Methods and Data Mining, 38 (4): 367–78.
Friedman, Jerome, Trevor Hastie, Holger HΓΆfling, and Robert Tibshirani. 2007. β€œPathwise Coordinate Optimization.” The Annals of Applied Statistics 1 (2): 302–32.
Friedman, Jerome, Trevor Hastie, and Rob Tibshirani. 2010. β€œRegularization Paths for Generalized Linear Models via Coordinate Descent.” Journal of Statistical Software 33 (1): 1–22.
Gasso, G., A. Rakotomamonjy, and S. Canu. 2009. β€œRecovering Sparse Signals With a Certain Family of Nonconvex Penalties and DC Programming.” IEEE Transactions on Signal Processing 57 (12): 4686–98.
Karampatziakis, Nikos, and John Langford. 2010. β€œOnline Importance Weight Aware Updates.” arXiv:1011.1576 [Cs], November.
Madsen, K, H.B. Nielsen, and O. Tingleff. 2004. β€œMethods for Non-Linear Least Squares Problems.”
Orr, Mark JL. 1996. β€œIntroduction to Radial Basis Function Networks.” Technical Report, Center for Cognitive Science, University of Edinburgh.
Portnoy, Stephen, and Roger Koenker. 1997. β€œThe Gaussian Hare and the Laplacian Tortoise: Computability of Squared-Error Versus Absolute-Error Estimators.” Statistical Science 12 (4): 279–300.
Rhee, Chang-Han, and Peter W. Glynn. 2015. β€œUnbiased Estimation with Square Root Convergence for SDE Models.” Operations Research 63 (5): 1026–43.
Rosset, Saharon, and Ji Zhu. 2007. β€œPiecewise Linear Regularized Solution Paths.” The Annals of Statistics 35 (3): 1012–30.
Transtrum, Mark K, Benjamin B Machta, and James P Sethna. 2011. β€œThe Geometry of Nonlinear Least Squares with Applications to Sloppy Models and Optimization.” Physical Review E 83 (3): 036701.
Yun, Sangwoon, and Kim-Chuan Toh. 2009. β€œA Coordinate Gradient Descent Method for β„“ 1-Regularized Convex Minimization.” Computational Optimization and Applications 48 (2): 273–307.

No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.