Multi-objective optimisation

Optimising for an objective defined as weighted sum of multiple objectives of unknown weights can be difficult. Useful in multi task learning, for example, or in weighting regularisation in regression including neural nets.

HT Cheng Soon Ong for pointing out Jonas Degrave and Ira Korshunova’s illustrated explanation of a tricky thing, Why machine learning algorithms are hard to tune (and the fix). His summary::

Machine learning hyperparameters are hard to tune. One way to think of why it is hard, is because it is a Pareto front of multiple objectives. One way to solve that problem is to look at Lagrange multipliers, as proposed by a paper in 1988 (Platt and Barr 1987).

A follow up post describes how we can make machine learning algorithms tunable.


Das, Indraneel, and John E. Dennis. 1997. “A Closer Look at Drawbacks of Minimizing Weighted Sums of Objectives for Pareto Set Generation in Multicriteria Optimization Problems.” Structural Optimization 14 (1): 63–69.
Jakob, Wilfried, and Christian Blume. 2014. Pareto Optimization or Cascaded Weighted Sum: A Comparison of Concepts.” Algorithms 7 (1): 166–85.
Kim, Il Yong, and O. L. De Weck. 2006. “Adaptive Weighted Sum Method for Multiobjective Optimization: A New Method for Pareto Front Generation.” Structural and Multidisciplinary Optimization 31 (2): 105–16.
Kim, Il Yong, and Oliver L. De Weck. 2005. “Adaptive Weighted-Sum Method for Bi-Objective Optimization: Pareto Front Generation.” Structural and Multidisciplinary Optimization 29 (2): 149–58.
Marler, R., and Jasbir Arora. 2010. The Weighted Sum Method for Multi-Objective Optimization: New Insights.” Structural and Multidisciplinary Optimization 41 (6): 853–62.
Platt, John C., and Alan H. Barr. 1987. Constrained Differential Optimization.” In Proceedings of the 1987 International Conference on Neural Information Processing Systems, 612–21. NIPS’87. Cambridge, MA, USA: MIT Press.
Ryu, Jong-hyun, Sujin Kim, and Hong Wan. 2009. “Pareto Front Approximation with Adaptive Weighted Sum Method in Multiobjective Simulation Optimization.” In Proceedings of the 2009 Winter Simulation Conference (WSC), 623–33. IEEE.

No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.