--- references: - id: Abdel-GawadAdaptive2007 accessed: - year: 2017 month: 8 day: 30 author: - family: Abdel-Gawad given: Ahmed - family: Ratner given: Simon citation-key: Abdel-GawadAdaptive2007 issued: - year: 2007 title: >- Adaptive optimization of hyperparameters in L2-regularised logistic regression type: report URL: >- http://cs229.stanford.edu/proj2007/AbdelGawadRatner-AdaptiveHyperparameterOptimization.pdf - id: AkibaOptuna2019 author: - family: Akiba given: Takuya - family: Sano given: Shotaro - family: Yanase given: Toshihiko - family: Ohta given: Takeru - family: Koyama given: Masanori citation-key: AkibaOptuna2019 container-title: >- Proceedings of the 25rd ACM SIGKDD international conference on knowledge discovery and data mining issued: - year: 2019 title: 'Optuna: A next-generation hyperparameter optimization framework' type: paper-conference - id: BengioGradientBased2000 author: - family: Bengio given: Yoshua citation-key: BengioGradientBased2000 container-title: Neural Computation container-title-short: Neural Computation DOI: 10.1162/089976600300015187 ISSN: 0899-7667 issue: '8' issued: - year: 2000 month: 8 day: 1 page: 1889-1900 title: Gradient-Based Optimization of Hyperparameters type: article-journal URL: http://www.iro.umontreal.ca/~lisa/bib/pub_subject/language/pointeurs/nc.pdf volume: '12' - id: BergstraAlgorithms2011 accessed: - year: 2015 month: 3 day: 27 author: - family: Bergstra given: James S. - family: Bardenet given: Rémi - family: Bengio given: Yoshua - family: Kégl given: Balázs citation-key: BergstraAlgorithms2011 container-title: Advances in Neural Information Processing Systems issued: - year: 2011 page: 2546–2554 publisher: Curran Associates, Inc. title: Algorithms for hyper-parameter optimization type: paper-conference URL: http://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization - id: BergstraMaking2013 author: - family: Bergstra given: J - family: Yamins given: D - family: Cox given: D D citation-key: BergstraMaking2013 container-title: ICML issued: - year: 2013 language: en page: '9' title: >- Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures type: paper-conference - id: BergstraRandom2012 accessed: - year: 2015 month: 3 day: 27 author: - family: Bergstra given: James - family: Bengio given: Yoshua citation-key: BergstraRandom2012 container-title: Journal of Machine Learning Research issued: - year: 2012 page: 281–305 title: Random Search for Hyper-Parameter Optimization type: article-journal URL: http://jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf volume: '13' - id: DomkeGeneric2012 accessed: - year: 2017 month: 8 day: 30 author: - family: Domke given: Justin citation-key: DomkeGeneric2012 container-title: International Conference on Artificial Intelligence and Statistics event-title: Artificial Intelligence and Statistics issued: - year: 2012 page: 318–326 title: Generic Methods for Optimization-Based Modeling type: paper-conference URL: http://proceedings.mlr.press/v22/domke12 - id: EggenspergerEmpirical accessed: - year: 2017 month: 8 day: 21 author: - family: Eggensperger given: Katharina - family: Feurer given: Matthias - family: Hutter given: Frank - family: Bergstra given: James - family: Snoek given: Jasper - family: Hoos given: Holger H. - family: Leyton-Brown given: Kevin citation-key: EggenspergerEmpirical title: >- Towards an Empirical Foundation for Assessing Bayesian Optimization of Hyperparameters type: article-journal URL: http://www.automl.org/papers/13-BayesOpt_EmpiricalFoundation.pdf - id: EigenmannGradient1999 author: - family: Eigenmann given: R. - family: Nossek given: J. A. citation-key: EigenmannGradient1999 container-title: >- Neural Networks for Signal Processing IX: Proceedings of the 1999 IEEE Signal Processing Society Workshop (Cat. No.98TH8468) DOI: 10.1109/NNSP.1999.788126 event-title: >- Neural Networks for Signal Processing IX: Proceedings of the 1999 IEEE Signal Processing Society Workshop (Cat. No.98TH8468) issued: - year: 1999 month: 8 page: 87-94 title: Gradient based adaptive regularization type: paper-conference - id: FeurerAutoSklearn2020 accessed: - year: 2020 month: 9 day: 29 author: - family: Feurer given: Matthias - family: Eggensperger given: Katharina - family: Falkner given: Stefan - family: Lindauer given: Marius - family: Hutter given: Frank citation-key: FeurerAutoSklearn2020 container-title: arXiv:2007.04074 [cs, stat] issued: - year: 2020 month: 7 day: 8 title: 'Auto-Sklearn 2.0: The Next Generation' type: article-journal URL: http://arxiv.org/abs/2007.04074 - id: FeurerEfficient2015 author: - family: Feurer given: Matthias - family: Klein given: Aaron - family: Eggensperger given: Katharina - family: Springenberg given: Jost - family: Blum given: Manuel - family: Hutter given: Frank citation-key: FeurerEfficient2015 container-title: Advances in Neural Information Processing Systems 28 editor: - family: Cortes given: C. - family: Lawrence given: N. D. - family: Lee given: D. D. - family: Sugiyama given: M. - family: Garnett given: R. issued: - year: 2015 page: 2962–2970 publisher: Curran Associates, Inc. title: Efficient and Robust Automated Machine Learning type: paper-conference URL: >- http://papers.nips.cc/paper/5872-efficient-and-robust-automated-machine-learning.pdf - id: FooEfficient2008 accessed: - year: 2017 month: 8 day: 30 author: - family: Foo given: Chuan-sheng - family: Do given: Chuong B. - family: Ng given: Andrew Y. citation-key: FooEfficient2008 container-title: Advances in Neural Information Processing Systems 20 editor: - family: Platt given: J. C. - family: Koller given: D. - family: Singer given: Y. - family: Roweis given: S. T. issued: - year: 2008 page: 377–384 publisher: Curran Associates, Inc. title: Efficient multiple hyperparameter learning for log-linear models type: chapter URL: >- http://papers.nips.cc/paper/3286-efficient-multiple-hyperparameter-learning-for-log-linear-models.pdf - id: FranceschiForward2017 accessed: - year: 2021 month: 4 day: 7 author: - family: Franceschi given: Luca - family: Donini given: Michele - family: Frasconi given: Paolo - family: Pontil given: Massimiliano citation-key: FranceschiForward2017 container-title: International Conference on Machine Learning event-title: International Conference on Machine Learning ISSN: 2640-3498 issued: - year: 2017 month: 7 day: 17 language: en page: 1165-1173 publisher: PMLR title: Forward and Reverse Gradient-Based Hyperparameter Optimization type: paper-conference URL: http://proceedings.mlr.press/v70/franceschi17a.html - id: FranceschiHyperparameter2017 accessed: - year: 2017 month: 4 day: 28 author: - family: Franceschi given: Luca - family: Donini given: Michele - family: Frasconi given: Paolo - family: Pontil given: Massimiliano citation-key: FranceschiHyperparameter2017 event-title: ICLR issued: - year: 2017 title: On Hyperparameter Optimization in Learning Systems type: paper-conference URL: https://arxiv.org/abs/1703.01785 - id: FuDrMAD2016 accessed: - year: 2017 month: 8 day: 20 author: - family: Fu given: Jie - family: Luo given: Hongyin - family: Feng given: Jiashi - family: Low given: Kian Hsiang - family: Chua given: Tat-Seng citation-key: FuDrMAD2016 container-title: PRoceedings of IJCAI, 2016 event-title: IJCAI, 2016 issued: - year: 2016 month: 1 day: 5 title: >- DrMAD: Distilling Reverse-Mode Automatic Differentiation for Optimizing Hyperparameters of Deep Neural Networks type: paper-conference URL: http://arxiv.org/abs/1601.00917 - id: GelbartBayesian2014 author: - family: Gelbart given: Michael A. - family: Snoek given: Jasper - family: Adams given: Ryan P. citation-key: GelbartBayesian2014 collection-title: UAI'14 container-title: >- Proceedings of the Thirtieth Conference on Uncertainty in Artificial Intelligence event-place: Arlington, Virginia, United States ISBN: 978-0-9749039-1-0 issued: - year: 2014 page: 250–259 publisher: AUAI Press publisher-place: Arlington, Virginia, United States title: Bayesian Optimization with Unknown Constraints type: paper-conference URL: http://hips.seas.harvard.edu/files/gelbart-constrained-uai-2014.pdf - id: GrunewalderRegret2010 author: - family: Grünewälder given: Steffen - family: Audibert given: Jean-Yves - family: Opper given: Manfred - family: Shawe-Taylor given: John citation-key: GrunewalderRegret2010 event-title: >- AISTATS 2010 - Thirteenth International Conference on Artificial Intelligence and Statistics issued: - year: 2010 month: 5 day: 13 language: en page: 273-280 title: Regret Bounds for Gaussian Process Bandit Problems type: paper-conference URL: https://hal-enpc.archives-ouvertes.fr/hal-00654517/document volume: '9' - id: HansenCMA2016 accessed: - year: 2020 month: 9 day: 25 author: - family: Hansen given: Nikolaus citation-key: HansenCMA2016 container-title: arXiv:1604.00772 [cs, stat] issued: - year: 2016 month: 4 day: 4 title: 'The CMA Evolution Strategy: A Tutorial' type: article-journal URL: http://arxiv.org/abs/1604.00772 - id: HutterEvaluation2013 author: - family: Hutter given: Frank - family: Hoos given: Holger - family: Leyton-Brown given: Kevin citation-key: HutterEvaluation2013 collection-title: GECCO '13 Companion container-title: >- Proceedings of the 15th Annual Conference Companion on Genetic and Evolutionary Computation DOI: 10.1145/2464576.2501592 event-place: New York, NY, USA ISBN: 978-1-4503-1964-5 issued: - year: 2013 page: 1209–1216 publisher: ACM publisher-place: New York, NY, USA title: >- An Evaluation of Sequential Model-based Optimization for Expensive Blackbox Functions type: paper-conference URL: http://www.cs.ubc.ca/~hutter/papers/13-GECCO-BBOB_SMAC.pdf - id: HutterSequential2011 author: - family: Hutter given: Frank - family: Hoos given: Holger H. - family: Leyton-Brown given: Kevin citation-key: HutterSequential2011 collection-title: Lecture Notes in Computer Science container-title: Learning and Intelligent Optimization DOI: 10.1007/978-3-642-25566-3_40 event-place: Berlin, Heidelberg event-title: International Conference on Learning and Intelligent Optimization ISBN: 978-3-642-25565-6 978-3-642-25566-3 issued: - year: 2011 month: 1 day: 17 language: en page: 507-523 publisher: Springer, Berlin, Heidelberg publisher-place: Berlin, Heidelberg title: Sequential Model-Based Optimization for General Algorithm Configuration type: paper-conference URL: http://cs.ubc.ca/~hutter/papers/11-LION5-SMAC.pdf volume: '6683' - id: JamiesonNonstochastic2015 accessed: - year: 2016 month: 7 day: 27 author: - family: Jamieson given: Kevin - family: Talwalkar given: Ameet citation-key: JamiesonNonstochastic2015 container-title: arXiv:1502.07943 [cs, stat] issued: - year: 2015 month: 2 day: 27 title: Non-stochastic Best Arm Identification and Hyperparameter Optimization type: article-journal URL: http://arxiv.org/abs/1502.07943 - id: KandasamyParallelised2018 accessed: - year: 2020 month: 9 day: 29 author: - family: Kandasamy given: Kirthevasan - family: Krishnamurthy given: Akshay - family: Schneider given: Jeff - family: Poczos given: Barnabas citation-key: KandasamyParallelised2018 container-title: International Conference on Artificial Intelligence and Statistics event-title: International Conference on Artificial Intelligence and Statistics ISSN: 2640-3498 issued: - year: 2018 month: 3 day: 31 language: en page: 133-142 publisher: PMLR title: Parallelised Bayesian Optimisation via Thompson Sampling type: paper-conference URL: http://proceedings.mlr.press/v84/kandasamy18a.html - id: LiEfficient2016 accessed: - year: 2016 month: 7 day: 27 author: - family: Li given: Lisha - family: Jamieson given: Kevin - family: DeSalvo given: Giulia - family: Rostamizadeh given: Afshin - family: Talwalkar given: Ameet citation-key: LiEfficient2016 container-title: arXiv:1603.06560 [cs, stat] issued: - year: 2016 month: 3 day: 21 title: Efficient Hyperparameter Optimization and Infinitely Many Armed Bandits type: article-journal URL: http://arxiv.org/abs/1603.06560 - id: LiHyperband2017 author: - family: Li given: Lisha - family: Jamieson given: Kevin - family: DeSalvo given: Giulia - family: Rostamizadeh given: Afshin - family: Talwalkar given: Ameet citation-key: LiHyperband2017 container-title: The Journal of Machine Learning Research ISSN: 1532-4435 issue: '1' issued: - year: 2017 month: 1 day: 1 page: 6765–6816 title: 'Hyperband: a novel bandit-based approach to hyperparameter optimization' type: article-journal URL: https://jmlr.csail.mit.edu/papers/v18/16-558.html volume: '18' - id: LiSystem2020 accessed: - year: 2020 month: 9 day: 25 author: - family: Li given: Liam - family: Jamieson given: Kevin - family: Rostamizadeh given: Afshin - family: Gonina given: Ekaterina - family: Hardt given: Moritz - family: Recht given: Benjamin - family: Talwalkar given: Ameet citation-key: LiSystem2020 container-title: arXiv:1810.05934 [cs, stat] issued: - year: 2020 month: 3 day: 15 language: en title: A System for Massively Parallel Hyperparameter Tuning type: article-journal URL: http://arxiv.org/abs/1810.05934 - id: LiuDARTS2019 accessed: - year: 2020 month: 10 day: 1 author: - family: Liu given: Hanxiao - family: Simonyan given: Karen - family: Yang given: Yiming citation-key: LiuDARTS2019 container-title: arXiv:1806.09055 [cs, stat] issued: - year: 2019 month: 4 day: 23 title: 'DARTS: Differentiable Architecture Search' type: article-journal URL: http://arxiv.org/abs/1806.09055 - id: LorraineOptimizing2020 accessed: - year: 2020 month: 8 day: 23 author: - family: Lorraine given: Jonathan - family: Vicol given: Paul - family: Duvenaud given: David citation-key: LorraineOptimizing2020 container-title: International Conference on Artificial Intelligence and Statistics event-title: International Conference on Artificial Intelligence and Statistics ISSN: 2640-3498 issued: - year: 2020 month: 6 day: 3 language: en page: 1540-1552 publisher: PMLR title: Optimizing Millions of Hyperparameters by Implicit Differentiation type: paper-conference URL: http://proceedings.mlr.press/v108/lorraine20a.html - id: LorraineStochastic2018 accessed: - year: 2018 month: 4 day: 3 author: - family: Lorraine given: Jonathan - family: Duvenaud given: David citation-key: LorraineStochastic2018 container-title: arXiv:1802.09419 [cs] issued: - year: 2018 month: 2 day: 26 title: Stochastic Hyperparameter Optimization through Hypernetworks type: article-journal URL: http://arxiv.org/abs/1802.09419 - id: MacKayComparison1999 accessed: - year: 2014 month: 12 day: 19 author: - family: MacKay given: David JC citation-key: MacKayComparison1999 container-title: Neural computation DOI: 10.1162/089976699300016331 ISSN: 0899-7667 issue: '5' issued: - year: 1999 page: 1035–1068 title: Comparison of approximate methods for handling hyperparameters type: article-journal URL: http://www.mitpressjournals.org/doi/abs/10.1162/089976699300016331 volume: '11' - id: MaclaurinGradientbased2015 accessed: - year: 2022 month: 4 day: 23 author: - family: Maclaurin given: Dougal - family: Duvenaud given: David - family: Adams given: Ryan citation-key: MaclaurinGradientbased2015 container-title: Proceedings of the 32nd International Conference on Machine Learning event-title: International Conference on Machine Learning ISSN: 1938-7228 issued: - year: 2015 month: 6 day: 1 language: en page: 2113-2122 publisher: PMLR title: Gradient-based Hyperparameter Optimization through Reversible Learning type: paper-conference URL: https://proceedings.mlr.press/v37/maclaurin15.html - id: MockusBayesian1975 accessed: - year: 2022 month: 8 day: 11 author: - family: Močkus given: J. citation-key: MockusBayesian1975 collection-title: Lecture Notes in Computer Science container-title: >- Optimization Techniques IFIP Technical Conference: Novosibirsk, July 1–7, 1974 DOI: 10.1007/978-3-662-38527-2_55 editor: - family: Marchuk given: G. I. event-place: Berlin, Heidelberg ISBN: 978-3-662-38527-2 issued: - year: 1975 language: en page: 400-404 publisher: Springer publisher-place: Berlin, Heidelberg title: On Bayesian Methods for Seeking the Extremum type: paper-conference - id: OHaganCurve1978 accessed: - year: 2020 month: 10 day: 21 author: - family: O’Hagan given: A. citation-key: OHaganCurve1978 container-title: 'Journal of the Royal Statistical Society: Series B (Methodological)' DOI: 10.1111/j.2517-6161.1978.tb01643.x ISSN: 2517-6161 issue: '1' issued: - year: 1978 language: en page: 1-24 publisher: Wiley Online Library title: Curve Fitting and Optimal Design for Prediction type: article-journal volume: '40' - id: PlattConstrained1987 accessed: - year: 2022 month: 10 day: 9 author: - family: Platt given: John C. - family: Barr given: Alan H. citation-key: PlattConstrained1987 collection-title: NIPS'87 container-title: >- Proceedings of the 1987 International Conference on Neural Information Processing Systems event-place: Cambridge, MA, USA issued: - year: 1987 month: 1 day: 1 page: 612–621 publisher: MIT Press publisher-place: Cambridge, MA, USA title: Constrained differential optimization type: paper-conference URL: https://openreview.net/forum?id=BJ-h_uZ_bB - id: RealAutoMLZero2020 accessed: - year: 2020 month: 3 day: 15 author: - family: Real given: Esteban - family: Liang given: Chen - family: So given: David R. - family: Le given: Quoc V. citation-key: RealAutoMLZero2020 issued: - year: 2020 month: 3 day: 6 language: en title: 'AutoML-Zero: Evolving Machine Learning Algorithms From Scratch' type: article-journal URL: https://arxiv.org/abs/2003.03384v1 - id: SalimansMarkov2015 accessed: - year: 2017 month: 8 day: 30 author: - family: Salimans given: Tim - family: Kingma given: Diederik - family: Welling given: Max citation-key: SalimansMarkov2015 collection-title: ICML'15 container-title: >- Proceedings of the 32nd International Conference on Machine Learning (ICML-15) event-place: Lille, France event-title: International Conference on Machine Learning ISSN: 1938-7228 issued: - year: 2015 language: en page: 1218–1226 publisher: JMLR.org publisher-place: Lille, France section: Machine Learning title: 'Markov chain monte carlo and variational inference: Bridging the gap' type: paper-conference URL: http://proceedings.mlr.press/v37/salimans15.html - id: SnoekInput2014 author: - family: Snoek given: Jasper - family: Swersky given: Kevin - family: Zemel given: Rich - family: Adams given: Ryan citation-key: SnoekInput2014 container-title: >- Proceedings of the 31st International Conference on Machine Learning (ICML-14) event-title: International Conference on Machine Learning issued: - year: 2014 page: 1674–1682 title: Input Warping for Bayesian Optimization of Non-Stationary Functions type: paper-conference URL: http://www.jmlr.org/proceedings/papers/v32/snoek14.pdf - id: SnoekPractical2012 author: - family: Snoek given: Jasper - family: Larochelle given: Hugo - family: Adams given: Ryan P. citation-key: SnoekPractical2012 container-title: Advances in neural information processing systems issued: - year: 2012 page: 2951–2959 publisher: Curran Associates, Inc. title: Practical bayesian optimization of machine learning algorithms type: paper-conference URL: http://arxiv.org/abs/1206.2944 - id: SrinivasGaussian2012 author: - family: Srinivas given: Niranjan - family: Krause given: Andreas - family: Kakade given: Sham M. - family: Seeger given: Matthias citation-key: SrinivasGaussian2012 container-title: IEEE Transactions on Information Theory DOI: 10.1109/TIT.2011.2182033 ISSN: 0018-9448, 1557-9654 issue: '5' issued: - year: 2012 month: 5 page: 3250-3265 title: >- Gaussian Process Optimization in the Bandit Setting: No Regret and Experimental Design type: article-journal URL: http://arxiv.org/abs/0912.3995 volume: '58' - id: SwerskyMultiTask2013 author: - family: Swersky given: Kevin - family: Snoek given: Jasper - family: Adams given: Ryan P citation-key: SwerskyMultiTask2013 container-title: Advances in Neural Information Processing Systems 26 editor: - family: Burges given: C. J. C. - family: Bottou given: L. - family: Welling given: M. - family: Ghahramani given: Z. - family: Weinberger given: K. Q. issued: - year: 2013 page: 2004–2012 publisher: Curran Associates, Inc. title: Multi-Task Bayesian Optimization type: paper-conference URL: http://papers.nips.cc/paper/5086-multi-task-bayesian-optimization.pdf - id: ThorntonAutoWEKA2013 accessed: - year: 2017 month: 7 day: 17 author: - family: Thornton given: Chris - family: Hutter given: Frank - family: Hoos given: Holger H. - family: Leyton-Brown given: Kevin citation-key: ThorntonAutoWEKA2013 collection-title: KDD '13 container-title: >- Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining DOI: 10.1145/2487575.2487629 event-place: New York, NY, USA ISBN: 978-1-4503-2174-7 issued: - year: 2013 page: 847–855 publisher: ACM publisher-place: New York, NY, USA title: >- Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms type: paper-conference URL: https://arxiv.org/abs/1208.3719 - id: TurnerBayesian2021 accessed: - year: 2021 month: 4 day: 29 author: - family: Turner given: Ryan - family: Eriksson given: David - family: McCourt given: Michael - family: Kiili given: Juha - family: Laaksonen given: Eero - family: Xu given: Zhen - family: Guyon given: Isabelle citation-key: TurnerBayesian2021 container-title: arXiv:2104.10201 [cs, stat] issued: - year: 2021 month: 4 day: 20 title: >- Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020 type: article-journal URL: http://arxiv.org/abs/2104.10201 - id: WangRethinking2020 accessed: - year: 2021 month: 4 day: 13 author: - family: Wang given: Ruochen - family: Cheng given: Minhao - family: Chen given: Xiangning - family: Tang given: Xiaocheng - family: Hsieh given: Cho-Jui citation-key: WangRethinking2020 event-title: International Conference on Learning Representations issued: - year: 2020 month: 9 day: 28 language: en title: Rethinking Architecture Selection in Differentiable NAS type: paper-conference URL: https://openreview.net/forum?id=PKubaeJkw3 ...