The sub-field of optimisation that specifically aims to automate model selection in machine learning. (and also occasionally ensemble construction)
There are two major approaches that I am aware of, both of which are related in a kind of abstract way, but which are in practice different
- Finding the right architecture for your nueral net, a.k.a architecture search
- Hyperparameter optimisation which I have made into a separate notebook.
The first one I might cover .
TODO: work out if this is the same as βmeta learningβ? I think not; I suspect that of being transfer learning.
Reinforcement learning approaches
Quoc Le & Barret Zoph discuss using reinforcement learning to learn neural models:
Typically, our machine learning models are painstakingly designed by a team of engineers and scientists. This process of manually designing machine learning models is difficult because the search space of all possible models can be combinatorially large β a typical 10-layer network can have ~1010 candidate networks! [β¦]
To make this process of designing machine learning models much more accessible, weβve been exploring ways to automate the design of machine learning models. [β¦] in this blog post, weβll focus on our reinforcement learning approach and the early results weβve gotten so far.
In our approach (which we call βAutoMLβ), a controller neural net can propose a βchildβ model architecture, which can then be trained and evaluated for quality on a particular task. That feedback is then used to inform the controller how to improve its proposals for the next round.
Differentiable architecture search
Various tricks I am not so familiar with. Overview in Elsken, Metzen, and Hutter (2019). See, e.g. Weight sharing for neural architecture search. ποΈ
I think there is a bunch of stuff implemented in archai (blogged).
Implementations
Lightwood
George - Epistemink, Lightwood
Specifically, George is interested in thinking about the incentive structures of designing, using and contributing to ML software, and trying to think about what useful and rigorous results can actually be achieved by benchmarking of algorithm performance.
auto-sklearn
auto-sklearn, The implementation of hyperparameter optimization by Feurer et al. (2015):
auto-sklearn
is an automated machine learning toolkit and a drop-in replacement for a scikit-learn estimator:import autosklearn.classification cls = autosklearn.classification.AutoSklearnClassifier() cls.fit(X_train, y_train) predictions = cls.predict(X_test)
auto-sklearn
frees a machine learning user from algorithm selection and hyperparameter tuning. It leverages recent advantages in Bayesian optimization, meta-learning and ensemble construction.
No comments yet. Why not leave one?