Configuring machine learning experiments



A dual problem to experiment tracking is experiment configuring; How can I nicely define experiments and the parameters which make them go?

Hydra

My go-to. If you are working in python this does more or less everything, and is the least annoying to work with IMO. See my Hydra page.

DrWatson.jl

As mentioned under experiment tracking, DrWatson which automatically attaches code versions to simulation and does some other work to generally keep simulations tracked and reproducible. Special feature: Works with julia, which is my other major language.

Allennlp Param

Allennlp’s Param system is a kind of introductory trainer-wheels configuration system, but not recommended in practice. It come with a lot of baggage — installing it will slurp in a large number of fragile and fussy dependencies for language parsing. Once I used this for a while I realised all the reasons I would would a better system, which is provided by hydra.

DIY

Why use an external library for this? I could of course, roll my own. I have done that a few times. It is a surprisingly large amount of work though, remarkably easy to get wrong, and there are perfectly good tools to do it already.


No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.