Hydra for tracking machine learning experiments

When I configure my ML experiments, i.e. try the various combinations of parameters to find the good ones, the one-stop shop (at least for python) is hydra. Related: hyperparameter search (which I also do using hydra).

DID YOU SAY A FRAMEWORK FOR ELEPHANTLY CONFIGURING COMPLEX APPLICATIONS

Hydra is a supposedly generic configuration system but they have been very careful to have good affordances for ML in particular. The fact that this is not obvious from the docs is the worst feature of Hydra; but it is good enough to push through that difficulty.

Hydra uses something called the OmegaConf system. Configuration is by YAML and they have thought about configuring experiments in particular.

Features:

• explicit flexible config syntax
• optional typing system to detect misconfigurations (although in practice I will never have time to actually set this up)
• a command-line system with sophisticated further configuration and overrides
• hierarchical configurations which may be overridden in various ways

Misfeatures:

• Most of the documentation and examples are about pytorch, presumably because of the facebook connection, which means that the examples are not at all generic.
• Manual is flat, affectless and unopinionated even where an opinion would be useful
• hierarchical configurations may be overridden in way too many ways and TBH it is a bit confusing and I’m worried about how you would do something like merge two configuration systems

The starting problem for me is that because it is so generic, powerful and clean, it is not clear where to even begin in configuring a given project. The manual is more a reference document than a learn-by-doing guide. Fortunately, there are 3rd party HOWTOs.

Tutorials and examples

Here are some examples of hydra in action which might be useful as templates:

In practice I found both of those way too fancy; they included too many features. It was easiest to start from scratch without all the overblown dependencies.

Instantiating arbitrary python classes

Instantiating objects with hydra.utils.instantiate — this allows you to specify which python class to instantiate using a special field, .e.g.

_target_: src.models.mnist_model.MNISTLitModel

Corollary: these config files are a security risk! Do not use a config file from anyone you do not trust.

Environment variable interpolation

Variable interpolation is supported but a little bit weird because it supports both internal variables (which we just saw) and system environment variables. This syntax supports system environment variables, but confusingly does not expand any paths it finds in them, as it would if they were internal hydra variables. 🤷 I gather this is because the system environment variables arrive via lower-level Omegaconf resolvers which hydra is built upon.

local_path: ${oc.env:SPECIAL_PATH} Pro tip: this is handy with an environment variable config system like dotenv. Paths Paths are complicated. I find it useful to be aware of the following patterns: For one thing, hydra hijacks working directory by changing it to the current log directory. It’s useful to have the original working dir path as a special variable. work_dir:${hydra:runtime.cwd}

# example of using the above to define some other folder
data_dir: ${work_dir}/data/ We can alternatively get at the original and configured paths using hydra.utils.get_original_cwd. Configuring that dir is is possible. The default looks like this: hydra: run: dir: ./outputs/${now:%Y-%m-%d}/${now:%H-%M-%S} Here is one tht is IMO more useful, that allows us to optionally use a custom output dir set by local environment vars and which uses the job name. hydra: run: dir:${oc.env:OUTPUT_DIR,./output}/${hydra.job.name}/${now:%Y-%m-%d}/\${now:%H-%M-%S}

UPDATE: hijacking is now optional: Set hydra.job.chdir=False to disable hijacking.

Paths are still slightly unintuitive. Hydra’s facility for creating nice paths is great, but they do not necessarily interpolate in the obvious waytl;dr if what we want is to change into the path that hydra lets us define with lots of luxurious command substitution everything is fine.

If we wish to interrogate hydra in arbitrary locations in the config file to do arbitrary things (e.g. using the hydra job name or job id as a parameter for something else) it does not seem to be so nice; the hydra object is not available there.

The path of least resistance (ahem) is to do everything in the folder that hydra gives us but if we have some artefact that needs to go somewhere else, we can import the hydra object explicitly and interrogate it for the variables of interest. This is slightly ugly and leaky.

Logs

Logs are per default configured through hydra to persist to a disk file as well as stdout, which is usually what I want. See Customizing logging.

Python packages with hydra configs

Feasible. See the example.

Go parallel with hydra

Hydra will sweep parameters in parallel via plugins

The docs are a little confusing, but check out this example.

Looking at joblib, for example, we might want to enable a param sweep from the command line. AFAICT this should launch

python job.py hydra/launcher=joblib 

TBC

No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.