Hydra for tracking machine learning experiments

Tutorials and examples

• Best tutorial: Kushajveer Singh, Complete tutorial on how to use Hydra in Machine Learning projects
• Julien Beaulieu, Building A Flexible Configuration System For Deep Learning Models.
• Simone Scardapane, Learning Hydra for configuring ML experiments
• Omry Yadan (hydra author) writes Hydra — A fresh look at configuration for machine learning projects

Here are some examples of hydra in action which might be useful as templates:

• lucmos/nn-template: Generic template to bootstrap your PyTorch project with PyTorch Lightning, Hydra, W&B, DVC, and Streamlit.
• ashleve/lightning-hydra-template: PyTorch Lightning + Hydra. A very general, feature-rich template for rapid and scalable ML experimentation with best practices. ⚡🔥⚡

In practice I found both of those way too fancy; too feature-stuffed. It was easiest to start from scratch without all the overblown dependencies and fancy chaos.

Instantiating arbitrary python classes

Instantiating objects with hydra.utils.instantiate — this allows you to specify which python class to instantiate using a special field, e.g.

_target_: src.models.mnist_model.MNISTLitModel

Corollary: these config files are a security risk! Do not use a config file from anyone you do not trust.

Environment variable interpolation

Variable interpolation is supported but a little bit weird because it supports both internal variables (which we just saw) and system environment variables and it takes a while to work out what is what. Hydra supports system environment variables, but confusingly does not expand any paths it finds in them, as it would if they were internal hydra variables. 🤷 I gather this is because the system environment variables arrive via lower-level Omegaconf resolvers upon which hydra is built.

local_path: ${oc.env:SPECIAL_PATH}

Pro tip: this is handy with an environment variable config system like dotenv.

Hydra support additional environment variable management.

Paths

Paths are complicated. I find it useful to be aware of the following patterns:

For one thing, hydra hijacks working directory by changing it to the current log directory. It’s useful to have the original working dir path as a special variable.

work_dir: ${hydra:runtime.cwd}

# example of using the above to define some other folder
data_dir: ${work_dir}/data/

We can alternatively get at the original and configured paths using hydra.utils.get_original_cwd.

Configuring that dir is possible. The default looks like this:

hydra:
  run:
    dir: ./outputs/${now:%Y-%m-%d}/${now:%H-%M-%S}

Here is one tht is IMO more useful, that allows us to optionally use a custom output dir set by local environment vars and which uses the job name.

hydra:
  run:
    dir: ${oc.env:OUTPUT_DIR,./output}/${hydra.job.name}/${now:%Y-%m-%d}/${now:%H-%M-%S}

UPDATE: hijacking is now optional: Set hydra.job.chdir=False to disable hijacking.

Paths are still slightly unintuitive. Hydra’s facility for creating nice paths is great, but they do not necessarily interpolate in the obvious way — tl;dr if what we want is to change into the path that hydra lets us define with lots of luxurious command substitution everything is fine.

If we wish to interrogate hydra in arbitrary locations in the config file to do arbitrary things (e.g. using the hydra job name or job id as a parameter for something else) it does not seem to be so nice; the hydra object is not available there.

The path of least resistance (ahem) is to do everything in the folder that hydra gives us but if we have some artefact that needs to go somewhere else, we can import the hydra object explicitly and interrogate it for the variables of interest. This is slightly ugly and leaky.

Logs

Logs are per default configured through hydra to persist to a disk file as well as stdout, which is usually what I want. See Customizing logging.

Python packages with hydra configs

Feasible. See the example.

Go parallel with hydra

Hydra will sweep parameters in parallel via plugins:

• Joblib Launcher plugin (for local jobs)
• Submitit Launcher plugin (for SLURM clusters)

The docs are a little confusing, but check out this example.

Looking at joblib, for example, we might want to enable a param sweep from the command line. AFAICT this should launch

python job.py hydra/launcher=joblib

TBC