Configuring machine learning experiments with Fiddle

1.1 Core Philosophy: Configuration as Code, Separated Concerns

• Business Logic is Fiddle-Agnostic: Write your core algorithms (models, data processing, physics simulators, training loops) in standard Python modules without any direct dependency on Fiddle. These components should accept parameters via their __init__ methods or function arguments.

  Example: my_project/src/my_project/business_logic/model_architectures.py

• Configuration Logic in Dedicated Modules: Define your Fiddle configurations (fdl.Config, fdl.Partial), base setups, and reusable transformations (fiddlers) in separate Python modules, typically within a configs/ directory at the top level of your project or inside your main package.

  Example: my_project/configs/base_model_configs.py, my_project/configs/fiddlers.py

• Launcher Scripts for Execution: Use scripts in a scripts/ directory as entry points for running experiments. These scripts will use fdl_flags to parse command-line arguments, assemble the final Fiddle configuration, and then, often via a job submission system like submitit, execute a worker function that uses the built Fiddle configuration.

1.2 Defining Configurations

# In configs/my_model_configs.py
import fiddle as fdl
from fiddle.experimental import auto_config
from src.my_project.business_logic import layers # Assuming layers.py exists

@auto_config.auto_config
def my_encoder_config(num_layers: int = 4, hidden_size: int = 256) -> layers.Encoder:
    sub_layer_cfgs = []
    for _ in range(num_layers):
        sub_layer_cfgs.append(layers.AttentionLayer(num_heads=8, head_dim=hidden_size // 8))
    return layers.Encoder(layers=sub_layer_cfgs, embedding_dim=hidden_size)

# To get the Fiddle config:
# cfg = my_encoder_config.as_buildable(num_layers=2)

# In configs/my_model_configs.py
import fiddle as fdl
import fiddle.printing  #must be explicitly imported to use fdl.printing.as_str_flattened

from src.my_project.business_logic import layers

def my_encoder_config_manual(num_layers: int = 4, hidden_size: int = 256) -> fdl.Config[layers.Encoder]:
    cfg = fdl.Config(layers.Encoder)
    cfg.embedding_dim = hidden_size

    sub_layer_cfgs = []
    for _ in range(num_layers):
        attn_cfg = fdl.Config(layers.AttentionLayer)
        attn_cfg.num_heads = 8
        attn_cfg.head_dim = hidden_size // 8
        sub_layer_cfgs.append(attn_cfg)
    cfg.layers = sub_layer_cfgs
    return cfg

# To get the Fiddle config:
# cfg = my_encoder_config_manual(num_layers=2)

1.3 Command-Line Interface with fdl_flags (New API)

IMPORTANT: We should always use the new Fiddle flags API (fdl_flags.DEFINE_fiddle_config and fdl_flags.DEFINE_fiddle_sweep) to let our scripts consume Fiddle configurations from the command line. Avoid the legacy --fdl.* flags.

from absl import app
import fiddle as fdl
from fiddle import absl_flags as fdl_flags

# Import modules that contain the functions/classes your CLI will reference
from configs import my_model_configs
from configs import fiddlers

# Define a flag. 'exp_cfg' becomes '--exp_cfg' on the CLI.
MY_EXPERIMENT_CONFIG = fdl_flags.DEFINE_fiddle_config(
    name="exp_cfg",  # The command-line flag name
    default_module=[my_model_configs, fiddlers], # List of modules to search for names
    default="config:my_model_configs.my_encoder_config_manual", # Default base config
    help_string="Main configuration for the experiment.",
    required=False # Because a default is provided
)

def main(argv):
    # Accessing the fully resolved fdl.Buildable:
    unbuilt_cfg: fdl.Buildable = MY_EXPERIMENT_CONFIG.value

    # If --exp_cfg was NOT provided on CLI, and default is a simple 'config:name' string:
    # You *might* find that unbuilt_cfg is a list like ['config:name'] instead of
    # the fdl.Buildable. This seems to be a nuance in how FiddleFlag.value
    # handles defaults when no other CLI processing for that flag occurs.
    #
    # ROBUST HANDLING (Workaround for potential FiddleFlag default behavior):
    if isinstance(unbuilt_cfg, list) and all(isinstance(x, str) for x in unbuilt_cfg):
        print("WARNING: Flag default resolved to command strings. Re-parsing...")
        # Create a temporary parser with the same context as the original flag
        # to correctly process the default command string(s).
        # The .default_module attribute IS available on the FlagHolder.
        temp_parser = fdl_flags.FiddleFlag(
            name="_temp_default_parser", default=None,
            parser=flags.ArgumentParser(), serializer=None, help_string="Internal",
            default_module=MY_EXPERIMENT_CONFIG.default_module,
            # allow_imports=MY_EXPERIMENT_CONFIG.allow_imports # If you explicitly set it
        )
        temp_parser.parse(unbuilt_cfg) # unbuilt_cfg here is the list of command strings
        unbuilt_cfg = temp_parser.value
        print(f"Re-parsed default. Type is now: {type(unbuilt_cfg)}")

    if not isinstance(unbuilt_cfg, fdl.Buildable):
        raise TypeError(f"Configuration did not resolve to an fdl.Buildable. Got: {unbuilt_cfg}")

    # Now, unbuilt_cfg is guaranteed to be an fdl.Buildable
    # print(fdl.printing.as_str_flattened(unbuilt_cfg))
    # built_object = fdl.build(unbuilt_cfg)
    # ... proceed with your experiment ...

uv run python -m scripts.my_launcher \
  --exp_cfg=config:my_model_configs.my_encoder_config \
  --exp_cfg=fiddler:fiddlers.use_bfloat16_activations \
  --exp_cfg=set:model.layers[0].attention.dropout_rate=0.15 \
  --exp_cfg=set:output_dir='"./artifacts/runs/my_exp/run001"'

1.4 Understanding Fiddle Flags: DEFINE_fiddle_config, default_module, default, and Name Resolution

The fdl_flags.DEFINE_fiddle_config (and DEFINE_fiddle_sweep) function is powerful but its name resolution can be subtle. Here’s how to use default_module and default effectively to minimize confusion and errors.

A. The name Argument:

• This defines the actual command-line flag, e.g., name="exp_cfg" creates --exp_cfg.
• Convention: Choose a short, descriptive name, and be consistent across our launcher scripts (e.g., always use --exp_cfg for single runs, --sweep_cfg for sweeps).

B. The default_module Argument:

• Purpose: This tells Fiddle where to look for function/class names when they are not fully qualified in a config: or fiddler: command on the CLI (or in the default= string, if it uses short names).

• Type: It can be a single Python module object or a list of module objects.

• How it Works with Short Names (e.g., config:my_recipe):

  1. If default_module is a single module (e.g., default_module=my_recipes_module), Fiddle will look for my_recipes_module.my_recipe.
  2. If default_module is a list of modules (e.g., default_module=[base_module, fiddler_module]), Fiddle will try base_module.my_recipe, then fiddler_module.my_recipe, etc., using the first one it finds.

• How it Works with Partially Qualified Names (e.g., config:sub_module.my_recipe):

  1. Fiddle will iterate through default_module. For each mod in the list, it will try to resolve mod.sub_module.my_recipe.

• IMPORTANT: Interaction with Fully Qualified Names (e.g., config:my_project.configs.my_recipe):

  • If a name in a config: or fiddler: command is already fully qualified (i.e., it’s an absolute import path like my_project.configs.my_recipe), Fiddle will first attempt to resolve it relative to the modules in default_module if the first part of the qualified name matches an attribute of a module in default_module. This behaviour can be surprising.
  • If that relative resolution fails, it will then try a direct, absolute import of the fully qualified name.

• Recommendation for default_module:

  • Be Specific: Provide the most specific module(s) that directly contain the functions we’ll commonly reference with short names.
  • Avoid Ambiguity: If module1.foo and module2.foo both exist and default_module=[module1, module2], then config:foo will resolve to module1.foo. Be explicit with config:module2.foo if that’s intended.
  • Use a List for Broader Scope: If our base configs are in configs.base and fiddlers in configs.fiddlers, then default_module=[configs.base, configs.fiddlers] allows CLI commands like config:my_base_config and fiddler:my_fiddler_fn.

C. The default Argument:

• Purpose: This string provides the default set of commands to apply if the flag is not provided (e.g., --exp_cfg=...) on the command line at all.

• Format: It’s a string (or list of strings for multi-string flags, though DEFINE_fiddle_config takes a single string which might contain multiple commands if properly formatted for the underlying flag parser, but usually it’s one config: command). This string must follow the same command:value syntax as used on the CLI.

• IMPORTANT: Name Resolution for default String:

  • The names within the default string (e.g., my_recipe in default="config:my_recipe") are resolved using the default_module provided to DEFINE_fiddle_config.

  • Best Practice: For the default= string, it is highly recommended to use fully qualified Python paths to our config/recipe functions to avoid any ambiguity. This makes the default behaviour independent of how default_module might be set or changed for CLI parsing convenience.

    # GOOD: Explicit and robust default
    EXPERIMENT_FDL_CONFIG = fdl_flags.DEFINE_fiddle_config(
        name="exp_cfg",
        default_module=[configs.experiment_recipes, configs.base, configs.fiddlers],
        default="config:configs.experiment_recipes.my_project_default_recipe",
        # ...
    )

  • Why? Fiddle internally processes this default string when the flag is defined. If it uses short names, it relies on the default_module at that definition time. If default_module is later changed or if the CLI resolution behaves slightly differently, the effective default can change unexpectedly. Fully qualified paths in the default= string are safer.

D. Understanding the “Inscrutable Error” (Could not resolve reference... available names: clear, copy, pop...)

This specific error means Fiddle was trying to find an attribute (like toy_vm_recipes) on a list object. This happens when: 1. We provide default_module=[my_actual_module_object] (a list containing one module). 2. And the default or CLI command string is something like config:my_actual_module_object.my_recipe_name. 3. Fiddle’s name resolution logic might try to see if my_actual_module_object (the string) is an attribute of the items in the default_module list. Since the item is my_actual_module_object, this can lead to a mismatch where it then tries to find my_recipe_name on the list itself if the initial attribute lookup fails or is misinterpreted.

How to Minimize Harm and Debug: 1. Be Explicit with default=: Use fully qualified paths in the default="config:..." string.

   *   *Your Example Corrected for Robust Default*:

       ```python
       PHYSICS_MODEL_CFG = fdl_flags.DEFINE_fiddle_config(
           name="physics_recipe_cfg",
           # default_module is primarily for CLI convenience with short names
           default_module=[configs.physics_specific_configs.pidm_physics_recipes, configs.physics_specific_configs],
           # Use the full Python path for the default string value
           default="config:configs.physics_specific_configs.pidm_physics_recipes.toy_vector_magnitude_physics_recipe",
           help_string="...",
       )
       ```

       (Assuming `configs.physics_specific_configs.pidm_physics_recipes` is the actual importable path to the module containing `toy_vector_magnitude_physics_recipe`).

2. Simplify default_module if Possible:

   • If most referenced functions are in a single module, pass that module directly: default_module=my_primary_configs_module.
   • If using a list for default_module, be aware of potential ambiguities if the same short names exist in multiple modules in that list.

3. Test the Defaults: Run the launcher script without any Fiddle flags to ensure the default configuration loads correctly. This is where the temp_flag_parser workaround in the previous response becomes relevant if FLAG.value isn’t directly giving us the Buildable from the default.

4. Use Fully Qualified Names on CLI for Unambiguity: When in doubt or when scripting, use the full Python path in CLI commands (--exp_cfg=config:my_project.configs.module.my_func). This is always the safest option, though more verbose. default_module is a convenience for interactive use or shorter commands.

5. Inspect FLAG.value: When debugging, always print type(YOUR_FLAG.value) and YOUR_FLAG.value itself.

   • If CLI flags are provided, YOUR_FLAG.value should be an fdl.Buildable.
   • If no CLI flags are provided and we used default="config:...", and YOUR_FLAG.value is still a list of strings (e.g., ['config:your.default.recipe'] or the config_str:eJx... form), then we need to re-process this list to get the fdl.Buildable as shown in the workaround using a temporary FiddleFlag instance. This indicates the .value property doesn’t fully resolve string defaults when no CLI overrides are present in our Fiddle/Abseil setup.

By making the default= string fully qualified and understanding how default_module is used for resolving short names given on the CLI, we can significantly reduce these “inscrutable” resolution errors. The primary role of default_module becomes providing convenience for CLI users who prefer shorter names.

1.5 Experiment Sweeps with Submitit

For parameter sweeps, the recommended pattern is:

1. Launcher Script (scripts/run_my_sweep.py):

   • Uses fdl_flags.DEFINE_fiddle_sweep("sweep_cfg", ...) to define the sweep parameters.

   • Iterates through SWEEP_CFG.value (which yields SweepItem objects).

   • For each sweep_item:

     • Determines a unique experiment_name and run_id from the sweep parameters (sweep_item.overrides_applied).
     • Gathers all launch-time metadata (launcher CLI, git hash, RNG seeds, sweep_item.overrides_applied).
     • Serializes sweep_item.config to a JSON string using fdl.experimental.serialization.dump_json().
     • Uses submitit’s executor.submit() or executor.map_array() to schedule the worker function.
     • Passes the serialized config string and all gathered metadata as distinct arguments to the worker function.

2. Worker Function (in src/mind_the_gap/business_logic/...):

   • Accepts the serialized config string and all metadata as arguments.
   • Deserializes the config string using fdl.experimental.serialization.load_json().
   • Sets up RNGs using the provided seeds.
   • Saves all received metadata (including the re-serialized Fiddle config JSON) to metadata.json in the designated artifact directory for this run.
   • Calls fdl.build() on the deserialized Fiddle config.
   • Runs the actual experiment logic.

This pattern ensures robust process isolation and clear metadata tracking for every trial in the sweep.

1.6 Command-Line Interface (New API)

Fiddle’s integration with absl.flags allows flexible configuration from the command line. Always use the new fdl_flags.DEFINE_fiddle_config API and its associated CLI syntax.

Defining the Flag (in our launcher script, e.g., scripts/your_launcher.py)

# scripts/your_launcher.py
from absl import app
import fiddle as fdl
from fiddle import absl_flags as fdl_flags

# IMPORTANT: Import ALL modules that define functions/classes
# you might want to reference by name from the CLI or in default strings.
from configs import base_configs, model_configs, data_configs, optimizer_configs
from configs import fiddlers, experiment_recipes, tags

EXP_CFG = fdl_flags.DEFINE_fiddle_config(
    name="exp_cfg", # This sets the CLI flag name to --exp_cfg
    default_module=[ # List of modules Fiddle will search for short names
        base_configs, model_configs, data_configs, optimizer_configs,
        fiddlers, experiment_recipes, tags
    ],
    # For the default, using the FULL Python path is MOST ROBUST
    default="config:configs.experiment_recipes.my_default_experiment_recipe",
    help_string="Main Fiddle configuration for the experiment.",
    required=False # False if a 'default' is provided
)

def main(argv):
    unbuilt_cfg: fdl.Buildable = EXP_CFG.value
    # ... (rest of your script) ...

1.7 Reproducibility and Artifacts

• metadata.json is Key: For each run, ensure metadata.json in artifacts/runs/<experiment_name>/<run_id>/ contains:
  • fiddle_config_json: The output of fdl.experimental.serialization.dump_json(unbuilt_cfg).
  • launcher_cli_command: The exact sys.argv of the launcher script.
  • git_hash: Captured at launch time.
  • rng_seeds: A dictionary of all seeds used (Python, NumPy, PyTorch/JAX, etc.).
  • schema_version: e.g., "1.0". Increment if you change the metadata.json structure.
  • sweep_params_json: (If applicable) JSON string of sweep_item.overrides_applied.
• Shared Resources: Paths to datasets, pre-trained models, etc., should be configured within your Fiddle Buildable (e.g., cfg.data.path = "/path/to/dataset"), not hardcoded in business logic.

1.8 Potential Gotchas & Troubleshooting

• FLAG.value for Defaults: As noted, if only a default="config:..." string is used with DEFINE_fiddle_config and no CLI flag is passed, FLAG.value might return the command string(s) as a list. The robust workaround is to re-parse this default using a temporary FiddleFlag instance as shown in the launcher script example.
• String Argument Quoting on CLI: When using set:path.to.str_arg="my value", ensure the string value itself is quoted if it contains spaces or special characters that the shell or Fiddle’s value parser might misinterpret: set:foo.name='"A String Value"'.
• Module Resolution for default_module: Ensure the modules passed to default_module are the actual imported module objects, and that the names used in config: or fiddler: commands can be resolved as attributes of those modules.
• Forgetting @auto_config or .as_buildable(): If using @auto_config, remember to call .as_buildable() on the decorated function to get the Fiddle config. If you pass the auto_config-decorated function itself to fdl.Config(), Fiddle will try to configure the AutoConfig wrapper object, not your intended function.