Configuring machine learning experiments with Fiddle

2025-03-16 — 2025-06-10

computers are awful
faster pussycat
how do science
premature optimization
provenance
python
Figure 1

Fiddle is the successor to gin, AFAICT, at Google for configuring ML experiments. It uses a code-based config system, where we define Python functions that change the configuration. It’s a slightly different approach. I’m not sure how to use it for things like hyperparam search, but it looks fine.

It generates a CLI via Google’s absl system. The keyword is Flag Support.

Docs are sparse; see: Fiddle documentation especially

It’s currently in the middle of a major API update, and the HEAD version plus elderly last release version are very far apart.

I got Google Gemini to update the docs from the repository. See the Fiddle User Guide section below for the output.

However, the experience of trying to use the code was ultimately negative. The codebase of fiddle is enormous and incomprehensible, totaling megabytes of Python code. Maybe this means it exposes the “right” API, but I don’t think it is worth it in terms of confusion. Reading the code to understand what is going on is beyond human capabilities, and in fact beyond vibe coding capabilities. Ultimately, even with an AI coding assistant, I ended up spending the majority of development time fighting fiddle rather than building my project. Resolving names is confusing. It’s confusing to me, it’s confusing to the AI and it’s confusing to the interactive stack debugger.

1 Fiddle User Guide: Configuration, CLI, and Experiment Best Practices

This guide provides practical advice and recommended patterns for using Fiddle, with a focus on command-line integration via fdl_flags and managing experimental workflows. It aims to clarify common points of confusion and offer robust solutions.

1.1 Core Philosophy: Configuration as Code, Separated Concerns

  • Business Logic is Fiddle-Agnostic: Write your core algorithms (models, data processing, physics simulators, training loops) in standard Python modules without any direct dependency on Fiddle. These components should accept parameters via their __init__ methods or function arguments.
    • Example: my_project/src/my_project/business_logic/model_architectures.py
  • Configuration Logic in Dedicated Modules: Define your Fiddle configurations (fdl.Config, fdl.Partial), base setups, and reusable transformations (fiddlers) in separate Python modules, typically within a configs/ directory at the top level of your project or inside your main package.
    • Example: my_project/configs/base_model_configs.py, my_project/configs/fiddlers.py
  • Launcher Scripts for Execution: Use scripts in a scripts/ directory as entry points for running experiments. These scripts will use fdl_flags to parse command-line arguments, assemble the final Fiddle configuration, and then (often via a job submission system like submitit) execute a worker function that uses the built Fiddle configuration.

1.2 Defining Configurations

1.2.2 Manual fdl.Config / fdl.Partial Construction (Explicit Control)

Manually create fdl.Config or fdl.Partial instances and set their arguments.

# In configs/my_model_configs.py
import fiddle as fdl
import fiddle.printing  #must be explicitly imported to use fdl.printing.as_str_flattened

from src.my_project.business_logic import layers

def my_encoder_config_manual(num_layers: int = 4, hidden_size: int = 256) -> fdl.Config[layers.Encoder]:
    cfg = fdl.Config(layers.Encoder)
    cfg.embedding_dim = hidden_size

    sub_layer_cfgs = []
    for _ in range(num_layers):
        attn_cfg = fdl.Config(layers.AttentionLayer)
        attn_cfg.num_heads = 8
        attn_cfg.head_dim = hidden_size // 8
        sub_layer_cfgs.append(attn_cfg)
    cfg.layers = sub_layer_cfgs
    return cfg

# To get the Fiddle config:
# cfg = my_encoder_config_manual(num_layers=2)

1.3 Command-Line Interface with fdl_flags (New API)

IMPORTANT: Always use the new Fiddle flags API (fdl_flags.DEFINE_fiddle_config and fdl_flags.DEFINE_fiddle_sweep) for defining how your scripts consume Fiddle configurations from the command line. Avoid the legacy --fdl.* flags.

1.3.1 Defining a Configuration Flag (in scripts/your_launcher.py)

from absl import app
import fiddle as fdl
from fiddle import absl_flags as fdl_flags

# Import modules that contain the functions/classes your CLI will reference
from configs import my_model_configs
from configs import fiddlers

# Define a flag. 'exp_cfg' becomes '--exp_cfg' on the CLI.
MY_EXPERIMENT_CONFIG = fdl_flags.DEFINE_fiddle_config(
    name="exp_cfg",  # The command-line flag name
    default_module=[my_model_configs, fiddlers], # List of modules to search for names
    default="config:my_model_configs.my_encoder_config_manual", # Default base config
    help_string="Main configuration for the experiment.",
    required=False # Because a default is provided
)

def main(argv):
    # Accessing the fully resolved fdl.Buildable:
    unbuilt_cfg: fdl.Buildable = MY_EXPERIMENT_CONFIG.value

    # If --exp_cfg was NOT provided on CLI, and default is a simple 'config:name' string:
    # You *might* find that unbuilt_cfg is a list like ['config:name'] instead of
    # the fdl.Buildable. This seems to be a nuance in how FiddleFlag.value
    # handles defaults when no other CLI processing for that flag occurs.
    #
    # ROBUST HANDLING (Workaround for potential FiddleFlag default behavior):
    if isinstance(unbuilt_cfg, list) and all(isinstance(x, str) for x in unbuilt_cfg):
        print("WARNING: Flag default resolved to command strings. Re-parsing...")
        # Create a temporary parser with the same context as the original flag
        # to correctly process the default command string(s).
        # The .default_module attribute IS available on the FlagHolder.
        temp_parser = fdl_flags.FiddleFlag(
            name="_temp_default_parser", default=None,
            parser=flags.ArgumentParser(), serializer=None, help_string="Internal",
            default_module=MY_EXPERIMENT_CONFIG.default_module,
            # allow_imports=MY_EXPERIMENT_CONFIG.allow_imports # If you explicitly set it
        )
        temp_parser.parse(unbuilt_cfg) # unbuilt_cfg here is the list of command strings
        unbuilt_cfg = temp_parser.value
        print(f"Re-parsed default. Type is now: {type(unbuilt_cfg)}")

    if not isinstance(unbuilt_cfg, fdl.Buildable):
        raise TypeError(f"Configuration did not resolve to an fdl.Buildable. Got: {unbuilt_cfg}")

    # Now, unbuilt_cfg is guaranteed to be an fdl.Buildable
    # print(fdl.printing.as_str_flattened(unbuilt_cfg))
    # built_object = fdl.build(unbuilt_cfg)
    # ... proceed with your experiment ...

1.3.2 Invoking from the Command Line

Use the flag name defined in DEFINE_fiddle_config (e.g., --exp_cfg) followed by Fiddle commands:

  • config:<name_or_path_to_fn>: Specifies the base configuration function or recipe.
    • --exp_cfg=config:my_model_configs.my_encoder_config_manual
    • --exp_cfg=config:'my_model_configs.my_encoder_config(num_layers=2)' (with arguments)
  • fiddler:<name_or_path_to_fn>: Applies a fiddler function. Can be repeated; applied in order.
    • --exp_cfg=fiddler:fiddlers.apply_large_model_tweaks
    • --exp_cfg=fiddler:'fiddlers.set_learning_rate(rate=0.005)'
  • set:<path.to.attribute>=<value>: Directly overrides a parameter. Applied after all config: and fiddler: commands from the CLI.
    • --exp_cfg=set:model.layers[0].num_heads=12
    • --exp_cfg=set:data_config.path='"gs://my-bucket/dataset.tfrecord"' (Note nested quotes for string values).
    • --exp_cfg=set:model.use_dropout=True (Booleans and numbers usually don’t need extra quotes).
  • config_file:<path_to_json>: Loads a base config from a Fiddle JSON file.
  • config_str:<serialized_json_string>: Loads a base config from a Fiddle JSON string (often used by sweep launchers).

Example:

uv run python -m scripts.my_launcher \
  --exp_cfg=config:my_model_configs.my_encoder_config \
  --exp_cfg=fiddler:fiddlers.use_bfloat16_activations \
  --exp_cfg=set:model.layers[0].attention.dropout_rate=0.15 \
  --exp_cfg=set:output_dir='"./artifacts/runs/my_exp/run001"'

IMPORTANT: The default_module argument to DEFINE_fiddle_config is crucial. It’s a list of Python module objects where Fiddle will search for names like my_encoder_config or fiddlers.use_bfloat16_activations when they are not fully qualified on the command line. If a name is fully qualified (e.g., my_project.configs.my_model_configs.my_fn), default_module is less critical for that specific name resolution.

1.3.3 Managing Multiple Experiment Types / CLI Entry Points

If you have distinct experimental setups (e.g., one using SGD, another using full-batch GD with a different optimizer family), it’s often best to have separate launcher scripts (scripts/train_sgd.py, scripts/train_gd.py).

  • Each launcher script will define its own fdl_flags.DEFINE_fiddle_config instance (e.g., --sgd_cfg in one, --gd_cfg in another).
  • The default_module and the default="config:..." string for each flag can be tailored to the specific experiment type, pointing to appropriate base configurations (e.g., configs.trainer_configs.sgd_trainer_base vs. configs.trainer_configs.gd_trainer_base).

1.4 Understanding Fiddle Flags: DEFINE_fiddle_config, default_module, default, and Name Resolution

The fdl_flags.DEFINE_fiddle_config (and DEFINE_fiddle_sweep) function is powerful but its name resolution can be subtle. Here’s how to use default_module and default effectively to minimize confusion and errors.

A. The name Argument: * This defines the actual command-line flag, e.g., name="exp_cfg" creates --exp_cfg. * Convention: Choose a short, descriptive name, and be consistent across your launcher scripts (e.g., always use --exp_cfg for single runs, --sweep_cfg for sweeps).

B. The default_module Argument: * Purpose: This tells Fiddle where to look for function/class names when they are not fully qualified in a config: or fiddler: command on the CLI (or in the default= string, if it uses short names). * Type: It can be a single Python module object or a list of module objects. * How it Works with Short Names (e.g., config:my_recipe): 1. If default_module is a single module (e.g., default_module=my_recipes_module), Fiddle will look for my_recipes_module.my_recipe. 2. If default_module is a list of modules (e.g., default_module=[base_module, fiddler_module]), Fiddle will try base_module.my_recipe, then fiddler_module.my_recipe, etc., using the first one it finds. * How it Works with Partially Qualified Names (e.g., config:sub_module.my_recipe): 1. Fiddle will iterate through default_module. For each mod in the list, it will try to resolve mod.sub_module.my_recipe. * IMPORTANT: Interaction with Fully Qualified Names (e.g., config:my_project.configs.my_recipe): * If a name in a config: or fiddler: command is already fully qualified (i.e., it’s an absolute import path like my_project.configs.my_recipe), Fiddle will first attempt to resolve it relative to the modules in default_module if the first part of the qualified name matches an attribute of a module in default_module. This can be surprising. * If that relative resolution fails, it will then try a direct, absolute import of the fully qualified name. * Recommendation for default_module: * Be Specific: Provide the most specific module(s) that directly contain the functions you’ll commonly reference with short names. * Avoid Ambiguity: If module1.foo and module2.foo both exist and default_module=[module1, module2], then config:foo will resolve to module1.foo. Be explicit with config:module2.foo if that’s intended. * Use a List for Broader Scope: If your base configs are in configs.base and fiddlers in configs.fiddlers, then default_module=[configs.base, configs.fiddlers] allows CLI commands like config:my_base_config and fiddler:my_fiddler_fn.

C. The default Argument: * Purpose: This string provides the default set of commands to apply if the user does not provide the flag (e.g., --exp_cfg=...) on the command line at all. * Format: It’s a string (or list of strings for multi-string flags, though DEFINE_fiddle_config takes a single string which might contain multiple commands if properly formatted for the underlying flag parser, but usually it’s one config: command). This string must follow the same command:value syntax as used on the CLI. * IMPORTANT: Name Resolution for default String: * The names within the default string (e.g., my_recipe in default="config:my_recipe") are resolved using the default_module provided to DEFINE_fiddle_config. * Best Practice: For the default= string, it is highly recommended to use fully qualified Python paths to your config/recipe functions to avoid any ambiguity. This makes the default behavior independent of how default_module might be set or changed for CLI parsing convenience. python # GOOD: Explicit and robust default EXPERIMENT_FDL_CONFIG = fdl_flags.DEFINE_fiddle_config( name="exp_cfg", default_module=[configs.experiment_recipes, configs.base, configs.fiddlers], default="config:configs.experiment_recipes.my_project_default_recipe", # ... ) * Why? Fiddle internally processes this default string when the flag is defined. If it uses short names, it relies on the default_module at that definition time. If default_module is later changed or if the CLI resolution behaves slightly differently, the effective default could change unexpectedly. Fully qualified paths in the default= string are safer.

D. Understanding the “Inscrutable Error” (Could not resolve reference... available names: clear, copy, pop...)

This specific error means Fiddle was trying to find an attribute (like toy_vm_recipes) on a list object. This happens when: 1. You provide default_module=[my_actual_module_object] (a list containing one module). 2. And your default or CLI command string is something like config:my_actual_module_object.my_recipe_name. 3. Fiddle’s name resolution logic might try to see if my_actual_module_object (the string) is an attribute of the items in the default_module list. Since the item is my_actual_module_object, this can lead to a mismatch where it then tries to find my_recipe_name on the list itself if the initial attribute lookup fails or is misinterpreted.

How to Minimize Harm and Debug: 1. Be Explicit with default=: Use fully qualified paths in your default="config:..." string. * Your Example Corrected for Robust Default: python PHYSICS_MODEL_CFG = fdl_flags.DEFINE_fiddle_config( name="physics_recipe_cfg", # default_module is primarily for CLI convenience with short names default_module=[configs.physics_specific_configs.pidm_physics_recipes, configs.physics_specific_configs], # Use the full Python path for the default string value default="config:configs.physics_specific_configs.pidm_physics_recipes.toy_vector_magnitude_physics_recipe", help_string="...", ) (Assuming configs.physics_specific_configs.pidm_physics_recipes is the actual importable path to the module containing toy_vector_magnitude_physics_recipe).

  1. Simplify default_module if Possible:
    • If most referenced functions are in a single module, pass that module directly: default_module=my_primary_configs_module.
    • If using a list for default_module, be aware of potential ambiguities if the same short names exist in multiple modules in that list.
  2. Test Your Defaults: Run your launcher script without any Fiddle flags to ensure the default configuration loads correctly. This is where the temp_flag_parser workaround in the previous response becomes relevant if FLAG.value isn’t directly giving you the Buildable from the default.
  3. Use Fully Qualified Names on CLI for Unambiguity: When in doubt or when scripting, using the full Python path in CLI commands (--exp_cfg=config:my_project.configs.module.my_func) is always the safest, though more verbose. default_module is a convenience for interactive use or shorter commands.
  4. Inspect FLAG.value: When debugging, always print type(YOUR_FLAG.value) and YOUR_FLAG.value itself.
    • If CLI flags are provided, YOUR_FLAG.value should be an fdl.Buildable.
    • If NO CLI flags are provided and you used default="config:...", and YOUR_FLAG.value is still a list of strings (e.g., ['config:your.default.recipe'] or the config_str:eJx... form), then you must re-process this list to get the fdl.Buildable as shown in the workaround using a temporary FiddleFlag instance. This indicates the .value property doesn’t fully resolve string defaults when no CLI overrides are present in your Fiddle/Abseil setup.

By making the default= string fully qualified and understanding how default_module is used for resolving short names given on the CLI, you can significantly reduce these “inscrutable” resolution errors. The primary role of default_module becomes providing convenience for CLI users who prefer shorter names.

1.5 Experiment Sweeps with Submitit

For parameter sweeps, the recommended pattern is:

  1. Launcher Script (scripts/run_my_sweep.py):
    • Uses fdl_flags.DEFINE_fiddle_sweep("sweep_cfg", ...) to define the sweep parameters.
    • Iterates through SWEEP_CFG.value (which yields SweepItem objects).
    • For each sweep_item:
      • Determines a unique experiment_name and run_id based on the sweep parameters (sweep_item.overrides_applied).
      • Gathers all launch-time metadata (launcher CLI, git hash, RNG seeds, sweep_item.overrides_applied).
      • Serializes sweep_item.config to a JSON string using fdl.experimental.serialization.dump_json().
      • Uses submitit’s executor.submit() or executor.map_array() to schedule your worker function.
      • Passes the serialized config string and all gathered metadata as distinct arguments to the worker function.
  2. Worker Function (in src/mind_the_gap/business_logic/...):
    • Accepts the serialized config string and all metadata as arguments.
    • Deserializes the config string using fdl.experimental.serialization.load_json().
    • Sets up RNGs using the provided seeds.
    • Saves all received metadata (including the re-serialized Fiddle config JSON) to metadata.json in the designated artifact directory for this run.
    • Calls fdl.build() on the deserialized Fiddle config.
    • Runs the actual experiment logic.

This pattern ensures robust process isolation and clear metadata tracking for each trial in the sweep.

1.6 Command-Line Interface (New AP)

Fiddle’s integration with absl.flags allows flexible configuration from the command line. Always use the new fdl_flags.DEFINE_fiddle_config API and its associated CLI syntax.

5.1. Defining the Flag (in your launcher script, e.g., scripts/your_launcher.py)

# scripts/your_launcher.py
from absl import app
import fiddle as fdl
from fiddle import absl_flags as fdl_flags

# IMPORTANT: Import ALL modules that define functions/classes
# you might want to reference by name from the CLI or in default strings.
from configs import base_configs, model_configs, data_configs, optimizer_configs
from configs import fiddlers, experiment_recipes, tags

EXP_CFG = fdl_flags.DEFINE_fiddle_config(
    name="exp_cfg", # This sets the CLI flag name to --exp_cfg
    default_module=[ # List of modules Fiddle will search for short names
        base_configs, model_configs, data_configs, optimizer_configs,
        fiddlers, experiment_recipes, tags
    ],
    # For the default, using the FULL Python path is MOST ROBUST
    default="config:configs.experiment_recipes.my_default_experiment_recipe",
    help_string="Main Fiddle configuration for the experiment.",
    required=False # False if a 'default' is provided
)

def main(argv):
    unbuilt_cfg: fdl.Buildable = EXP_CFG.value
    # IMPORTANT: See section 5.3 if `unbuilt_cfg` is a list when only default is used.
    # ... (rest of your script) ...

1.6.1 Understanding default_module and default in DEFINE_fiddle_config

This is a common source of confusion and “inscrutable errors”.

  • name="flag_name": This defines your CLI flag (e.g., --flag_name).
    • Convention: Standardize these across your project (e.g., --exp_cfg for single runs, --sweep_cfg for sweeps).
  • default_module=[module1, module2, ...]:
    • Purpose: Tells Fiddle where to look for function/class names when they are not fully qualified in a config: or fiddler: command given on the CLI (or in the default= string if that string uses short names).
    • Behavior with Short Names (e.g., config:my_recipe): Fiddle searches module1.my_recipe, then module2.my_recipe, etc., using the first match.
    • Behavior with Partially Qualified Names (e.g., config:sub_module.my_recipe): Fiddle tries module1.sub_module.my_recipe, then module2.sub_module.my_recipe, etc.
    • Behavior with Fully Qualified Names (e.g., config:project.configs.my_recipe): Fiddle will first try to resolve this relative to each module in default_module if the leading part matches an attribute of a module in the list (e.g., if default_module=[project_configs_module] and project_configs_module has an attribute configs, it might try project_configs_module.configs.my_recipe). If these relative attempts fail, it falls back to a direct absolute import of project.configs.my_recipe.
    • Recommendation:
      • Provide the actual imported module objects, not strings of module names.
      • Be specific. If you have configs.base and configs.fiddlers, include both: default_module=[configs.base, configs.fiddlers].
      • If using short names on the CLI, be mindful of potential name collisions if they exist in multiple modules in the default_module list (the first one found wins).
  • default="config:path.to.default_recipe":
    • Purpose: Specifies the Fiddle command string(s) to use if the user provides no instances of this flag (e.g., no --exp_cfg) on the command line.
    • IMPORTANT - Best Practice: Always use the fully qualified Python import path to your default recipe function within the default string. python default="config:my_project.configs.experiment_recipes.default_experiment_setup" This makes your default behavior robust and independent of how default_module might be configured for CLI convenience.
    • Internal Behavior: Fiddle processes this default string when the flag is defined. It often serializes the result into a config_str:... format for its internal default storage. This is normal and visible in --helpfull output.

1.6.2 Accessing the fdl.Buildable (Getting the Config Object)

  • After absl.app.run(main) has processed flags, YOUR_FLAG_HOLDER.value (e.g., EXP_CFG.value) is the property to access the final fdl.Buildable.

  • Typical Behavior (CLI Overrides Present): If the user provides --exp_cfg=... on the command line, EXP_CFG.value will directly return the processed fdl.Buildable object.

  • Potential Issue (Default Value Only - The Bug You Encountered):

    • You observed that if only the default="config:..." string is used (no --exp_cfg on CLI), EXP_CFG.value might return a list containing the command string(s) from the default, e.g., ['config:your.default.recipe'].

    • This is the “flaky” behavior. The FiddleFlag.value property should ideally handle this transparently.

    • Robust Workaround (if the above issue persists in your Fiddle version):

      # Inside your main() function
      raw_flag_value = EXP_CFG.value
      unbuilt_cfg: fdl.Buildable
      
      if isinstance(raw_flag_value, fdl.Buildable):
          unbuilt_cfg = raw_flag_value
      elif isinstance(raw_flag_value, list) and all(isinstance(x, str) for x in raw_flag_value):
          print("WARNING: Flag default resolved to command strings. Re-parsing default explicitly.")
          # EXPERIMENT_FDL_CONFIG.default_module is the correct attribute on the FlagHolder
          temp_parser = fdl_flags.FiddleFlag(
              name="_temp_default_parser", default=None,
              parser=flags.ArgumentParser(), serializer=None, help_string="Internal",
              default_module=EXPERIMENT_FDL_CONFIG.default_module, # Correct way to get it
              # allow_imports=EXPERIMENT_FDL_CONFIG.allow_imports # If you use this setting
          )
          temp_parser.parse(raw_flag_value) # Parse the list of command strings
          unbuilt_cfg = temp_parser.value
      else:
          raise TypeError(f"Unexpected type from flag value: {type(raw_flag_value)}")
      
      # Now, unbuilt_cfg should definitely be an fdl.Buildable
      exp_cfg_instance = fdl.build(unbuilt_cfg)
    • Alternative Simpler Workaround (If the “list” problem is only for defaults): Always provide at least one --exp_cfg=config:your_default_recipe on the command line for your default runs, and make the flag required=True in DEFINE_fiddle_config (and remove the default= string from the definition). This forces the CLI parsing path which consistently returns a Buildable.

1.6.3 CLI Syntax for Setting Parameters (set:)

  • Target Arguments, Not Internals: When using config:my_recipe_fn, subsequent set: commands operate on the arguments of my_recipe_fn.

    • If my_recipe_fn(param1: int, sub_config: fdl.Config[AnotherClass]), then:
      • --exp_cfg=set:param1=10 (Correct)
      • --exp_cfg=set:sub_config.attr_of_another_class=value (Correct)
    • Avoid paths that refer to the internal structure of objects returned by my_recipe_fn. This was the cause of your AttributeError for thermal_diffusivity. thermal_diffusivity is an argument to your recipe, and the recipe then uses it to configure PhysicsComponents.physics_solver_config.thermal_diffusivity.
      • Correct CLI: --exp_cfg=set:thermal_diffusivity=0.2
      • Incorrect CLI (what caused your error): --exp_cfg=set:physics_solver_config.thermal_diffusivity=0.2 (This assumes physics_solver_config is a top-level argument to the recipe, which it is, but then you want to set an attribute of that argument directly if it’s a config object, not an attribute of the PhysicsComponents object that the recipe returns using a path from the root of PhysicsComponents.*)

    Corrected Example from Bug Report (Conceptual): Assuming thin_plate_physics_recipe takes heat_source_cfg and temperature_field_cfg as arguments:

    --physics_recipe_cfg="config:configs...thin_plate_physics_recipe" \
    --physics_recipe_cfg="set:thermal_diffusivity=0.2" \
    --physics_recipe_cfg="set:base_thickness_value=0.015" \
    --physics_recipe_cfg="set:heat_source_cfg.center=(0.0,0.0)" \
    --physics_recipe_cfg="set:temperature_field_cfg.perlin_noise_cfg.amplitude=3.0"

1.6.4 Other Commands (fiddler:, config_file:, config_str:)

  • These follow the same name resolution rules via default_module if short names are used.
  • They are applied sequentially in the order they appear on the command line for a given flag instance.

1.7 Reproducibility and Artifacts

  • metadata.json is Key: For each run, ensure metadata.json in artifacts/runs/<experiment_name>/<run_id>/ contains:
    • fiddle_config_json: The output of fdl.experimental.serialization.dump_json(unbuilt_cfg).
    • launcher_cli_command: The exact sys.argv of the launcher script.
    • git_hash: Captured at launch time.
    • rng_seeds: A dictionary of all seeds used (Python, NumPy, PyTorch/JAX, etc.).
    • schema_version: e.g., "1.0". Increment if you change the metadata.json structure.
    • sweep_params_json: (If applicable) JSON string of sweep_item.overrides_applied.
  • Shared Resources: Paths to datasets, pre-trained models, etc., should be configured within your Fiddle Buildable (e.g., cfg.data.path = "/path/to/dataset"), not hardcoded in business logic.

1.8 Potential Gotchas & Troubleshooting

  • FLAG.value for Defaults: As noted, if only a default="config:..." string is used with DEFINE_fiddle_config and no CLI flag is passed, FLAG.value might return the command string(s) as a list. The robust workaround is to re-parse this default using a temporary FiddleFlag instance as shown in the launcher script example.
  • String Argument Quoting on CLI: When using set:path.to.str_arg="my value", ensure the string value itself is quoted if it contains spaces or special characters that the shell or Fiddle’s value parser might misinterpret: set:foo.name='"A String Value"'.
  • Module Resolution for default_module: Ensure the modules passed to default_module are the actual imported module objects, and that the names used in config: or fiddler: commands can be resolved as attributes of those modules.
  • Forgetting @auto_config or .as_buildable(): If using @auto_config, remember to call .as_buildable() on the decorated function to get the Fiddle config. If you pass the auto_config-decorated function itself to fdl.Config(), Fiddle will try to configure the AutoConfig wrapper object, not your intended function.