Repository navigation
Nest cloudlab/local config under a providers: section in config.yaml #440
Description
Context
#424 added local-machine execution via a new top-level local: config section, injected at the CLI with Hydra's +local.home_dir=... syntax (since local isn't a structural key in config/config.yaml by default — providers/factory.py's create_provider() and detect_provider_type() use hasattr(cfg, "local") to decide which provider to build, and that only works because the key is absent unless explicitly added).
Proposal
Restructure config.yaml so cloudlab: and local: both live under a providers: namespace:
providers:
cloudlab:
num_nodes: ???
node_offset: 0
username: ???
hostname_suffix: ???
local:
home_dir: nullThis is more symmetric (no more "one provider gets a real CLI flag, the other needs Hydra's + add-key syntax") and lets detection switch from presence of a section to the value of providers.local.home_dir (set vs null) — actually a cleaner signal than hasattr, and avoids the + prefix entirely (providers.local.home_dir=... works with normal Hydra override syntax once the key exists with a default).
Why this is a separate issue, not part of #424
config/config.yaml is shared by every Hydra-based entrypoint in this directory, not just the two #424 touched (experiment_run_e2e.py, experiment_run_clickhouse.py). In particular:
- The YAML interpolation
streaming.remote_write.ip: "${remote_write_ip:${cloudlab.node_offset}}"lives inconfig.yamlitself and would need to become${providers.cloudlab.node_offset}— this affects every script that loadsconfig_name="config", including ones Support running experiments fully locally (no CloudLab dependency) #424 explicitly left untouched (experiment_teardown_everything.py,experiment_run_sketchdboffline.py,experiment_run_exporters_and_prometheus.py,experiment_run_grafana_demo.py,experiment_only_ingest_path.py,grafana_config.py). grafana_config.pyandexperiment_run_clickhouse.pyhave CLI usage examples in docstrings/help text (cloudlab.num_nodes=2, etc.) that would go stale.HYDRA_CONFIG_USAGE.mdandCONFIG_PARAMETERS_REFERENCE.mddocument the current flatcloudlab.*paths extensively.
Scope of the change (when picked up)
config/config.yaml— nest both sections underproviders:, update theremote_write_ipresolver interpolationexperiment_utils/config.py—get_node_params(),required_cloudlab_params(),Args.__init__(all currently readcfg.cloudlab.*)experiment_utils/providers/factory.py—create_provider()/detect_provider_type()(currentlyhasattr(cfg, "local")/hasattr(cfg, "cloudlab"))experiment_run_clickhouse.py,grafana_config.py— update CLI usage docstringsHYDRA_CONFIG_USAGE.md,CONFIG_PARAMETERS_REFERENCE.md— update documented config pathsexperiment_utils/test_local_provider.py— update the self-check's assertions
Not urgent; current local:/+local.home_dir=... setup (#424) works and is tested. This is a cosmetic/ergonomic follow-up.
Activity