Support running experiments fully locally (no CloudLab dependency) #424

Currently experiment_run_e2e.py and experiment_run_clickhouse.py require a CloudLab cluster — create_provider() (experiment_utils/providers/factory.py) is hardcoded to always return CloudLabProvider, and config validation requires cloudlab.username/cloudlab.hostname_suffix.

Goal: add a provider that runs an experiment entirely on a single dev machine, no CloudLab account/cluster needed at all.

Note on CloudLabLocalProvider: there's already a cloudlab_local.py provider, but it's for a different use case — running locally after SSHing into a CloudLab node, still using CloudLab paths/conventions (CLOUDLAB_HOME_DIR, etc). It's not a dev-machine-independent provider as-is, though some of its no-SSH execute_command logic may be reusable as a starting point.

What's already reusable:

• The InfrastructureProvider base interface (providers/base.py) — execute_command, execute_command_parallel, get_node_ip, get_home_dir, get_query_log_file — is provider-agnostic and most services call it correctly.
• factory.py already has commented-out scaffolding for provider-type detection (detect_provider_type), just needs wiring up.

Suggested approach: introduce a new LocalProvider (distinct from CloudLabLocalProvider) implementing the base interface for single-machine execution — get_node_ip → 127.0.0.1, get_home_dir → some local scratch dir, file "sync" → no-op since everything's already on one filesystem — then fix the sync.py/services rsync bypass to go through the provider instead of hardcoding SSH.

Scope refinement (post-grilling)

Implementation landed with a narrower, more concrete scope than the original checklist:

• Single local node only (local.home_dir=..., equivalent to cloudlab.num_nodes=0 — coordinator only, node_offset hardcoded to 0). No multi-node local cluster, no SSH-to-localhost.
• Streaming engines precompute and arroyo only — Flink dropped from scope. flink.py's CLOUDLAB_HOME_DIR sites are intentionally untouched.
• Both bare-metal and containerized (Docker) deployment paths, for both entrypoints (experiment_run_e2e.py and experiment_run_clickhouse.py).
• New top-level local: config section (added via Hydra's +local.home_dir=/path syntax, not a config-group file) replaces the originally-suggested experiment_type=local approach — checked with hasattr(cfg, "local") before any cfg.cloudlab.* access, since those fields are Hydra ??? mandatory-missing markers that raise on mere attribute access otherwise.
• Provisioning the actual binaries (Prometheus, Arroyo, asap-summary-ingest, query_engine_rust, fake_exporter) into the local home dir is out of scope — separate manual/setup concern.
• Filed follow-up issue Move CloudLab-specific sync logic onto the provider class when a second remote provider is added #435 for moving CloudLab-specific sync logic onto the provider class once a second remote provider is ever added (the is_remote() gate used here is correct for "CloudLab or local," not infinitely extensible).

Checklist

Provider & config wiring

• Add LocalProvider implementing InfrastructureProvider (single-machine: get_node_ip → 127.0.0.1, get_home_dir → configured local.home_dir, is_remote() → False)
• Wire provider-type detection into providers/factory.py (detect_provider_type now checks for local first)
• Decouple experiment_utils/config.py's Args.__init__ from unconditionally reading cfg.cloudlab.num_nodes / node_offset / username / hostname_suffix (new get_node_params()/required_cloudlab_params() helpers, used by both entrypoints)
• Add a local-mode config path (+local.home_dir=..., not experiment_type=local) that skips the cloudlab.* validation requirement

File sync (rsync-over-SSH bypasses the provider)

• experiment_utils/sync.py — all 7 rsync functions now gated on provider.is_remote()
• experiment_utils/services/query_engine.py
• experiment_utils/services/clickhouse_service.py (3 call sites — the dataset-prepare and DDL/script-asset sites fall back to a local copy instead of a no-op, since source and destination genuinely differ there even in local mode)
• experiment_utils/services/docker_victoriametrics.py

Hardcoded constants.CLOUDLAB_HOME_DIR (bypasses provider.get_home_dir())

• experiment_utils/services/flink.py — out of scope for this pass (Flink dropped, see scope refinement above)
• experiment_utils/services/arroyo.py (5 sites)
• experiment_utils/services/fake_exporters.py (2 sites)
• experiment_run_e2e.py / experiment_run_clickhouse.py — experiment_root_output_dir (both entrypoints had this same hardcoding at the top level, not just inside services; found during implementation)

Hardcoded CloudLab IP scheme

• experiment_run_e2e.py — OmegaConf resolver remote_write_ip now routes through provider.get_node_ip, registered after the provider exists (required reordering provider creation ahead of Args(cfg))

Needs audit (not yet scoped) → resolved, not just audited

• experiment_utils/services/prometheus.py reset() — turned out reset_prometheus.py (the CloudLab-only script it shelled out to) just duplicated provider.execute_command(...) with hardcoded SSH; removed the branch entirely rather than adding a provider-type check

Verification

Code-complete; not yet run against a real local experiment (binaries aren't provisioned by this change — see scope note above). Outstanding before closing:

• Run experiment_run_e2e.py locally with local.home_dir=..., streaming_engine=precompute, bare-metal — confirm no ssh/rsync subprocess is ever invoked
• Repeat with streaming_engine=arroyo
• Repeat both with containerized flags (use_container_query_engine, use_container_arroyo, etc.)
• Run experiment_run_clickhouse.py locally
• Confirm PrometheusService.reset() actually deletes local prometheus/data between runs