feat(planner): run the planning stages as the planner pipeline; retire MajorPass

[zzylol]

Why

#576 added Stage 2 (physical ASAP-aware optimization) and Stage 3 (plan selection), but the planner facade still ran MajorPass, the legacy search. Many tests and devtools also still called global_selection, search_workload or assemble_selected_dag. This is G1 of the upper-stack consolidation plan: it makes the planning stages the planner pipeline.

What

• run_stage1_to_stage3 (asap_aware_mapping::pass, re-exported from the crate root and from asap_planner; named plan_stages in this PR, renamed in refactor(planner): name the planner pipeline Stage 1 to Stage 3 and its fields by stage #675) runs the stages in order on the roots from Stage 0 (query-language frontends), which the facade lowers first:

  1. Stage 1 (logical ASAP-aware optimization): Pass 1 per-computation candidates, then composition of every combination;
  2. Stage 2 (physical ASAP-aware optimization): optimize_physical_asap;
  3. Stage 3 (plan selection): select_plan.

  It returns a Stage1ToStage3Run with fields named by stage: stage1_pass1_per_computation (the per-computation candidates), combinations, stage1_logical_asap_optimization (the logical candidates, with per-candidate composition errors), stage2_physical_asap_optimization and stage2_pruned, and stage3_plan_selection. Each candidate carries a display_name. There is no cap on enumeration. Early search-space pruning is Prune the candidate search space early without discarding a possibly-best valid plan #673.

• Stage1ToStage3Pass, an OptimizationPass registered as "stage1-to-stage3", replaces MajorPass as the facade default (let default_pass = Stage1ToStage3Pass). It:

  • adds the PromQL series identity where it can;
  • plans scalar roots against their entry's accuracy target;
  • outputs the selected physical candidate's roots.

  PlanOutput.selection reports the choice. OptimizeError gains PerComputationCandidates and Selection, and drops Realization and Search.

• One pipeline implementation: test: planner-layering Examples 1–4 acceptance specs on the planning stages #577's devtool run_planning_stages now calls run_stage1_to_stage3.

• Legacy callers ported to the pipeline: executor, frontend-promql, frontend-sql, planner and integration tests, and the sketch_coverage devtool (now Pass 1 summary candidates). Callers that G2 deletes are left alone.

• Composition fix: a per-series avg composes over rows that carry the series identity. The regression test was written first.

• Cost independence: Pass 1, composition and physical ASAP-aware optimization import no cost model (tests/cost_independence.rs, from refactor(planner): Stage 1 no longer depends on the cost model #582).

• Stage identifiers carry number and name:

  • modules: stage2_physical_asap_optimization, stage3_plan_selection;
  • tests: stage1_logical_asap_optimization.rs, stage2_physical_asap_optimization_sql.rs;
  • stage_pipeline JSON keys: stage0_query_language_frontends, stage1_logical_asap_optimization, stage2_physical_asap_optimization, stage3_plan_selection.

  Code docs say "query-language frontends". The function names optimize_physical_asap, select_plan and generate_per_computation_candidates are kept.

How

run_stage1_to_stage3 enumerates combinations in a fixed order (last target fastest), composes each one, implements each one physically, prices all of them, and selects one. Shared NotMaterialized nodes are charged once per consuming query (#576).

Before this PR

e2e_plan and the facade ran MajorPass (legacy search with DefaultCostModel). Only the devtool showed what plan selection would choose.

After this PR

The facade returns plan selection's choice over every candidate. The Example 1 fixture is unchanged apart from the stage-key renames. MajorPass merged structurally identical producers (share_common_sub_dags) and identical scalar references; the pipeline does neither until Pass 2 sharing (#580), so two identical queries deploy separate states and scalar(rate(x[5m])) - scalar(rate(x[5m])) yields two scalar plans. The facade fails with OptimizeError::Selection if no candidate is selectable, which only happens when physical ASAP-aware optimization prunes the exact candidate.

Evidence

• Test list: 2 cost-independence tests; the avg regression test; enumerates_every_combination_and_selects_one (crates/planner/tests/plan_stages.rs); and kubernetes_mixin_aggregates_over_topk_compose, ignored and waiting for Top-k results drop the item columns (port/job); heavy-hitter candidates resolve columns by name #672.
• Corpus totality: composition errors are 0 except two Kubernetes mixin alerts that aggregate over a topk output. The frontend declares that output as [keys.., topk_k: Utf8] with no value column, so the outer sum has nothing numeric to read. The fix is Top-k results drop the item columns (port/job); heavy-hitter candidates resolve columns by name #672's top-k output schema.
• Newly ignored tests, each with its reason:
  • Pass 2 sharing and Stage 1 coverage parity with the retired MajorPass #580 (Pass 2 sharing): 8 tests.
  • feat(planner): whole-expression top-k heap sketch over raw samples #589 (whole-expression heap): 2.
  • Top-k results drop the item columns (port/job); heavy-hitter candidates resolve columns by name #672 (top-k schema): 3, including the Kubernetes mixin test.
  • feat(planner): Pass 1 offers HydraCms for grouped approximate counts #605 (Hydra): 1.
  • Stage 2 materialization: ingestion vs query time per summary #604 (materialization): 1.

Verification

• cargo fmt --all -- --check
• cargo clippy --workspace --all-targets -- -D warnings
• cargo nextest run --workspace: 1632 passed, 0 failed, 50 skipped (including test: planner-layering Examples 1–4 acceptance specs on the planning stages #577's ignored acceptance tests)
• dag-viewer test_render.py: OK
• tools/verify_metricsql_external_consumer.sh: OK

Architectural decisions

• feat(planner): physical ASAP-aware optimization and plan selection for candidate logical ASAP DAGs #576's enumeration is kept, and feat(planner): run the #509 stage pipeline with tree-DP selection; retire MajorPass #581's tree DP is deferred. Search strategies are Planning roadmap: move ASAPQuery planning into ASAPPlanner, MILP plan selection over shared DAGs, benchmark-based accuracy #671; early pruning is Prune the candidate search space early without discarding a possibly-best valid plan #673.
• No Pass 2 sharing yet (Pass 2 sharing and Stage 1 coverage parity with the retired MajorPass #580). The canonical top-k output schema is Top-k results drop the item columns (port/job); heavy-hitter candidates resolve columns by name #672's.
• Stage identifiers carry number and name. Stage 0 is "query-language frontends". The design docs keep their own wording.

Later changes: #675 renamed the pipeline without changing behavior: plan_stages → run_stage1_to_stage3, StagePipelineRun → Stage1ToStage3Run, StagePipeline → Stage1ToStage3Pass (registered as "stage1-to-stage3"), the run's fields to stage1_pass1_per_computation etc., and the candidate label (also in the stage_pipeline JSON) → display_name.

Limitations and follow-up

• Top-k results drop the item columns (port/job); heavy-hitter candidates resolve columns by name #672: top-k output schema, plus the two Kubernetes mixin alerts.
• Pass 2 sharing and Stage 1 coverage parity with the retired MajorPass #580, feat(planner): whole-expression top-k heap sketch over raw samples #589, Stage 2 materialization: ingestion vs query time per summary #604, feat(planner): Pass 1 offers HydraCms for grouped approximate counts #605: the ignored tests above.
• Deployment inputs: capabilities and raw-data retention #609: deployment capabilities. The exact plan for quantile(...) has no native runtime implementation.
• Prune the candidate search space early without discarding a possibly-best valid plan #673: early search-space pruning, since enumeration is exponential in the number of targets.

Absorbs: #581 (except the tree DP and the top-k schema commits), #579 (superseded by #672 and #577), #588's pipeline (plan_stages) part, #582's cost-independence test, #629, #633. Those PRs will be closed and will link here.

Stack: tracker #528 · main → G1 → G2 (legacy removal) → G3 (crates by planning stage) → …

Human review — do not complete with an agent

• The MVP boundary is correct.
• New conceptual layers or public interfaces are necessary.
• The before/after description matches the intended product behavior.
• Human reviewer:
• Decision and rationale:

🤖 Generated with Claude Code

https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW