Repository navigation
feat(planner): tumbling-pane window composition as a Pass 2 rule (Example 3) - #713
Merged
Merged
Conversation
…ummary A PaneChain is the SummaryAgg of a family built over each of pane_count back-to-back range windows of pane_width_ms, newest first, over the input a range selector would read. A SummaryMerge over the chain is the state of one build over the whole window: the chain's coverage is that build's, and schemas, timing and the readers' schema derivation look through the merge. `ir::pane_chain` gives the equivalences: `whole_window_build`, `pane(i)` (a range over the input shifted back i panes, folded into an existing offset) and `expand_pane_chains`, which rewrites each merge over a chain into a merge over its explicit panes, the shape the executor already runs. The executor rejects an unexpanded chain with that instruction.
…mple 3) Pass 2's window-composition rule, for one query's tumbling panes: a SummaryAgg over a range TimeRange(W), whose family merges in the family-capability table (exact accumulators included; heap top-k merges only approximately), read by queries repeating at a fixed interval E, can be built as W / w panes of w = gcd(W, E). Every non-empty subset of such builds is a candidate in which each becomes a SummaryMerge over one PaneChain, enumerated lazily after the identical-expression and summary-capability candidates, each followed by its own identical-expression candidates. A read of merged panes gets the guarantee of the same read of one build over the whole window when the family's merged error is exact or the same as that build's, and none when it is approximate. Stage 3 prices a chain as one build over its whole window plus one operation per row to place it, and a merge at one operation per state. The generic per-operator pricing now makes a TimeShift free and prices a TimeRange by the rows it keeps; a query-time scan reads the longest window above it through any time shift. Example 1 has 27 Stage 1 candidates (Q1's rate and Q2's option in six 10-s panes); Example 3b has 5 (KLL and DDSketch in five 1-min panes). Selections are unchanged: panes rebuilt at query time cost the whole build plus the merge.
The Example 3 acceptance spec and the stage viewer's Example 1 and 3b rows describe the tumbling-pane candidates and their costs.
…shifted ranges A Rate or Increase evaluation took its lookback from the TimeRange it reads and ended it at the evaluation time. Over expanded tumbling panes each pane's range is one pane wide, so rate(m[5m]) divided the merged increase by 60 s instead of 300 s (0.333 vs 0.0667), and Increase extrapolated to the newest pane's start. Under `offset`, even one build ended its window at the evaluation time instead of before the offset. The counter window is now the union of the range windows the evaluation reads, each moved back by the time shifts below its range: its lookback and its offset (`ExactEvaluation::offset_ms`) come from that union, and ranges that do not form one window are rejected as before.
The Example 1 acceptance spec counts the 27 candidates with tumbling panes. `expand_pane_chains` documents that a chain's materialization is dropped and left to Stage 2.
This was referenced Oct 15, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #601 (its window-form-as-Pass-1-axis design is replaced; #591/#592 landed in #696). Part of #509 (Example 3, Pattern B) and #580.
Why
#509's window-composition rule lets a repeating query build its summary as tumbling panes:
quantile_over_time(0.99, latency_ms[5m])every minute merges five 1-min KLLs. #696 added the family-capability table and made the executor run hand-written pane plans, but the planner never generated them. Agreed design for this step:asap_aware_cse, not a Pass 1 axis.What
ASAPOp::PaneChain. One node: aSummaryAgg's family, update, reduction, grouping and filter, built overpane_countback-to-back range windows ofpane_width_ms, over the input a range selector would read. ASummaryMergeover the chain is the state of one build over the whole window:ir::pane_chaingiveswhole_window_build,pane(i)andexpand_pane_chains.asap_aware_cse::{pane_sites, window_composition_variants}):SummaryAggover a rangeTimeRange(W)whose family has a merge in the family-capability table. Exact accumulators count; heap top-k counts, though it merges only approximately. The site must be read only by queries repeating at a fixed intervalE.w = gcd(W, E); a site needs at least 2 panes.SummaryMerge(PaneChain).ExactorSameAsUnionBuild. It gets none when the merged error isApproximate(evidence only) or the family has no merge.PhysicalOperator::{TimeShift, TimeRange}: a shift costs nothing, and a range is priced by the rows it keeps.with_resized_summaries,approximate_inputs,at_most_one_row) and the Count-Min weight check also cover chains.expand_pane_chains, using feat(types): family-capability table and executor tumbling pane merges #696's merge path.compilerejects an unexpanded chain and says to expand it.rate(m[5m])gave 0.333 instead of 0.0667, and Increase extrapolated to the newest pane's start. The window also ends before a PromQLoffset(ExactEvaluation::offset_ms). That fixes a pre-existing bug:rate(m[5m] offset 1m)gave 0.065 instead of 0.0667 even for one build.stage1_logical_asap_optimizationnow takes the queries' recurrences.Pass2Candidate/Pass2Ofgainpanes(e.g.Kll for Q1 in 5 × 1m panes), which also appears in display names.How
Pass 2 rewrites each chosen build
SummaryAgg(TimeRange(W, X))intoSummaryMerge([PaneChain{child: X, …, w, W/w}]), keeping the build's output names, then composes every guarantee again. Coverage, schema and guarantee derivations all reduce a merged chain towhole_window_build, so no per-family logic is duplicated. PaneiisSummaryAgg(TimeRange(w, TimeShift(i·w, X))), with the query's own offset folded into the shift. Expansion turns each merge over a chain into a merge over these panes, newest first.Key code interfaces added / modified
crates/types/src/ir/asap.rs,node.rscrates/types/src/ir/pane_chain.rs(new module)crates/asap-aware-mapping/src/asap_aware_cse.rscrates/asap-aware-mapping/src/pass/stage1_to_stage3.rscrates/asap-aware-mapping/src/analytical_cost.rscrates/asap-physical-operators/src/summary_kernels/exact.rsUsage, end to end (checked by running the planning stages at
3466f32fwith a scratch test, andstage_pipeline --example planner-layering-3b): Example 3, Pattern B.Before this PR
TimeShiftwas charged one operation per row.TimeShiftwas priced as one minute of data (the default lookback) whatever the window.TimeRangepassed every row of a shared scan through.TimeRange, ending at the evaluation time: one pane wide over panes, and ignoringoffset.After this PR
TimeRange)At query time the panes never win: they read the same rows as the whole build and then merge. They pay off only when kept between evaluations (Stage 2 materialization, G8).
Verification
Run on head
07a86fbd, rebased on maina18fa4a6:cargo nextest run -p asap-types -p asap-aware-mapping -p asap-physical-operators -p asap-integration-tests -p asap-devtools: 1329 passed, 31 skipped.cargo fmt --all -- --check: passed.cargo clippy --workspace --all-targets --all-features --locked -- -D warnings: passed.tools/verify_metricsql_external_consumer.sh: passed.New and changed tests:
types/tests/pane_chain.rs:(-5m, 0]like the whole build;icovers(-(o+(i+1)w), -(o+iw)]with the offset folded;asap_aware_cseunit tests:SummaryMerge(PaneChain)with the build's family, schema, coverage and guarantee;Pricing:
analytical_cost: TimeShift is free; TimeRange costs what it keeps and cannot keep more than it reads.stage3_plan_selection::time_shift_is_free_and_the_scan_reads_the_shifted_windowwas written first and failed on2c674f69: the shift cost 0.01.tumbling_pane_merge.rs: forsum_over_time,quantile_over_time,increaseandrate, the planner's chain:compile.Review-fix tests, each written first and failing before the fix:
exact_rate_over_tumbling_panes_matches_single_build: 0.0667 per series;exact_increase_off_the_pane_boundaries_matches_single_build: samples 7 s off the pane boundaries;offset_counter_window_ends_at_the_offset:rate/increase(m[5m] offset 1m)one minute later equals the plain query.Example 3:
b_window_composition_adds_tumbling_per_mergeable_option: the 5 candidates, 1-min × 5 panes, a merge only under panes;b_panes_rebuilt_at_query_time_add_the_merge: 2.100 vs 2.183;b_window_parameters_are_legalis no longer vacuous;window_formreads aPaneChainasTumbling.Example 1: Pass 1 and identical-expression tests read the whole-window candidates. New tests cover the 10-s panes (27 candidates) and pane plans costing more or being invalid.
Devtools: Example 1 counts 9 → 27, and the committed Example 1 fixture is regenerated (0.6 → 1.8 MB).
Evidence types: no screenshots (no UI change beyond the viewer's
PaneChainkind category) and no measurements; the costs above are the illustrative model's.Limitations and follow-ups
a_shared_scan_is_not_costlierstays ignored on Stage 3: price TimeShift as free and TimeRange by the rows it keeps #632: a shared scan that is not materialized is still charged per query at the longest window it serves, so sharing it costs more (80,592 vs 46,252.8/s).windows_readprices a shared scan by the longest window above it, not the union of its shifted windows (e.g.[1y] offset 2yand[1y]read 3 years of span but are priced as 1 year). This belongs with Stage 3: price TimeShift as free and TimeRange by the rows it keeps #632.Approximatewhile its whole-window read is certified (heap top-k and entropy are uncertified either way). So the "approximate merge drops the guarantee" path has no end-to-end test that tells the two apart.planner-layering-example1-acceptance.mdnow counts the 27 candidates. Its other stale passages, which predate this PR (feat(planner): Pass 2 identical-expression rule (shared-input variants) #588 notes), are left to Update the docs after the G1–G11 consolidation groups land #677.