Repository navigation
feat(ir): define compatible logical summary merges - #560
Merged
Merged
Conversation
This was referenced Oct 3, 2026
zzylol
force-pushed
the
stack/528-02b-merge-structure
branch
6 times, most recently
from
October 3, 2026 17:29
3ab6be7 to
77628c8
Compare
zzylol
force-pushed
the
stack/528-02b-merge-structure
branch
from
October 3, 2026 17:40
77628c8 to
02a6a1b
Compare
zzylol
force-pushed
the
stack/528-02b-merge-structure
branch
2 times, most recently
from
October 3, 2026 19:45
60a9e4f to
cb00197
Compare
zzylol
marked this pull request as draft
October 3, 2026 20:02
zzylol
force-pushed
the
stack/528-02b-merge-structure
branch
from
October 3, 2026 20:48
cb00197 to
700838b
Compare
This was referenced Oct 3, 2026
zzylol
force-pushed
the
stack/528-02b-merge-structure
branch
2 times, most recently
from
October 6, 2026 20:02
f2b7b3f to
3f5d369
Compare
zzylol
marked this pull request as ready for review
October 6, 2026 20:02
zzylol
force-pushed
the
feat/summary-coverage-contract
branch
from
October 6, 2026 20:17
e5e4f67 to
fc8eec9
Compare
zzylol
force-pushed
the
stack/528-02b-merge-structure
branch
from
October 6, 2026 20:17
3f5d369 to
52b4003
Compare
zzylol
added a commit
that referenced
this pull request
Oct 6, 2026
…low multi-source summaries Review of #560/#646 found four problems: 1. Population names came from each node's schema, which `with_schema` may rename. A scan whose `tier` column is named "region" made `tier = 'eu'` read as `{region: eu}`, so a merge with a real `{region: us}` state was accepted and double-counted. Columns are now named from the Scan operator's own schema, and a path that renames a field leaves the population unknown. 2. For the same reason a merge could mix states of different columns (`Named("latency")` reading `size` on a renamed scan). An unknown population only merges with the same input, so this is rejected too. 3. `OperatorNode::map_children` dropped a SummaryAgg's coverage, so rebuilding a merge (e.g. in canonicalize) failed. A rebuild now keeps the declared time bounds and reads source and population again. 4. A SummaryAgg over two sources (a join, an IN subquery over another table) could never validate. It now carries no coverage and cannot be merged. Adds summary_coverage_derivation.rs; the four regression tests fail before this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Selvomega
requested changes
Oct 6, 2026
zzylol
added a commit
that referenced
this pull request
Oct 6, 2026
…low multi-source summaries Review of #560/#646 found four problems: 1. Population names came from each node's schema, which `with_schema` may rename. A scan whose `tier` column is named "region" made `tier = 'eu'` read as `{region: eu}`, so a merge with a real `{region: us}` state was accepted and double-counted. Columns are now named from the Scan operator's own schema, and a path that renames a field leaves the population unknown. 2. For the same reason a merge could mix states of different columns (`Named("latency")` reading `size` on a renamed scan). An unknown population only merges with the same input, so this is rejected too. 3. `OperatorNode::map_children` dropped a SummaryAgg's coverage, so rebuilding a merge (e.g. in canonicalize) failed. A rebuild now keeps the declared time bounds and reads source and population again. 4. A SummaryAgg over two sources (a join, an IN subquery over another table) could never validate. It now carries no coverage and cannot be merged. Adds summary_coverage_derivation.rs; the four regression tests fail before this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Selvomega
requested changes
Oct 6, 2026
Selvomega
requested changes
Oct 6, 2026
zzylol
added a commit
that referenced
this pull request
Oct 7, 2026
SummaryCoverage records which columns a state summarizes (input, group_by) as well as which rows. Update the SummaryAgg and SummaryMerge examples to #560: merges compare coverage columns, carry no coverage until #646, and merged_coverage is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…coverage examples SummaryCoverage no longer repeats input/reduction, so SummaryMerge compares them through OperatorNode::summary_update. summary_coverage_examples.rs builds each example in docs/develop_docs/summary-coverage.md as a SummaryAgg -> SummaryMerge plan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… doc Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A merged top-k heap can miss an item that is heavy in only one input, so CmsWithHeap, CountSketchWithHeap and UnivMon states do not merge. Also correct the merge_disjoint doc: SummaryMerge checks schema, update, reduction and heap families, not accuracy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`ASAPOp::merged_coverage` only applied to SummaryMerge and returned an error for every other operator. Replace it with `SummaryCoverage::of_merge`, which takes the merge's inputs; the callers already have them. Also explain what `OperatorNode::summary_update` returns and why merging compares it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sketches SummaryMerge no longer derives or checks coverage: of_merge, UnknownInput and MergeOutputMismatch are removed and summary_coverage.rs matches main. Coverage for all summary nodes will be derived by one function in #646. The heap-based sketch restriction is also dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It reads the producing SummaryAgg's update expression and reduction; it does not update the summary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SummaryCoverage now records which columns a state summarizes as well as which rows: `input` (the SummaryAgg update expression) and `group_by` (its reduction). with_coverage rejects a SummaryAgg declaration whose columns differ from the node's own (ColumnMismatch), and SummaryMerge requires coverage on every input (UnknownInput) with identical columns. merge_disjoint checks columns too. summary_input_data is removed. A nested SummaryMerge carries no coverage until #646, so it is rejected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol
force-pushed
the
stack/528-02b-merge-structure
branch
from
October 7, 2026 14:55
a5eb728 to
7ce0660
Compare
Remove the coverage columns (input, group_by), check_columns and the merge's coverage checks. SummaryMerge now only checks structure: at least one input, every input is State with one state column and an identical schema. Whether a structurally valid merge is semantically valid (same computation, disjoint selections) is decided by summary coverage in #646, following the design in #573. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
zzylol
added a commit
that referenced
this pull request
Oct 9, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Builds on #645 (generic IR, CSE) and #567 (summary coverage), both merged. Extracts structural summary merge support from #555.
Before this PR: constructing
ASAPOp::SummaryMergefails because the operation is reserved.After this PR: two structurally compatible KLL states can form a typed logical merge with no execution-phase assignment. Merge inputs must be nonempty, carry state with exactly one state field, and have identical family/parameter/grouping schemas. Empty, raw and mismatched states fail.
This PR is structural only and does not touch
SummaryCoverage. Equal schemas cannot tell a KLL overlatencyfrom a KLL oversize, and cannot show that the inputs cover disjoint rows. Both are decided by summary coverage, derived for every summary node by oneSummaryCoverage::derivein #646, following the design in #573.Schema/result-kind/state identity and regression tests are included. Runtime execution, materialization and derived timing belong to the later physical scopes. Subtract/delete remain reserved.
Key code interface
SummaryMergeis a logical ASAP operator whose inputs are existing operator nodes producing partial state:It uses the same node construction and validation APIs as other operators:
Here
pane_aandpane_bareRc<OperatorNode>state producers.new_sharedreturnsResult<Rc<OperatorNode>, SchemaDerivationError>and derives the output schema/result kind.validate_structurechecks the complete reachable DAG, including the producers. No timing assignment is required.The key
ASAPOpmethods are:For
SummaryMerge:validate_inputsenforces the compatibility rules below.output_schemavalidates inputs, then clones the first input's complete schema.output_kindisOperatorResultKind::State.produced_statereturns the non-plain field type from the first input. This is a metadata accessor, not a runtime merge or an independent validation step.Grouped merge keeps groups apart:
job='api'andjob='worker'stay separate states. A laterSummaryEstimateis the readout boundary that turns state into a plain value.Requirements for merging two summaries
This PR enforces these structural requirements:
childrenlist is rejected. The API is n-ary; a single compatible state is structurally allowed.result_kind == State; raw relations/vectors are rejected.Plainfield. Identical schemas enforce this on every other input too. Plain grouping fields may accompany it.FieldDataType. KLLk=200andk=300, or KLL and CMS, cannot merge here.time_index,unique_keysandclosedmust match as well. Matching only the state algorithm is insufficient.validate_structurechecks each producer's own contracts.Compatibility is deliberately strict: even differently named but otherwise equivalent schemas need an explicit normalization before this interface accepts them.
These checks establish typed structural compatibility only. They do not prove that the inputs summarize the same computation or cover disjoint rows (#646), that they cover the intended population/window, that a runtime implements the family's merge operation, or that the merged result meets an accuracy requirement. Those belong to #646, the logical composition rule and subsequent physical planning/selection. For example, overlapping frequency panes must not silently double-count observations; matching schemas alone cannot establish that.
When SummaryMerge can be used
During logical planning: a composition rule can construct
SummaryMergewhen it needs to combine compatible partial summary states—for example, several tumbling-window KLL panes answering one larger query window, or compatible partition summaries feeding a coarser computation. The planner must establish the intended input coverage and grouping semantics. The node can be constructed and structurally validated before choosing materialization.During execution: a physical implementation can merge the state contents once its inputs are available and the selected family/runtime supports the operation. Timing follows the materialization choice in #509. For example, a query can merge previously ingested/stored panes, or merge states rebuilt for that query. #560 itself adds no runtime kernel, storage, retention, window generation or execution-phase policy, and does not imply runtime support for every declared state family.
Source:
asap.rs, with regressions incrates/types/tests/summary_merge_structure.rs.Validation: workspace tests,
cargo fmt, and workspace/all-target Clippy with warnings denied all pass.Base: main · Next: #646 · Tracker: #528
Order: #645 (merged) → #567 (merged) → #560 → #646 → #539 → #540 → #541 → #542 → #543.
🤖 Generated with Claude Code