Skip to content

feat(sds): define stored summaries from the plan's ingestion-time outputs - #819

Draft
zzylol wants to merge 1 commit into
feat/physical-dag-handofffrom
feat/sds
Draft

zzylol wants to merge 1 commit into
feat/physical-dag-handofffrom
feat/sds

Conversation

@zzylol

@zzylol zzylol commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Stack 2/5. Base: #818. Next: #820.

Before this PR

The data plane loaded the plan but didn't know which outputs it had to store.

After this PR

Sds::from_dag turns the plan's ingestion-time frontier into stored-summary definitions:

  • Pane shape: each frontier output must read TimeRange(width) ← TimeShift(offset)* ← Scan. That makes it a pane of width ending at T − offset; any other shape is rejected as UnsupportedShape.
  • Shared definitions: outputs that differ only in their offset share one definition. A definition is identified by a SHA-256 over its payloads and schemas, with the TimeShift offsets zeroed. In Example 4b, the six 10-minute KLL panes become one definition with offsets 0, 10, …, 50 min.
  • Storage: SummaryStore keeps panes by (definition, end) and evicts them by end time.

Testing

data_plane/tests/sds.rs (6 tests): sharing, offsets, identity, rejected shapes, store eviction.

🤖 Generated with Claude Code

…puts

Each ingestion-time output the query side reads (the executor's
materialization frontier) reads one tumbling pane of raw samples. Outputs that
differ only in which pane they read share one SummaryDefinition, identified
by a SHA-256 of their sub-DAG with TimeShift offsets zeroed. Each output binds
to its definition with an offset: at evaluation time T it reads the pane
ending at T - offset. SummaryStore keeps built panes by definition and end.

#509 Example 4b's six 10-minute KLL panes become one definition with offsets
0..50 min.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant