feat(query-engine): roll up single-subpopulation aggregations to a coarser grouping by milindsrivastava1997 · Pull Request #848 · ProjectASAP/ASAPQuery

milindsrivastava1997 · 2026-10-13T02:55:45Z

Part 2/5 of #839. Stacked on #847. Stack: Sliding cover fix (#847) → engine roll-up (this) → compliance suite → MILP sketches → MILP exact types.

What

The engine can now answer a query grouped by Q from a single-subpopulation aggregation grouped by G ⊋ Q, by merging the states of every G-group with the same projection onto Q. For example, quantile by (service) can be served from a KLL grouped by (endpoint, service).

No config change. The roll-up is inferred from G (value config grouping_labels) vs Q (query_output_labels).
Where (read_range_query_inputs): right after the store read, roll_up_groups re-keys each G-group by its projection and appends its buckets. The existing per-step get_merged() then folds time and subpopulations in one batch merge, and fallback_key becomes the Q-key, so output labels come out right. Instant (run as a one-step range) and range queries, and PromQL, SQL and Elastic, all pass through here.
Gate (rollup_projection): only applies when the value type isn't keyed, there's no key aggregation, aggregated_labels is empty and the statistic isn't top-k. Otherwise behaviour is unchanged.
Errors (not fall-through, since either means a wrong config):
- Q ⊄ G (can't serve).
- A roll-up of a type outside {Sum, MinMax, DatasketchesKLL, DDSketch, HLL}, e.g. Increase (can't be merged across groups).
Sliding: the instant all-or-nothing cover check still runs per raw G-key. The per-step range check counts distinct windows (fix(query-engine): judge a Sliding range step's cover by distinct windows #847), so a subgroup with no samples in one cover window still contributes its other windows.

Tests (`src/tests/rollup_tests.rs`, `roll_up_groups_tests` in `mod.rs`)

Sum: a roll-up equals the same data deployed at Q. Also sum(x) with no grouping rolls everything into one group, and dropping a middle grouping label keeps the remaining label values aligned.
MinMax: max by (service) over subgroups.
KLL: within rank error of the union's median. KLL compaction is randomized, so a second hand-merge isn't bit-identical.
DDSketch: equals the sketch of the union. HLL equals the union sketch, tested through SQL COUNT(DISTINCT), the only path that reaches HLL.
Range, Tumbling: a subgroup appearing mid-range. Range, Sliding: a subgroup missing an early cover window.
Errors: Q ⊄ G, Increase roll-up, and a malformed store key in roll_up_groups.

Existing suites are unchanged: 667 lib tests and the workspace tests pass.

🤖 Generated with Claude Code

…arser grouping A query grouped by Q can now read an aggregation grouped by G ⊋ Q. Right after the store read, every G-group is re-keyed by its projection onto Q, so the per-step merge folds all subpopulations of an output group into one state before the statistic is queried. Instant and range queries, and all query languages, share this path. Only Sum, MinMax, DatasketchesKLL, DDSketch and HLL roll up. A query whose grouping isn't a subset of the aggregation's, or a roll-up of any other type, is an error rather than a fall-through. Keyed and top-k aggregations keep their existing path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

milindsrivastava1997 · 2026-10-13T05:35:43Z

            values.len(),
            values.values().map(|buckets| buckets.len()).sum::<usize>()
        );
+        let values = match rollup_projection {


Instant Sliding roll-ups fail if any subgroup has an incomplete cover. execute_sliding_cover_query runs its all-or-nothing check (output_timestamps.len() == 1) on the raw G-keys, before this roll-up. So if any one (endpoint, service) subgroup started partway through the lookback, the whole instant query returns NoLocalData and falls through, even though every Q-group is servable. A one-step range query at the same timestamp would answer it. The finer G is, the more often this happens. There's no instant Sliding roll-up test. Consider doing the instant completeness check after the roll-up (per Q-key), or using the per-step distinct-window check for both.

milindsrivastava1997 · 2026-10-13T05:35:44Z

            values.len(),
            values.values().map(|buckets| buckets.len()).sum::<usize>()
        );
+        let values = match rollup_projection {


Sliding range: rolling up before the per-step cover check can hide a subgroup's missing window. sliding_cover_window_count now runs on the merged Q-group map, so a window counts as present if any subgroup has it. Suppose subgroup /x really is missing [4s,6s) (late or lost flush) while /new has it. The step passes as complete and returns a sum that silently leaves out /x's window, where before that series' step would have been skipped. This is fine for series that truly had no samples, but it can't be told apart from missing data.

milindsrivastava1997 · 2026-10-13T05:35:44Z

+            .iter()
+            .map(|label| grouping.iter().position(|g| g == label))
+            .collect::<Option<Vec<usize>>>()
+            .ok_or_else(|| {


This is a hard error, not a fallback. Q ⊄ G and non-mergeable roll-ups return QueryExecutionError::Native, which map_local_execution_outcome passes through as Err. The HTTP server then returns an error instead of forwarding to Prometheus (that only happens on NoLocalData/None). A query config that points a query at an incompatible aggregation used to get an answer (if mislabelled). Now the user gets a failure and there's no fallback. Consider NoLocalData plus a warn log, so the query is still answered upstream.

milindsrivastava1997 · 2026-10-13T05:35:46Z

+        // The store sorts each group's buckets chronologically; keep that
+        // order across the merged groups.
+        for buckets in rolled.values_mut() {
+            buckets.sort_by_key(|(range, _)| *range);


Merge order isn't deterministic. sort_by_key is stable on the range only, so buckets from different subgroups with the same range stay in values' HashMap iteration order, which is randomized per process. Float Sum merges (non-associative) and KLL merges can then change from run to run. The sort also adds little: build_bucket_map regroups by start timestamp anyway. Either drop it, or sort by (range, source key) if you want determinism.

milindsrivastava1997 · 2026-10-13T05:35:47Z

+            && context.metadata.statistic_to_compute != Statistic::Topk;
+        let grouping = &context.grouping_labels.labels;
+        let output = &context.metadata.query_output_labels.labels;
+        if !single_subpopulation || grouping == output {


grouping == output is an order-sensitive Vec equality. It relies on both KeyByLabelNames having been built through new() (which sorts), but labels is a pub field and some code builds it as a struct literal (e.g. planner/elastic_dsl.rs KeyByLabelNames { labels: rollup }). The same set in a different order would go down the projection path, and for Increase it would error with "can't be merged across groups". A set comparison (or projection == 0..n) would be safer.

milindsrivastava1997 · 2026-10-13T05:35:48Z

+/// Single-subpopulation types whose merge of two groups' states is the state
+/// of their union. Increase is excluded: its merge joins pieces of one
+/// counter in time.
+const ROLLUP_AGGREGATION_TYPES: [AggregationType; 5] = [


ROLLUP_AGGREGATION_TYPES is really a property of AggregationType, the same kind of thing as is_keyed() in promql_utilities/.../enums.rs. An exhaustive match method there (e.g. is_mergeable_across_groups) would force a decision whenever a new variant is added. With this list, a new variant silently becomes non-roll-upable.

milindsrivastava1997 · 2026-10-13T05:35:49Z

+        assert_eq!(rows.len(), 2);
+        assert_eq!(rows[0].0, vec!["a".to_string()]);
+        assert!(
+            (rows[0].1 - 500.0).abs() <= 1_000.0 * 0.02,


Possible flake. KLL k=200 has a normalized rank error of about 1.65%, but that's a probabilistic bound (about 99% confidence), and compaction is randomized. Unless the sketch's RNG is seeded, this 2% bound on 1000 items can occasionally fail. Consider a wider tolerance, or seeding the sketch.

milindsrivastava1997 · 2026-10-13T05:35:50Z

+                    ))
+                })?;
+            let projected = KeyByLabelValues::new_with_labels(
+                projection.iter().map(|&i| key.labels[i].clone()).collect(),


Small thing: key is owned here, so the projected values could be moved out with std::mem::take(&mut key.labels[i]) (the projection indices are distinct) instead of cloning one String per label per store group.

This was referenced Oct 13, 2026

test(promql-compliance): roll-up suite with fixed planner output #850

Open

feat(planner): MILP roll-ups for KLL, DDSketch and HLL #851

Open

feat(planner): MILP roll-ups for exact sum, count, min and max #852

Open

milindsrivastava1997 deleted the branch 839-query-engine-merge-sketches-across-subpopulations-roll-up-to-a-coarser-grouping October 13, 2026 03:37

milindsrivastava1997 closed this Oct 13, 2026

milindsrivastava1997 mentioned this pull request Oct 13, 2026

feat(query-engine): roll up single-subpopulation aggregations to a coarser grouping #853

Merged

milindsrivastava1997 commented Oct 13, 2026

View reviewed changes

Navigation Menu

Repository navigation

feat(query-engine): roll up single-subpopulation aggregations to a coarser grouping - #848

feat(query-engine): roll up single-subpopulation aggregations to a coarser grouping#848
milindsrivastava1997 wants to merge 1 commit into
839-query-engine-merge-sketches-across-subpopulations-roll-up-to-a-coarser-groupingfrom
839-2-engine-rollup

milindsrivastava1997 commented Oct 13, 2026

Uh oh!

milindsrivastava1997 Oct 13, 2026

Uh oh!

milindsrivastava1997 Oct 13, 2026

Uh oh!

milindsrivastava1997 Oct 13, 2026

Uh oh!

milindsrivastava1997 Oct 13, 2026

Uh oh!

milindsrivastava1997 Oct 13, 2026

Uh oh!

milindsrivastava1997 Oct 13, 2026

Uh oh!

milindsrivastava1997 Oct 13, 2026

Uh oh!

milindsrivastava1997 Oct 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

milindsrivastava1997 commented Oct 13, 2026

What

Tests (src/tests/rollup_tests.rs, roll_up_groups_tests in mod.rs)

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Tests (`src/tests/rollup_tests.rs`, `roll_up_groups_tests` in `mod.rs`)