Add LTTB downsampling as a built-in Table Model table function by DaZuiZui · Pull Request #18725 · apache/iotdb

DaZuiZui · 2026-09-24T14:05:06Z

Description

Adds LTTB (Largest-Triangle-Three-Buckets) as a built-in Table Model table function for visualization-oriented downsampling, following the conventions of the existing M4 table function.

Function definition

-- Target-count mode
SELECT * FROM LTTB(DATA => t PARTITION BY device ORDER BY time, TIMECOL => 'time', N => 500);

-- Count-window mode
SELECT * FROM LTTB(DATA => t PARTITION BY device ORDER BY time, TIMECOL => 'time', SIZE => 100, SLIDE => 100);

-- Time-window mode
SELECT * FROM LTTB(DATA => t PARTITION BY device ORDER BY time, TIMECOL => 'time',
                   SIZE => 1m, SLIDE => 1m, ORIGIN => 2026-01-01T00:00:00.000+00:00);

DATA (set semantics) / TIMECOL / SIZE / SLIDE / ORIGIN reuse the M4 argument semantics and window construction (half-open [start, start+SIZE) windows, ORIGIN alignment, count windows for integer SIZE).
N selects target-count mode: the whole partition is buffered and standard LTTB reduces each participant column to exactly N points (all points when M <= N), always preserving the first and last eligible points. Buckets follow [floor(i*(M-2)/(N-2))+1, floor((i+1)*(M-2)/(N-2))+1); the final bucket uses the last point as vertex C. Output uses window_index = 0.
SIZE selects window/bucket mode: each window is one LTTB bucket. Per participant column the anchor is chained across windows (the first eligible point seeds it), vertex C is the average point of the next window that carries data for the column (or the window's own last eligible point when there is none), and empty buckets leave the anchor untouched.
Participant columns are inferred automatically (INT32/INT64/FLOAT/DOUBLE; other types are rejected with the shared "not allowed columns" message). NULL/non-finite values are ignored per column; columns with no eligible points emit NULL. Area ties resolve to the earliest timestamp (stable input order). Integral columns keep exact long values in the output.
Output schema follows M4: window_start, window_end | window_index, partition columns, then <col>_time, <col> per participant column, aligned by output position and padded with NULL.

Validation

Rejected at analysis time: neither/both of N and SIZE; N < 3; N given as a duration; N with SLIDE/ORIGIN; SLIDE/ORIGIN without SIZE; ORIGIN in count-window mode; SLIDE mode differing from SIZE; descending ORDER BY; non-TIMESTAMP TIMECOL; unsupported participant types. All new messages are i18n constants in both en and zh CommonMessages.

Execution and distribution

TableDistributedPlanGenerator.canSplitTableFunctionProcessor now excludes LTTB (like FFT), so the whole partition is gathered and ordered (MergeSort over exchanges) before the function runs — fragment-local LTTB samples are not mergeable.
Both modes share a 64 MiB per-processor budget for point buffers across all columns; target-count selection indexes and array-growth peaks are reserved before allocation. The per-column MAX_BUFFERED_POINTS cap also remains in effect. Pending windows waiting for a non-empty successor per column share the same budget.
Triangle calculations subtract timestamps before converting to double, preserving nanosecond differences at modern epoch timestamps. Window lookahead links each column to its next non-empty closed bucket and emits completed windows in order.

Alternatives considered

Fully independent windows in SIZE mode (no chained anchor) would simplify overlapping windows but diverges from the issue's "previously selected point / next-bucket average" definition; chained anchors were chosen.
Reusing M4TableFunction's ValueOperator was rejected because LTTB requires numeric values for triangle areas.

This PR has:

been self-reviewed.
added Javadocs for most classes and all non-trivial methods.
added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
added unit and planner tests (LTTBTableFunctionTest, LTTBWindowMemoryTest, LTTBRegressionTest, TableFunctionTest). The 18 LTTB unit tests pass in both en and zh; the 24 planner tests pass in en. Memory regression suites also pass individually with a 128 MiB heap.
added integration tests (IoTDBLTTBTableFunctionIT, all four tests pass with both -PTableSimpleIT and -PTableClusterIT).

Key changed/added classes (or packages if there are too many classes) in this PR

org.apache.iotdb.commons.udf.builtin.relational.tvf.LTTBTableFunction (new)
org.apache.iotdb.commons.queryengine.plan.relational.function.TableBuiltinTableFunction
org.apache.iotdb.db.queryengine.plan.relational.analyzer.StatementAnalyzer (tryAppendLTTBModeArgument)
org.apache.iotdb.db.queryengine.plan.relational.planner.RelationPlanner
org.apache.iotdb.db.queryengine.plan.relational.planner.distribute.TableDistributedPlanGenerator
org.apache.iotdb.commons.i18n.CommonMessages (en/zh)

🤖 Generated with Claude Code

Recompute a bucket with exact arithmetic when floating-point area or lookahead calculations overflow, preserving finite peaks and stable ties. Keep the existing fast path for ordinary input values. Check count-window ends only when their start is reached, reject an unrepresentable end, and stop advancing before the next start overflows. This prevents stalled processing and spurious windows for large sizes. Add six numeric boundary regressions covering all three sampling modes, lookahead sums, opposite-sign differences, area ties, and count bounds.

JackieTien97

Two reproduced behaviors need to be addressed against the clarified definition in #18532: duplicate input timestamps must produce semantic error 701, and non-NULL floating-point values must not be silently discarded in target-count mode. Both issues reproduce in TableSimpleIT and TableClusterIT. The existing 33 LTTB unit tests, 24 planner tests, and five LTTB integration tests pass; the additional regression cases expose the gaps described inline.

JackieTien97 · 2026-10-12T11:36:32Z

+      hasRows = true;
+      capturePartitionValues(input);
+      collect(input, input.getLong(0), buffers);


[P1] Reject duplicate input timestamps with semantic error 701

This input path, and AbstractWindowLTTBDataProcessor.process, consume records without checking whether the timestamp has already occurred in the partition. With a single partition containing (time, value) = (1, 1), (1, 9), (2, 2), (3, 3), all three invocations (N => 3, SIZE => 2, and SIZE => 2ms) succeed instead of reporting the required 701 error. In the standalone and cluster reproductions, the count-window query even selected different values at timestamp 1.

Please validate timestamp uniqueness at the partition input boundary, before per-column NULL filtering and window expansion, and throw a semantic exception exposed to the client as 701. Preserve this check across input blocks and after gathering distributed input. Equal timestamps in different partitions must remain valid, and a row appearing in multiple overlapping windows must not be mistaken for a duplicate input row. This follows the clarified requirement in #18532 (comment).

JackieTien97 · 2026-10-12T11:36:32Z

+        case DOUBLE:
+          double doubleValue = record.getDouble(inputIndex);
+          if (Double.isFinite(doubleValue)) {
+            buffer.addFloating(time, doubleValue);
+          }


[P2] Do not silently discard non-NULL values in short series

The finite-value check here, and the equivalent FLOAT check, drops NaN and positive/negative infinity before counting eligible points. For three non-NULL input points (0, NaN), (1, Infinity), (2, Infinity) with N => 3, the query returns a single (window_index=0, value_time=NULL, value=NULL) row. This reproduces in both standalone and cluster execution with a table subquery computing s1 / 0.0; the input subquery itself returns all three points.

The current definition only excludes NULL participant values and requires returning all eligible points when their count does not exceed N. Please preserve those values for this short-series case, or explicitly define non-finite values as unsupported and report an error. Silently turning populated input into an all-NULL result changes the data and violates the current return-all contract.

DaZuiZui · 2026-10-12T15:17:08Z

@JackieTien97 thank

…alues Address the review findings on the LTTB table function: - Duplicate timestamps within one input partition are now rejected with SEMANTIC_ERROR (701) instead of silently selecting one of the rows. The check runs at the partition input boundary, before per-column NULL filtering and before window expansion, and carries the last seen timestamp across input blocks. Equal timestamps in different partitions stay valid, and a row shared by overlapping windows is still a single input row. - Non-NULL participant values are no longer dropped when they are NaN or +/-Infinity. The definition only excludes NULL participants, so discarding non-finite values silently turned populated input into an all-NULL result for short series. Add unit coverage for the uniqueness checker (including the block boundary) and for keeping non-finite points, plus integration coverage for all three invocations (N, SIZE count window, SIZE time window), the different-partition case, and the non-finite short series. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…eat/lttb-table-function Bring in the four LTTB fix commits from the remote branch and re-apply the review fixes on top of them: - Duplicate input timestamps within one partition are rejected with SEMANTIC_ERROR (701) at the partition input boundary, before per-column NULL filtering and window expansion, carrying the last timestamp across input blocks. Equal timestamps in different partitions stay valid. - Non-NULL participant values that are NaN or +/-Infinity are kept instead of being discarded, so a short series no longer collapses to all NULL. The memory-regression fixtures in LTTBWindowMemoryTest and LTTBRegressionTest fed the same timestamp for every buffered row; they now advance the input time per row, since repeated timestamps are invalid input. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The 701 integration test built its duplicated partition with a UNION ALL whose second branch was a literal row. That plan carries a ValuesNode with inlined rows, and dispatching such a fragment over the cluster fails in ValuesNode deserialization with BufferUnderflowException, so the test reported 301 instead of 701 under TableClusterIT. Both branches now scan the real metrics table, matching the UNION ALL shape already used by other table-model ITs. The partition still repeats every d1 row, so the duplicate-timestamp check is exercised identically, and the expected timestamp is now 0. Verified with TableSimpleIT and TableClusterIT: 8 tests pass in both. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

JackieTien97

Thank you for addressing the duplicate-timestamp (P1) and non-finite-value (P2) findings in the 10-12 commits. Four remaining items need to be resolved before merge. See inline comments for specifics.

JackieTien97 · 2026-10-13T01:21:15Z

+  private static BigDecimal exactValue(double[] values, long[] integralValues, int index) {
+    // The constructor preserves the actual binary floating-point value, including tiny deltas.
+    return integralValues == null
+        ? new BigDecimal(values[index])


[P2 follow-up] new BigDecimal(values[index]) throws NumberFormatException when the value is NaN or ±Infinity.

The fast path (argMaxArea, around line 453) returns NOT_SELECTED when the computed area is non-finite, which causes the caller to invoke argMaxAreaExactly as a fallback. The Javadoc for that method still says "Overflow fallback for finite input values", and new BigDecimal(Double.NaN) throws NumberFormatException unconditionally — it does not produce a numeric result.

The existing integration test only covers the short-series path (M <= N) where all points are returned as-is and no area is computed. A series longer than N where any bucket contains a NaN or Infinity participant will crash with an unchecked NumberFormatException at this line.

Options:

Guard exactValue: check Double.isFinite(values[index]) and treat non-finite candidates as zero area (i.e. always pick the first point in the bucket when all candidates are non-finite), or skip them.

Reject non-finite participant values at the point they enter the buffer (in collect()) with a clear 701 error.

Either way, please add a regression test for the case where N < M and at least one candidate in a bucket is NaN or Infinity.

JackieTien97 · 2026-10-13T01:21:15Z

+  public static final int MAX_BUFFERED_POINTS = 1 << 22;
+
+  /** Shared budget for point buffers and selection state across all columns of one processor. */
+  static final long MAX_BUFFER_BYTES = 64L * 1024 * 1024;


[config] MAX_BUFFER_BYTES is a compile-time constant. Please expose it as a hot-modifiable configuration parameter (following the same pattern used by other per-query memory limits in IoTDB) so operators can tune it for workloads with large partitions or many overlapping windows without restarting the node.

In addition, when a query plan includes an LTTB node, this budget should be added to the query's estimated execution memory. Currently TableFunctionOperator.calculateMaxPeekMemory() and calculateRetainedSizeAfterCallingNext() do not account for LTTB's internal buffers, so the engine cannot reclaim or back-pressure against them under memory pressure.

JackieTien97 · 2026-10-13T01:21:16Z

+    // LTTB is not mergeable: fragment-local samples cannot be combined into a globally correct
+    // result, so the whole partition must be gathered and ordered before the function runs.
+    return !TableBuiltinTableFunction.FFT.getFunctionName().equalsIgnoreCase(node.getName())
+        && !TableBuiltinTableFunction.LTTB.getFunctionName().equalsIgnoreCase(node.getName());


[planner] Please remove the LTTB name from this exclusion list.

ParallelizeGrouping + GroupNode already guarantees that, when PARTITION BY covers the device-level partition keys, each fragment receives complete partitions before any set-semantic TVF runs. That is the only property LTTB needs — it is identical to what M4 relies on to remain splittable without any exclusion here.

Excluding LTTB by name serialises all partitions through the coordinator unnecessarily and will degrade performance on large tables with many partitions.

More broadly, the non-mergeability property should live in the planner rather than in a growing name-based list. Consider expressing it as an attribute on the TVF registration or on TableFunctionProcessorNode so future non-mergeable TVFs do not require a new entry here.

JackieTien97 · 2026-10-13T01:21:16Z

+              + " the same partition.";
+  public static final String EXCEPTION_UNSUPPORTED_LTTB_VALUE_TYPE_ARG_3E60F6FE =
+      "Unsupported LTTB value type: %s";
+  public static final String EXCEPTION_LTTB_COUNT_WINDOW_BOUNDARIES_EXCEED_THE_INT64_RANGE =


[i18n] This constant name is missing its hash suffix. Following the project naming rule (<PREFIX>_<NORMALIZED>_<HASH>), it should be:

EXCEPTION_LTTB_COUNT_WINDOW_BOUNDARIES_EXCEED_THE_INT64_RANGE_5BC980CA

Please apply the same rename to the zh file to keep the two files in parity.

- Guard the exact-area fallback against non-finite participants: a NaN or Infinity candidate is scored as the smallest area (earliest point wins) instead of reaching `new BigDecimal(Double.NaN)`, which threw an unchecked NumberFormatException whenever N < M and a bucket held such a value. The fallback still runs for a non-finite lookahead average over representable points, since it rebuilds the sum from the original points. - Replace the compile-time MAX_BUFFER_BYTES with the hot-reloadable `lttb_buffer_size_in_bytes` parameter, resolved through loadFixedSizeLimitForQuery so the default scales with the node's operator memory (as sort_buffer_size_in_bytes does) rather than charging a fixed 64 MB to every LTTB fragment, which tripped the fragment admission check on small-memory nodes. - Report the LTTB buffers to the engine: a new TableFunctionProcessorProvider#getEstimatedMaxBufferedBytes hook feeds TableFunctionOperator's peek and retained estimates, matching how AbstractSortOperator accounts for its sort buffer. - Drop LTTB from the planner's name-based split exclusion. The partitioned GroupNode the function requires already guarantees whole partitions per fragment, so fragment-local samples are valid standard LTTB results. - Give the count-window INT64 overflow constant its hash suffix. Add unit coverage for non-finite sampling (N < M, all-non-finite bucket) and integration coverage for the new configuration parameter.

…ary rules Document what the clarification fixes and pin it with tests: - SIZE defines a bucket, not the input range of a separate LTTB invocation. One anchor chain per partition and participant column: A is the point the preceding bucket selected, B becomes the anchor of the following bucket, and overlapping buckets (SLIDE < SIZE) do not restart the chain. State stays independent between partitions and between columns. - Bucket mode emits one row per window that carries input, at most one point per column. A column with no eligible point emits NULL, and an all-NULL window still emits a single all-NULL row. - Bucket mode does not preserve the first and last eligible points: an endpoint used as A or as C is a triangle vertex, not a selected point, so it is returned only when it also wins its bucket. A bucket with two eligible points contributes one row, the tie resolving to the earlier point. First/last preservation and return-all-points remain target-count behaviour. Tests: the anchor chain across overlapping buckets (with the count-window and time-window expected indexes), two eligible points in one bucket emitting one row, and an all-NULL column still emitting one NULL row per window. The count-window case selects t=1 in bucket [1,4), which is only reachable with a chained anchor.

feat(table): add LTTB table function

1284723

DaZuiZui mentioned this pull request Sep 24, 2026

[Feature request] Add LTTB downsampling as a built-in table function for the Table Model #18532

Open

1 of 2 tasks

DaZuiZui added 4 commits October 8, 2026 16:00

fix(table): bound LTTB overlapping window buffer memory

472494e

fix(table): correct LTTB memory bounds and sparse sampling

b2f9b77

fix(table): preserve LTTB integer precision and guard window bounds

4074d36

JackieTien97 requested changes Oct 12, 2026

View reviewed changes

DaZuiZui and others added 3 commits October 13, 2026 00:33

JackieTien97 requested changes Oct 13, 2026

View reviewed changes

DaZuiZui added 2 commits October 13, 2026 12:40

Navigation Menu

Repository navigation

Add LTTB downsampling as a built-in Table Model table function - #18725

Add LTTB downsampling as a built-in Table Model table function#18725
DaZuiZui wants to merge 10 commits into
apache:masterfrom
DaZuiZui:feat/lttb-table-function

DaZuiZui commented Sep 24, 2026 •

edited

Loading

Uh oh!

JackieTien97 left a comment

Uh oh!

JackieTien97 Oct 12, 2026

Uh oh!

JackieTien97 Oct 12, 2026

Uh oh!

DaZuiZui commented Oct 12, 2026

Uh oh!

JackieTien97 left a comment

Uh oh!

JackieTien97 Oct 13, 2026

Uh oh!

JackieTien97 Oct 13, 2026

Uh oh!

JackieTien97 Oct 13, 2026

Uh oh!

JackieTien97 Oct 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

DaZuiZui commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Function definition

Validation

Execution and distribution

Alternatives considered

Key changed/added classes (or packages if there are too many classes) in this PR

Uh oh!

JackieTien97 left a comment

Choose a reason for hiding this comment

Uh oh!

JackieTien97 Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

JackieTien97 Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

DaZuiZui commented Oct 12, 2026

Uh oh!

JackieTien97 left a comment

Choose a reason for hiding this comment

Uh oh!

JackieTien97 Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

JackieTien97 Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

JackieTien97 Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

JackieTien97 Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

DaZuiZui commented Sep 24, 2026 •

edited

Loading