Skip to content

Support auto-sync of tree-to-table views: detect new measurements and add the matching columns - #18874

Open
alpass163 wants to merge 21 commits into
apache:masterfrom
alpass163:cyb/auto_sync_schema
Open

alpass163 wants to merge 21 commits into
apache:masterfrom
alpass163:cyb/auto_sync_schema

Conversation

@alpass163

Copy link
Copy Markdown
Contributor

Motivation

A tree-to-table view created with auto_sync_schema = true must keep its table schema
aligned with the tree timeseries its __tree_path_pattern covers. Measurements created
after the view existed stayed invisible to the table model until the table was altered by
hand.

Design

DataNode — detection. TreeToTableViewCache indexes each view by its path prefix.
TriggerAddViewColumnVisitor runs from SchemaExecutionVisitor ahead of every schema-region
write plan: create timeseries, aligned/multi creation, the internal create nodes behind
insert into, template activation and template extension. It resolves the written devices
against the index and sends the covering views with their new measurements to ConfigNode via
the new updateTree2TableViews RPC. The trigger is best-effort: failures are logged and never
fail the write.

ConfigNode — column addition. AddViewColumnProcedure adds the columns, serialized per
table through checkDuplicateTableTask. Requests are deduplicated by queryId, so the replicas
replaying one log entry yield a single operation; overlapping ones are retried within a
bounded deadline.

Index maintenance. The index is mutated only at commit-release, always through the HA
propagator, so a rollback (CREATE OR REPLACE) restores the previous view registration
instead of leaving a stale entry.

Catch-up. Creating a view, turning auto_sync_schema on for an existing one, and extending
a schema template each scan the timeseries that already exist and add whatever columns are
missing, so a view never lags behind the tree model.

@Caideyipi Caideyipi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed commit f62eec46ca3f37565ca8e782f8d52d6929c0def6. The five inline findings cover a Chinese-locale compilation failure, a template-extension lock cycle, catch-up rollback for renamed source fields, stale matching metadata after view replacement, and view columns committed for rejected schema writes.

Validation:

  • Compiled the PR's cache implementation and its new tests against locally cached dependencies; all four TreeToTableViewCacheTest cases passed.
  • Ran two additional regression cases for same-path replacement with changed TAG count and changed restrict flag; both failed against this commit.
  • Compiling the Chinese ProcedureMessages.java with UTF-8 produced seven syntax errors at the conflict markers. Removing only the markers in a temporary copy also exposed the duplicated commit-release constant.

The other findings were checked through the RPC, locking, column-validation and parent/child rollback paths. A full cluster integration test was not run.

@alpass163
alpass163 force-pushed the cyb/auto_sync_schema branch from 51c12c8 to 11eee5f Compare October 14, 2026 11:12

@JackieTien97 JackieTien97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head 11eee5f. auto_sync_schema is gated at analysis (the key isn't in TsTable.TABLE_ALLOWED_PROPERTIES, and TableConfigTaskVisitor.convertPropertiesToMap has no case for it, so even an allowed value would be dropped), so I un-gated it locally (8 lines), built the distribution and tested end to end on 1C1D side by side with master 8effe5d6ce, and on 1C3D with schema_replication_factor=3. I also ran IoTDBTableIT#testTreeViewTable on this head and on master.

Inline comments: P1 = blocking, P2 = performance / availability, P3 = minor. All were reproduced on a real cluster unless marked as code reading.

Caideyipi's open findings: reproduced, not repeated inline

  • Template-extension lock cycle: alter device template tE add (e2 DOUBLE) took 30,024 ms (AddTableColumn-s6db.vte-COLUMN_CHECK costs 30006ms).
  • Catch-up with a FROM-mapped field: 552, and the view creation is rolled back. What the rollback leaves on the DataNodes is in the comment on CreateTableViewProcedure.java.
  • Same-path CREATE OR REPLACE from restrict + 1 TAG to 2 TAGs: the view got n_lvl1, whose data can't be read, instead of n_lvl2, until a DataNode restart.
  • Failed creation: create timeseries root.s12.m.x INT32 fails with 506, but the view gets x INT32; root.s12.d9.x FLOAT = 2.5, created afterwards, is then invisible through the view (select ... where dev='d9' returns 0 rows; a view created afterwards has x FLOAT and returns 2.5). So the problem is not a column without data, but a wrongly typed column that hides real data.

Not inline

  • Un-gating needs both the allowed-property entry and the convertPropertiesToMap case; the allowAutoSyncSchema check can't be reached today.
  • Cleanup: leftover // todos (DataNodeInternalRPCServiceImpl, ClusterSchemaManager ×2, TreeDeviceViewFieldDetector, TreeToTableViewCache); @parm → @param in TreeToTableViewCache; the raw literal "Some other task is operating table with same name." in executeWithoutDuplicateWithoutWaiting. From reading the code: when the repeated procedure has already finished, submitAddViewColumn reports INTERNAL_SERVER_ERROR although that procedure most likely succeeded.
  • Tests: only TreeToTableViewCacheTest. The trigger, the ConfigNode RPC, AddViewColumnProcedure, the catch-up and the detector changes have no tests, and there is no IT for the feature.

Verified OK

  • Property validation: 701 on a plain table (including DEFAULT); invalid values and duplicate properties are rejected.
  • For 6 view shapes and every creation path (single, aligned, both insert auto-create forms, Session createMultiTimeseries, template explicit / auto / batch activation), the added columns match a plain view created afterwards. Mixed-case measurements map to FROM; special-character names work.
  • Rename, drop, replace with auto-sync off, SET PROPERTIES false / true; adding a TAG column and renaming a column keep tracking the source measurement; the index is rebuilt correctly after a DataNode restart; the catch-up at creation / enabling is correct when no FROM conflict is involved; with 3 replicas the queryId dedup works (10 new columns → 10 procedures, no errors).
  • zh-locale compile, en / zh key parity, spotless, and the related DataNode (94) / ConfigNode (113) unit tests pass.

Caideyipi

This comment was marked as outdated.

@alpass163
alpass163 force-pushed the cyb/auto_sync_schema branch from 2981b5a to 7a1b1c5 Compare October 19, 2026 11:07
# Conflicts:
#	iotdb-core/confignode/src/main/i18n/en/org/apache/iotdb/confignode/i18n/ProcedureMessages.java
#	iotdb-core/confignode/src/main/i18n/zh/org/apache/iotdb/confignode/i18n/ProcedureMessages.java
#	iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/schema/table/view/DropViewProcedure.java
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants