Repository navigation
Support auto-sync of tree-to-table views: detect new measurements and add the matching columns - #18874
Support auto-sync of tree-to-table views: detect new measurements and add the matching columns#18874alpass163 wants to merge 21 commits into
Conversation
Caideyipi
left a comment
There was a problem hiding this comment.
Reviewed commit f62eec46ca3f37565ca8e782f8d52d6929c0def6. The five inline findings cover a Chinese-locale compilation failure, a template-extension lock cycle, catch-up rollback for renamed source fields, stale matching metadata after view replacement, and view columns committed for rejected schema writes.
Validation:
- Compiled the PR's cache implementation and its new tests against locally cached dependencies; all four
TreeToTableViewCacheTestcases passed. - Ran two additional regression cases for same-path replacement with changed TAG count and changed restrict flag; both failed against this commit.
- Compiling the Chinese
ProcedureMessages.javawith UTF-8 produced seven syntax errors at the conflict markers. Removing only the markers in a temporary copy also exposed the duplicated commit-release constant.
The other findings were checked through the RPC, locking, column-validation and parent/child rollback paths. A full cluster integration test was not run.
51c12c8 to
11eee5f
Compare
JackieTien97
left a comment
There was a problem hiding this comment.
Reviewed head 11eee5f. auto_sync_schema is gated at analysis (the key isn't in TsTable.TABLE_ALLOWED_PROPERTIES, and TableConfigTaskVisitor.convertPropertiesToMap has no case for it, so even an allowed value would be dropped), so I un-gated it locally (8 lines), built the distribution and tested end to end on 1C1D side by side with master 8effe5d6ce, and on 1C3D with schema_replication_factor=3. I also ran IoTDBTableIT#testTreeViewTable on this head and on master.
Inline comments: P1 = blocking, P2 = performance / availability, P3 = minor. All were reproduced on a real cluster unless marked as code reading.
Caideyipi's open findings: reproduced, not repeated inline
- Template-extension lock cycle:
alter device template tE add (e2 DOUBLE)took 30,024 ms (AddTableColumn-s6db.vte-COLUMN_CHECK costs 30006ms). - Catch-up with a
FROM-mapped field: 552, and the view creation is rolled back. What the rollback leaves on the DataNodes is in the comment onCreateTableViewProcedure.java. - Same-path
CREATE OR REPLACEfrom restrict + 1 TAG to 2 TAGs: the view gotn_lvl1, whose data can't be read, instead ofn_lvl2, until a DataNode restart. - Failed creation:
create timeseries root.s12.m.x INT32fails with 506, but the view getsx INT32;root.s12.d9.x FLOAT= 2.5, created afterwards, is then invisible through the view (select ... where dev='d9'returns 0 rows; a view created afterwards hasx FLOATand returns 2.5). So the problem is not a column without data, but a wrongly typed column that hides real data.
Not inline
- Un-gating needs both the allowed-property entry and the
convertPropertiesToMapcase; theallowAutoSyncSchemacheck can't be reached today. - Cleanup: leftover
// todos (DataNodeInternalRPCServiceImpl,ClusterSchemaManager×2,TreeDeviceViewFieldDetector,TreeToTableViewCache);@parm→@paraminTreeToTableViewCache; the raw literal"Some other task is operating table with same name."inexecuteWithoutDuplicateWithoutWaiting. From reading the code: when the repeated procedure has already finished,submitAddViewColumnreportsINTERNAL_SERVER_ERRORalthough that procedure most likely succeeded. - Tests: only
TreeToTableViewCacheTest. The trigger, the ConfigNode RPC,AddViewColumnProcedure, the catch-up and the detector changes have no tests, and there is no IT for the feature.
Verified OK
- Property validation: 701 on a plain table (including
DEFAULT); invalid values and duplicate properties are rejected. - For 6 view shapes and every creation path (single, aligned, both insert auto-create forms, Session
createMultiTimeseries, template explicit / auto / batch activation), the added columns match a plain view created afterwards. Mixed-case measurements map toFROM; special-character names work. - Rename, drop, replace with auto-sync off,
SET PROPERTIESfalse / true; adding a TAG column and renaming a column keep tracking the source measurement; the index is rebuilt correctly after a DataNode restart; the catch-up at creation / enabling is correct when noFROMconflict is involved; with 3 replicas the queryId dedup works (10 new columns → 10 procedures, no errors). - zh-locale compile, en / zh key parity, spotless, and the related DataNode (94) / ConfigNode (113) unit tests pass.
…mns for related tree-to-table view
2981b5a to
7a1b1c5
Compare
# Conflicts: # iotdb-core/confignode/src/main/i18n/en/org/apache/iotdb/confignode/i18n/ProcedureMessages.java # iotdb-core/confignode/src/main/i18n/zh/org/apache/iotdb/confignode/i18n/ProcedureMessages.java # iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/schema/table/view/DropViewProcedure.java
Motivation
A tree-to-table view created with
auto_sync_schema = truemust keep its table schemaaligned with the tree timeseries its
__tree_path_patterncovers. Measurements createdafter the view existed stayed invisible to the table model until the table was altered by
hand.
Design
DataNode — detection.
TreeToTableViewCacheindexes each view by its path prefix.TriggerAddViewColumnVisitorruns fromSchemaExecutionVisitorahead of every schema-regionwrite plan:
create timeseries, aligned/multi creation, the internal create nodes behindinsert into, template activation and template extension. It resolves the written devicesagainst the index and sends the covering views with their new measurements to ConfigNode via
the new
updateTree2TableViewsRPC. The trigger is best-effort: failures are logged and neverfail the write.
ConfigNode — column addition.
AddViewColumnProcedureadds the columns, serialized pertable through
checkDuplicateTableTask. Requests are deduplicated by queryId, so the replicasreplaying one log entry yield a single operation; overlapping ones are retried within a
bounded deadline.
Index maintenance. The index is mutated only at commit-release, always through the HA
propagator, so a rollback (
CREATE OR REPLACE) restores the previous view registrationinstead of leaving a stale entry.
Catch-up. Creating a view, turning
auto_sync_schemaon for an existing one, and extendinga schema template each scan the timeseries that already exist and add whatever columns are
missing, so a view never lags behind the tree model.