Skip to content

[dev/1.3] Batch aligned memtable column expansion to reduce region lock contention - #18830

Open
Caideyipi wants to merge 3 commits into
apache:dev/1.3from
Caideyipi:cp-18816-dev-1.3
Open

Caideyipi wants to merge 3 commits into
apache:dev/1.3from
Caideyipi:cp-18816-dev-1.3

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

An aligned tablet that introduces many measurements repeatedly copied full-width value, bitmap and accounting arrays while holding the DataRegion write lock. This backport registers missing measurements in insertion order and extends the TVList in one batch, resizing the reordered arrays once. Writes without schema extension keep the single-pass path, and new columns retain implicit nulls and lazy value blocks.

Cherry-picks #18816 to dev/1.3, including both 1f05d5d3fb858519d580e4df6c1b766e47c03f63 and 5f27eb1ac3bc00f8568f672a14cc1ef9bdf581a3.

The backport uses the 1.3 measurement IDs, memchunk/tablet APIs and memory-accounting method. The upstream regression tests exposed two gaps in 1.3's handling of lazy blocks: an existing bitmap could hide a null value block, and WAL sizing/writing could dereference null Binary entries or unmaterialized arrays. Handle implicit nulls before consulting bitmaps and serialize null blocks with the existing value placeholders and null flags, preserving the WAL layout. A cross-block WAL round-trip test covers every supported scalar type and verifies the serialized byte count and that serialization keeps the source value blocks lazy.

Regression coverage also includes existing/new/deleted and duplicate measurements, reordering with and without schema extension, nonzero tablet offsets, binary and primitive memory accounting, clone/clear/reuse, WAL recovery, handover, and a 10,000-column tablet followed by another write.

Validation

  • mvn spotless:apply -pl iotdb-core/datanode
  • mvn test -pl iotdb-core/datanode -am -Dtest=AlignedWritableMemChunkTest,AlignedTVListTest,MemChunkDeserializeTest,MemTableFlushTaskTest,PrimitiveMemTableTest,AlignedTVListIteratorTest -Dsurefire.failIfNoSpecifiedTests=false — all 26 reactor modules built successfully; 67 tests passed in both configured Surefire executions with no failures, errors or skips. The initial run compiled DataNode and its dependencies from a fresh worktree; the final run recompiled the compatibility changes.
  • git diff --check

This PR has:

  • been self-reviewed, including the existing region locking and query/flush ownership rules.
  • added Javadocs and comments explaining batch expansion and implicit nulls.
  • added unit tests for the new paths and 1.3 compatibility handling.

Key changed classes: AlignedWritableMemChunk, AlignedTVList.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant