Skip to content

Refactor shared Pipe, consensus and subscription helpers - #18837

Open
Caideyipi wants to merge 1 commit into
apache:masterfrom
Caideyipi:refactor/align-shared-pipe-consensus-subscription
Open

Caideyipi wants to merge 1 commit into
apache:masterfrom
Caideyipi:refactor/align-shared-pipe-consensus-subscription

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

Shared Pipe, IoTConsensus, and subscription code has diverged structurally between upstream and downstream, creating avoidable merge conflicts. Align the common interfaces, helper methods, and names while preserving the current open-source behavior.

  • Introduce PipeBatchMetricsSettable for sink and batch-builder metrics, and extract batch detachment without changing atomic ownership handoff.
  • Extract tree-pattern splitting, modification-map construction, schema-snapshot batch flushing, and ConfigNode snapshot file-list helpers; align source naming and database/table matching.
  • Extract consensus retry, missing-group response, and snapshot-cleanup helpers while retaining existing retry decisions, status codes, messages, and locking.
  • Move tablet projector initialization into TableViewTabletProjectorProvider and extract consensus tablet pruning. Providers remain scoped to individual batches, preserving schema waits and the lifetime of each batch's cached projector.

Existing public constructors and metric setters remain available. Serialization, column filtering, WAL retry protection, memory admission, and diagnostics retain their current behavior. Enterprise-specific functionality is outside this refactor.

Validation

  • 185 focused regression tests passed, covering Pipe batching/sinks/receivers/modifications/history, consensus RPC responses and retry handling, subscription conversion/prefetching/projectors, and ConfigNode request serialization.
  • Full reactor mvn test-compile -DskipTests -Ddevelocity.off=true.
  • Full reactor mvn test-compile -DskipTests -P with-zh-locale -Ddevelocity.off=true.
  • Spotless, Checkstyle, and git diff --check.
Focused regression command
mvn test -Dtest=TreePatternParseTest,IoTConsensusRPCServiceProcessorTest,DispatchLogHandlerTest,PipeTransferBatchReqBuilderTest,ModsOperationUtilTest,IoTDBDataNodeReceiverTest,IoTDBDataRegionAsyncSinkTest,IoTDBDataRegionAirGapSinkTest,PipeHistoricalDataRegionTsFileAndDeletionSourceTest,PipeSinkSubtaskTest,ConsensusLogToTabletConverterTest,ConsensusPrefetchingQueue*Test,TableViewTabletProjectorProviderTest,SubscriptionPipeTabletEventBatchTest,ColumnFilterBinderTest,PipeConfigNodeThriftRequestTest -Dsurefire.failIfNoSpecifiedTests=false -Ddevelocity.off=true

This PR has:

  • been self-reviewed, including batch synchronization and ownership.
  • added regression tests for missing consensus groups and projector schema/cache lifetime, and adapted the historical-source reflection test to the renamed field.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant