Skip to content

[Subscription] Restore consensus subscription brokers after restart - #18916

Merged
jt2594838 merged 1 commit into
apache:masterfrom
Caideyipi:fix/subscription-restart-recovery
Oct 20, 2026
Merged

jt2594838 merged 1 commit into
apache:masterfrom
Caideyipi:fix/subscription-restart-recovery

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

After a full DataNode restart, subscription metadata can be restored before IoTConsensus has any local DataRegions. The initial subscription setup therefore finds no regions. Recovering existing peers did not invoke the callback used to bind subscription queues, leaving the consumer group without a consensus broker even when its consumers reconnected.

Notify the existing peer-availability callback after each recovered peer starts. Keep the pre-start recovery callback ahead of start() so WAL retention guards are installed first. Share the existing exception handling with peer creation so a failed notification does not prevent other peers from recovering.

The existing subscription binding and progress recovery logic is reused. This change does not alter WAL durability or the commit-progress protocol.

Regression coverage

  • Two consensus tests exercise real peer persistence and recovery: recovery notifications run after the state machine starts, and one failed notification does not interrupt recovery of other regions. Both tests failed before the fix and pass afterward.
  • A table subscription IT force-stops all ConfigNodes and DataNodes, recreates two consumers with the same IDs and group, and checks that 64 pending rows arrive before any new writes. It then verifies delivery of 16 new rows. SYNC WAL and flushed writes isolate subscription recovery; row-key sets allow redelivery around the forced shutdown.

Validation

  • mvn test -pl iotdb-core/consensus -am -Dtest=IoTConsensusRecoveryTest -Dsurefire.failIfNoSpecifiedTests=false: 2 tests passed, no failures/errors/skips.
  • mvn test-compile -DskipTests -P with-integration-tests: full 54-module reactor passed.
  • mvn test-compile -DskipTests -Dcheckstyle.skip=true -P with-integration-tests,with-zh-locale: full 54-module Chinese reactor passed; Checkstyle was already covered by the English reactor.
  • IoTDBConsensusSubscriptionRecoveryTableIT: passed in 29 seconds in an isolated local 1C1D subscription-enabled runtime assembled from current reactor classes. The standard community runtime disables subscriptions and excludes subscription ITs from normal execution.
  • Spotless applied to both changed modules; git diff --check passed.

This PR has:

  • been self-reviewed.
  • added comments explaining the recovery ordering.
  • added unit and integration regression tests.
  • been tested in a local IoTDB cluster.

@jt2594838
jt2594838 merged commit 9647e38 into apache:master Oct 20, 2026
37 of 39 checks passed
@jt2594838
jt2594838 deleted the fix/subscription-restart-recovery branch October 20, 2026 14:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants