Skip to content

[Subscription] Index consensus ACK progress by writer - #18835

Open
Caideyipi wants to merge 1 commit into
apache:masterfrom
Caideyipi:fix/consensus-subscription-ack-scan
Open

Caideyipi wants to merge 1 commit into
apache:masterfrom
Caideyipi:fix/consensus-subscription-ack-scan

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

Description

ConsensusSubscriptionCommitState.stageCommittedAndAdvance previously found the next committed and outstanding entries by scanning the full backlog maps. With N outstanding entries from one writer and reverse-order ACKs, every ACK repeatedly scanned O(N) entries, producing a real O(N^2) ACK path.

I reproduced the behavior by calling the production ConsensusSubscriptionCommitState with a single writer:

backlog baseline reverse-ACK time indexed time
2,000 39.749 ms 3.116 ms
4,000 152.420 ms 5.222 ms
8,000 1,116.485 ms 5.737 ms

Each value is the best of three warmed runs. The baseline grows super-linearly as the backlog doubles; the indexed path does not. The new per-writer TreeSet indexes preserve the existing slot maps for exact ACK removal and make successor lookup logarithmic.

Remote-change check:

Testing

  • mvn -Ddevelocity.off=true spotless:apply -pl iotdb-core/datanode
  • mvn -Ddevelocity.off=true -pl iotdb-core/datanode checkstyle:check -DskipTests
  • ConsensusSubscriptionCommitStateTest: 19/19 passed, including a 2,048-entry reverse-ACK regression case.
  • git diff --check

The normal full DataNode compile is currently blocked by existing workspace SNAPSHOT/source inconsistencies (SubscriptionQueueAdmission and related classes) and a Windows compiler native-memory failure; the changed manager and targeted test were compiled directly against the existing DataNode build artifacts.

Changed files

  • ConsensusSubscriptionCommitManager
  • ConsensusSubscriptionCommitStateTest

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant