Repository navigation
[Fix] Guard stopped Pipe polling and retain LOAD payloads until replicated - #18913
Merged
jt2594838 merged 2 commits intoOct 20, 2026
Merged
Conversation
Treat the default IoTConsensus V1 WAL watermark as unconfirmed replication in both retention and the staged-directory cleaner. Keep immediate release for V2 and protocols without a V1 WAL. Cover commit payload readability, terminal task recovery, and cleanup before the first watermark.
Caideyipi
changed the title
jt2594838
approved these changes
Oct 20, 2026
2 tasks done
jt2594838
pushed a commit
that referenced
this pull request
Oct 20, 2026
…8922) Partially backport 4943c73. Check the stop flag before and after polling, retain fetched events for restart, and release events returned after the task is dropped. Adapt the regression tests to the 1.3 extractor and outputPipeConnector fields. The LOAD retention changes do not apply because 1.3 sends LOAD payloads and terminal commands directly to all replicas instead of replicating staged-payload references through the WAL.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Two races can break Pipe lifecycle integration tests:
STOP PIPEfinishes while a ConfigNode task is blocked insource.supply(), the task can still transfer events created afterward.IoTDBPipeLifeCycleIT.testLifeCycleWithHistoryDisabledcan then seeroot.ln0on the receiver while the Pipe is stopped. Check the stop flag before polling and again after the source returns. Keep a fetched event inlastEventso restarting the Pipe transfers and commits it once; release it without committing if the task was dropped during polling.Long.MIN_VALUEwas treated as permission to delete staged payloads. A lagging replica then receives WAL references whose payloads are gone, which can leave the receiver missing rows after cluster restart inIoTDBPipeLifeCycleIT.testLifeCycleWithClusterRestart. Keep the staged bytes until a real watermark reaches the terminal command, in both task retention and the cleaner. IoTConsensus V2 and protocols without a V1 WAL continue to release immediately.Add deterministic regression coverage for stop/restart/drop during polling, LOAD payload readability through WAL references before the first watermark, recovery of finished LOAD tasks, and cleanup after replication is confirmed.
Validation
PipeConfigNodeSubtaskTestandPipeConfigNodeSubtaskExecutorTest: 8 tests passed.IoTDBPipeLifeCycleIT#testLifeCycleWithHistoryDisabled: previously passed with twoScalableSingleNodeModeclusters using the rebuilt ConfigNode jar.LoadTsFileManagerTestandLoadTsFileCleanerTest: 34 tests passed, covering the default watermark, recovery, WAL payload expansion, and V2 immediate cleanup.git diff --checkpassed.IoTDBPipeLifeCycleIT#testLifeCycleWithClusterRestarthas not been rerun locally after the LOAD retention fix.