Skip to content

Restore DataNode removal cleanup after child procedure failures - #18947

Merged
CRZbulabula merged 1 commit into
apache:masterfrom
d-wang-commit:fix-remove-datanode-rollback-cleanup
Oct 20, 2026
Merged

CRZbulabula merged 1 commit into
apache:masterfrom
d-wang-commit:fix-remove-datanode-rollback-cleanup

Conversation

@d-wang-commit

Copy link
Copy Markdown
Contributor

Description

Restore cleanup after a migration child fails

After #18909, the executor skips rollback for procedures whose isRollbackSupported() returns false. RemoveDataNodesProcedure always returned false even though its STOP_DATA_NODE rollback performs necessary node cleanup. If a migration child exhausts its metadata-removal retries, the root procedure finishes without restoring the original statuses of nodes that still host Regions or unregistering and stopping drained nodes. Removal targets can therefore remain in Removing.

Allow rollback at STOP_DATA_NODE so the existing cleanup runs while preserving the child failure and completed Region migrations. Keep execution retries independent of rollback eligibility: a transient error during normal cleanup must still retry instead of immediately failing the procedure.

Tests

  • Modified UT: strengthen RemoveDataNodePersistenceTest.testProcedureExecutorCanActuallyRetryCompensationCommitFailure to assert that a transient cleanup failure and its successful retry leave no procedure failure.
  • Existing regression: RemoveDataNodesProcedureTest.metadataRetryExhaustionRetriesFailedNodeStatusCleanup now passes, including node-status restoration, stopping drained nodes, and delayed retry after an injected cleanup failure.
  • Passed all 31 UT cases in RemoveDataNodesProcedureTest (3), RemoveDataNodePersistenceTest (23), and StateMachineProcedureTest (5), with no failures, errors, or skipped tests. Spotless passed.
  • No ITs added, modified, or run for this change. The failure and cleanup-retry paths are covered by the existing ProcedureExecutor-based UT.

Side effects and risks

A failed removal can now remain active while its existing rollback cleanup retries. Completed Region migrations are retained, and the original child failure remains reported. Rollback eligibility changes only at STOP_DATA_NODE; normal execution retains its existing retry threshold. No procedure-state IDs or serialization formats change.


This PR has:

  • been self-reviewed.
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths.

Key changed/added classes (or packages if there are too many classes) in this PR
  • RemoveDataNodesProcedure: allow failure cleanup at STOP_DATA_NODE and preserve normal execution retries.
  • RemoveDataNodePersistenceTest: verify transient cleanup failures remain retryable without leaving a procedure failure.

Allow STOP_DATA_NODE rollback cleanup under the commit-point guard.
Preserve execution retries and verify transient cleanup failures stay retryable.
@CRZbulabula
CRZbulabula merged commit 38b8969 into apache:master Oct 20, 2026
39 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants