Skip to content

[fix](cloud) Only register watershed txn id once when BE enters decommissioning state - #67322

Open
qzsee wants to merge 1 commit into
apache:masterfrom
qzsee:fix-demission-state
Open

[fix](cloud) Only register watershed txn id once when BE enters decommissioning state#67322
qzsee wants to merge 1 commit into
apache:masterfrom
qzsee:fix-demission-state

Conversation

@qzsee

@qzsee qzsee commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
This caused two issues:

  1. Wrong guard flag. The block checked isDecommissioned() (the final decommissioned state) instead of isDecommissioning() (the in-progress state). As long as the BE had not yet reached the terminal DECOMMISSIONED state, every scheduling round of CloudClusterChecker would re-enter this branch and call registerWaterShedTxnId(be.getId()) again. The watershed txn id is only meaningful to be registered once per decommissioning session; repeatedly registering it on every check interval produces redundant RPCs/log entries and pollutes the upgrade manager's internal state.

  2. Inconsistent state on RPC failure. be.setDecommissioning(true) was executed after the try/catch block, meaning even if registerWaterShedTxnId threw a UserException and the watershed txn id was never successfully registered, the BE would still be marked as decommissioning=true. Subsequent checks would then skip the retry path entirely, leaving the BE in an inconsistent state where it is flagged as decommissioning but has no associated watershed txn id.

Fix
Change the guard from !be.isDecommissioned() to !be.isDecommissioning(), so the branch is only entered on the first transition into the decommissioning state.
Move be.setDecommissioning(true) inside the try block, right after registerWaterShedTxnId succeeds. This guarantees the in-memory decommissioning flag is only flipped when the watershed txn id has been registered successfully; if the RPC fails, the BE stays in its previous state and will be retried on the next check round.

Impact
Eliminates repeated registerWaterShedTxnId calls for a BE that is stuck in DECOMMISSIONING state.
Makes the decommissioning state transition atomic with respect to the watershed txn id registration: either both succeed, or neither takes effect (and the operation is retried next round).
No behavior change for BEs that have already reached NODE_STATUS_DECOMMISSIONED.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@qzsee
qzsee requested a review from gavinchou as a code owner August 31, 2026 02:46
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants