Skip to content

[Pipe] Improve timeout diagnostics and DROP PIPE observability - #18297

Merged
jt2594838 merged 4 commits into
apache:masterfrom
Caideyipi:fix/pipe-timeout-diagnostics
Jul 29, 2026
Merged

[Pipe] Improve timeout diagnostics and DROP PIPE observability#18297
jt2594838 merged 4 commits into
apache:masterfrom
Caideyipi:fix/pipe-timeout-diagnostics

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

Description

This PR improves the observability of Pipe control procedures, especially when the client times out while the Procedure continues running in the background.

Improve timeout diagnostics for Pipe control statements

When CREATE PIPE, ALTER PIPE, START PIPE, STOP PIPE, or DROP PIPE times out while waiting for its Procedure, the returned diagnostic message now includes:

  • the Pipe operation and Procedure ID
  • the current execution or rollback stage
  • a stage-specific possible cause, or the latest retry/failure exception when available
  • an explicit reminder that the Procedure is still running in the background

The diagnostic state is volatile and non-persistent, so it does not affect Procedure execution or recovery. Internal consensus Pipe procedures retain their optimistic timeout behavior.

Make timed-out DROP PIPE observable through SHOW PIPES

The new DROP PIPE flow is:

VALIDATE_TASK
  -> CALCULATE_INFO_FOR_TASK
  -> PRE_DELETE
  -> OPERATE_ON_DATA_NODES
  -> WRITE_CONFIG_NODE_CONSENSUS

At PRE_DELETE, ConfigNode persists the Pipe runtime status as PRE_DELETE. Therefore:

  • while the Drop Procedure is unfinished, SHOW PIPES still returns the Pipe with state PRE_DELETE
  • after DataNode cleanup completes, the final ConfigNode consensus write removes the Pipe metadata
  • heartbeat parsing, metadata synchronization, failure recording, and auto-restart cannot overwrite PRE_DELETE
  • START PIPE, STOP PIPE, and ALTER PIPE reject a Pipe in PRE_DELETE; DROP PIPE can still be retried
  • task agents treat PRE_DELETE as a coordinator-only marker and wait for the explicit DROPPED metadata

Compatibility

The implementation reuses DropPipeProcedureV2 while preserving persisted data compatibility:

  • OperatePipeTaskState.PRE_DELETE is appended at ordinal 4; existing ordinals 0-3 are unchanged
  • PipeStatus.PRE_DELETE uses byte code 3; existing codes 0-2 are unchanged
  • persisted state history distinguishes the new flow from recovered legacy Procedures
  • legacy Procedures already at WRITE_CONFIG_NODE_CONSENSUS or OPERATE_ON_DATA_NODES continue with the old ordering
  • legacy Procedures that did not persist pipeMetaToDrop restore it from PipeTaskInfo before entering PRE_DELETE

Tests

Added coverage for:

  • timeout diagnostics and execution stages
  • new and legacy Drop Procedure state ordering
  • Procedure state ordinal and Pipe status byte compatibility
  • legacy Procedure recovery without persisted pipeMetaToDrop
  • PRE_DELETE consensus-plan and runtime-metadata serialization
  • SHOW PIPES output
  • heartbeat protection and auto-restart protection

Local verification:

  • Spotless passed for node-commons and confignode
  • ConfigNode and Node Commons Checkstyle passed
  • PipeMetaDeSerTest: 3 tests passed
  • focused state-machine tests: 7 tests passed
  • focused Drop serialization/recovery/auto-restart tests passed
  • all modified ConfigNode main sources and focused tests passed isolated compilation
  • git diff --check passed

The complete ConfigNode Maven test invocation is currently blocked locally by pre-existing generated Thrift/cache API mismatches in this checkout.


This PR has:

  • been self-reviewed
    • concurrent read and write
  • preserved persisted Procedure and Pipe status compatibility
  • added comments explaining the compatibility and lifecycle decisions
  • added unit tests for the new behavior and recovery paths

Key changed/added classes
  • AbstractOperatePipeProcedureV2
  • DropPipeProcedureV2
  • OperatePipeTaskState
  • StateMachineProcedure
  • PipeStatus
  • PipeTaskAgent
  • PipeTaskInfo
  • PipeHeartbeatParser
  • PipeMetaSyncProcedure
  • ProcedureManager

@Caideyipi Caideyipi changed the title [Pipe] Improve timeout diagnostics for Pipe control procedures [Pipe] Improve timeout diagnostics and DROP PIPE observability Jul 24, 2026
Comment on lines +2266 to 2270
private static String wrapTimeoutMessageForPipeProcedure(final TSStatus status) {
if (isProcedureTimeout(status)) {
return status.getMessage()
+ " Please manually check later whether the procedure is executed successfully.";
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Supplement i18n for this.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Replaced the raw timeout suffix with a localized full-message template in ManagerMessages and added matching English and Chinese entries. Both default and with-zh-locale full-reactor test-compile pass. Implemented in dc6c666.

@jt2594838
jt2594838 merged commit 834418e into apache:master Jul 29, 2026
41 checks passed
@jt2594838
jt2594838 deleted the fix/pipe-timeout-diagnostics branch July 29, 2026 02:01
jt2594838 pushed a commit that referenced this pull request Jul 30, 2026
… (#18340)

* Improve Pipe procedure timeout diagnostics

* Add pre-delete state to drop pipe procedure

* Address pipe timeout diagnostic review comments

* Fix pipe timeout message i18n
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants