Skip to content

[Feature] Make abort a reliable recovery path for wedged terminating jobs #4343

Open

Description

@sachintandon

Summary

For a job wedged in TERMINATING, abort: true may update the termination reason without causing the job to be reprocessed or clearing an expired worker lock. Operators then have no run-level recovery path and may need a separate fleet-instance deletion operation.

Expected behavior

Provide a documented, idempotent force-recovery operation that safely requeues or finalizes the selected job, handles stale locks, records an event/result, and cannot terminate an instance still serving unrelated active jobs.

Acceptance criteria

  • The caller can target a specific run/job and inspect whether force recovery took effect.
  • Repeated requests are safe.
  • Active shared-instance workloads are protected.
  • Tests cover a stale terminating job, an active job, and a failed recovery attempt.

This is distinct from the older abort/container report #893; the concern here is a server-side job-processing wedge.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions