Skip to content

Recover runs whose jobs are never claimed while the scale set has capacity #73

Description

@Mearman

A workflow run can have jobs that stay queued while the scale set has runner capacity, and what recovers it today is a person cancelling the run and re-running it. Observed once so far with an idle runner pod and two queued runs on a two-runner fleet; it is not yet known whether this is a listener or controller fault or a genuine, separate shape from starvation behind long jobs.

If it proves to be a distinct shape, a recovery CronJob in the collection, beside the stuck-runner reaper and off by default, would cancel and re-run an affected run at most once. Two design points:

  • Detect it from in-cluster signals where possible, not the GitHub API: the scale set listener reports how many jobs are assigned, and the EphemeralRunner resources show how many runners hold a job, so assigned jobs with an idle runner is observable without a permission the reaper's token may not have. The API's run and job lists are the fallback, and need the token's Actions read permission.
  • Never act on queue age alone: a job waiting behind long builds looks identical from the outside, and cancelling it re-queues it behind the same builds. Never touch a run with any job in progress; log every action.

Capture first: the listener log around the queue time, the EphemeralRunner list and the scale set's desired and current counts, the next time a job stalls with capacity free, before any recovery cancels the evidence. If the listener or controller is at fault, fix that at its source and close this as not needed.

Activity

  1. added theissue type on Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions