You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Recover runs whose jobs are never claimed while the scale set has capacity #73
A workflow run can have jobs that stay queued while the scale set has runner capacity, and what recovers it today is a person cancelling the run and re-running it. Observed once so far with an idle runner pod and two queued runs on a two-runner fleet; it is not yet known whether this is a listener or controller fault or a genuine, separate shape from starvation behind long jobs.
If it proves to be a distinct shape, a recovery CronJob in the collection, beside the stuck-runner reaper and off by default, would cancel and re-run an affected run at most once. Two design points:
Detect it from in-cluster signals where possible, not the GitHub API: the scale set listener reports how many jobs are assigned, and the EphemeralRunner resources show how many runners hold a job, so assigned jobs with an idle runner is observable without a permission the reaper's token may not have. The API's run and job lists are the fallback, and need the token's Actions read permission.
Never act on queue age alone: a job waiting behind long builds looks identical from the outside, and cancelling it re-queues it behind the same builds. Never touch a run with any job in progress; log every action.
Capture first: the listener log around the queue time, the EphemeralRunner list and the scale set's desired and current counts, the next time a job stalls with capacity free, before any recovery cancels the evidence. If the listener or controller is at fault, fix that at its source and close this as not needed.
Activity