Repository navigation
Runners are lost mid-job during heavy builds #105
Description
Jobs routed to the self-hosted runners intermittently end with "The self-hosted runner lost communication with the server" part-way through a build, after the job has started and restored its caches. Seen on several runs of one private repository's rust job within a few hours (workflow runs 39417093677, 39416260510 first attempt, 39421236914), on both architectures; other jobs on the same runners passed in between, and a rerun of the same commit passes.
The jobs are CPU and memory heavy (a full Rust workspace build and test, several minutes of compiling), so a runner pod being evicted, OOM-killed or starved of CPU while the listener process misses its heartbeat is the first thing to rule out. The runner pods' own logs and the nodes' events for those times would say which; the workflow side has nothing more to show than the annotation.
Asking for: the cause for those three runs from the pod and node logs, and, if it is resource pressure, requests and limits on the runner pods that fit a Rust build (or a documented size the workflows should assume), so a lost runner is not mistaken for a failing test.
Activity
Mearman commentedon Oct 20, 2026
Another one on 2026-10-20: a
rustjob on a private repository's pull request ran on a fleet runner for about 16 minutes and was marked failed with "The self-hosted runner lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error." It was lost during the formatting and clippy step (the steps after the toolchain have no conclusion) and its log blob was never uploaded. The rerun on another runner passed every step.Mearman commentedon Oct 30, 2026
Same loss on a light job, so it isn't only heavy builds. On 2026-10-30, https://github.com/ExaDev/combinatrix/actions/runs/41064547260 lost Lint, Typecheck and Package (Node 24) on the first attempt and again on the rerun, each on a different runner than before. None got past the first steps: Lint and Typecheck sat in
actions/setup-nodeor "Set up job" until the 15 minute cutoff (the jobs settimeout-minutes: 10, which never fired), and Package (Node 24) stopped at its secondsetup-nodeafter about 11 minutes. Package (Node 22), Browser, Commitlint, Actionlint and Zizmor in the same run passed. No project code had run in any of the failed jobs.