ops(runners): periodic and low-disk-triggered cleaning for self-hosted runners

[goastler]

The self-hosted fleet is non-ephemeral and nothing ever reclaims its disk. Every job leaves build state behind — a single Protect checkout is ~17 GB of target/, over half of it incremental state that is worthless once the job ends — so a 60 GB runner fills after a handful of branches and then fails jobs late, with a linker error (ar: … No space left on device) that looks like anything but a disk problem. That persistence is also exactly why save_cargo_cache skips the GitHub cache backend on self-hosted: local disk is the cache there, and nobody prunes it.

Adds runner-maintenance/:

• runner-disk-clean.sh — the sweep. Incremental dirs, cold target/ trees, cold workspace checkouts, ~/.cargo/registry/src (regenerated from the .crate archives, which are kept), cold node_modules, and dangling docker layers.
• runner-disk-clean.{service,timer} — nightly at 03:30 with a 30m jitter, Persistent=true, Nice=19 + idle IO so it never competes with a running build. Runs as the runner's user, not root.
• job-started-hook.sh — for ACTIONS_RUNNER_HOOK_JOB_STARTED. No-ops unless free space is under MIN_FREE_GB (default 15), then sweeps with a 1-day cut instead of 7. Checking before the job means the reclaim happens before the build burns its full time.

Everything is age-based, never a wipe. Jobs run back to back on these hosts and share ~/.cargo, so state a running job might hold has to survive; the age cut (7 days scheduled, 1 day under pressure) is far longer than any single job. The hook also never fails a job that is merely short on space — a build that would have squeaked through shouldn't be pre-emptively killed; the warning line is the signal that the fleet needs a bigger disk.

Deliberately not reusing the per-ecosystem actions from #4 here — those delete shared preinstalled toolchains, which on a persistent runner is permanent and breaks every later job.

Nothing has been installed on any runner; this is the artefact only. Deploying it needs a decision on where runner config should live (there is no ansible inventory covering the fleet today).

Related: #5, Protect#450.

[goastler]

Superseded by https://github.com/prosopo/gh-runner/pull/17, which does this from the two job hooks that already exist in the gh_runner_native ansible role.

Three things changed rather than being ported:

• No MIN_FREE_GB. The target is the largest disk label the host advertises — that label is a promise GitHub schedules against, so it is exactly the amount that has to be free.
• No timer. Disk only fills because jobs run, so a job starting or ending is the whole set of moments worth checking, and a hook knows whether a build is in flight where a timer has to guess.
• No mtime sweep. The runner executes one job at a time, so every workspace but the current one is idle by definition. mtime was wrong in the dangerous direction anyway: a long cargo build writes into target/debug/deps/ and never touches the workspace directory above it.

Also worth recording why the location moved: this repo holds composite actions, so the runner's user, work dir and cargo home all had to be restated here as defaults, and all three were wrong for the fleet — actions-runner vs ghrunner, $RUNNER_HOME/_work vs /var/lib/actions-runner/_work, $RUNNER_HOME/.cargo vs /opt/rust. df on a work dir that does not exist returns nothing, which read as 0 GB free, so the unit would have looked installed and healthy while sweeping paths that were not there. In the ansible role those values are defined once.