You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Lower the runner pod memory request once job peaks are measured #46
Runner pods are sized at 2 CPU and 7 Gi regardless of the job, which limits how many run at once on nodes with about 8 Gi allocatable, and observed use is a fraction of that. The measured peaks are now collected (the autoscaler's job-peak key) and summarised in the heartbeat gist's job-peak-summary.json: job count, median, 95th percentile and maximum in MiB next to the pod's memory limit.
Deferred because it needs real CI traffic: the measurement only began recently and the figures so far come from a handful of jobs. Revisit when the summary's job count covers a representative spread of the repositories that use the fleet (a week of ordinary traffic is a reasonable minimum) and the 95th percentile and maximum are stable between readings.
Then decide from the data, not a formula: if the maximum sits comfortably below the limit, lower the request in the fleet's shared runner values file and watch for out-of-memory kills; if a few jobs sit near the limit while most sit far below, that is the case for size tiers (see issue 40), not for a lower default. A lower request also raises how many runners fit, so the autoscaler's ceiling and the static maxRunners in the values file need revisiting with it.
Status note, first reading (2026-10-04, about nine hours after measurement began): 79 jobs, median 1949 MiB, 95th percentile 3248 MiB, maximum 3340 MiB, against the 7168 MiB limit. Not enough to act on: the jobs come from three repositories, and the eight heaviest are all one repository's type-aware lint job, so the sample is dominated by a single busy day of PRs. The trigger in the body (a representative spread, about a week) is not met. The heartbeat's job-peak-summary.json will gain the number of distinct repositories and the period covered, so that trigger can be checked from the gist alone.
Scheduled reading, 2026-10-10: 181 jobs from 5 repositories between 2026-10-09T21:07:53Z and 2026-10-10T15:59:02Z (0.8 days). Median 2807 MiB, 95th percentile 7168 MiB, maximum 7169 MiB, against the pod limit of 7168 MiB. Fleet health at the time: healthy (true). This is a reading only: the Revisit trigger field on this issue says whether it is enough to act on.
Activity
Mearman commentedon Oct 4, 2026
Status note, first reading (2026-10-04, about nine hours after measurement began): 79 jobs, median 1949 MiB, 95th percentile 3248 MiB, maximum 3340 MiB, against the 7168 MiB limit. Not enough to act on: the jobs come from three repositories, and the eight heaviest are all one repository's type-aware lint job, so the sample is dominated by a single busy day of PRs. The trigger in the body (a representative spread, about a week) is not met. The heartbeat's job-peak-summary.json will gain the number of distinct repositories and the period covered, so that trigger can be checked from the gist alone.
Mearman commentedon Oct 10, 2026
Scheduled reading, 2026-10-10: 181 jobs from 5 repositories between 2026-10-09T21:07:53Z and 2026-10-10T15:59:02Z (0.8 days). Median 2807 MiB, 95th percentile 7168 MiB, maximum 7169 MiB, against the pod limit of 7168 MiB. Fleet health at the time: healthy (true). This is a reading only: the Revisit trigger field on this issue says whether it is enough to act on.
Generated by Claude Code