Skip to content

Lower the runner pod memory request once job peaks are measured #46

Open

Description

@Mearman

Runner pods are sized at 2 CPU and 7 Gi regardless of the job, which limits how many run at once on nodes with about 8 Gi allocatable, and observed use is a fraction of that. The measured peaks are now collected (the autoscaler's job-peak key) and summarised in the heartbeat gist's job-peak-summary.json: job count, median, 95th percentile and maximum in MiB next to the pod's memory limit.

Deferred because it needs real CI traffic: the measurement only began recently and the figures so far come from a handful of jobs. Revisit when the summary's job count covers a representative spread of the repositories that use the fleet (a week of ordinary traffic is a reasonable minimum) and the 95th percentile and maximum are stable between readings.

Then decide from the data, not a formula: if the maximum sits comfortably below the limit, lower the request in the fleet's shared runner values file and watch for out-of-memory kills; if a few jobs sit near the limit while most sit far below, that is the case for size tiers (see issue 40), not for a lower default. A lower request also raises how many runners fit, so the autoscaler's ceiling and the static maxRunners in the values file need revisiting with it.

Related: #40

A one-time scheduled run (https://claude.ai/code/routines/trig_01Q4AQebKvmVHtCMSbyUVXiq) posts the gist figures on this issue on 2026-10-10 at 16:00 UTC. It posts figures only.

Activity

  1. added theissue type on Oct 3, 2026
  2. Mearman commented on Oct 4, 2026

    @Mearman
    MemberAuthor

    Status note, first reading (2026-10-04, about nine hours after measurement began): 79 jobs, median 1949 MiB, 95th percentile 3248 MiB, maximum 3340 MiB, against the 7168 MiB limit. Not enough to act on: the jobs come from three repositories, and the eight heaviest are all one repository's type-aware lint job, so the sample is dominated by a single busy day of PRs. The trigger in the body (a representative spread, about a week) is not met. The heartbeat's job-peak-summary.json will gain the number of distinct repositories and the period covered, so that trigger can be checked from the gist alone.

  3. Mearman commented on Oct 10, 2026

    @Mearman
    MemberAuthor

    Scheduled reading, 2026-10-10: 181 jobs from 5 repositories between 2026-10-09T21:07:53Z and 2026-10-10T15:59:02Z (0.8 days). Median 2807 MiB, 95th percentile 7168 MiB, maximum 7169 MiB, against the pod limit of 7168 MiB. Fleet health at the time: healthy (true). This is a reading only: the Revisit trigger field on this issue says whether it is enough to act on.


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions