Skip to content

The end-to-end CI shards fail when a runner is slow to install the browser or ffmpeg #838

Closed

Description

@thcp

What happens

An end-to-end shard fails before a single test runs when its hosted runner is slow to install the browser's system packages or ffmpeg. The tests are fine; the install steps are not.

The first e2e-full run on big-release (run 39827053220, commit 6c3cf58) shows it. Shard 2 finished in 7 minutes. Shards 1 and 3 were cancelled by the 30 minute job limit:

Shard npx playwright install --with-deps chromium apt-get install ffmpeg
1 12m23s 16m28s, killed by the job limit
3 11m38s 17m41s, killed by the job limit
2 22s 11s

The job log shows where the time went: apt fetched 20.9 MB from azure.archive.ubuntu.com at 29.2 kB/s ("Fetched 20.9 MB in 11min 56s"), with several Ign: lines where apt gave up on a package and fetched it again. The Chromium download itself, from Playwright's CDN, took 9 seconds. The runner image was ubuntu-24.04, version 20261012.334.1.

The same stall hit other runs the same week:

Every job also carries a notice that the ubuntu-latest label moves to Ubuntu 26 from October 19, 2026 (actions/runner-images#14748), so the environment under the suite can change between two runs of the same commit.

Who it affects

Anyone with a pull request open, and the full-suite run on every merge to big-release. A cancelled shard reads as a red PR, needs a manual re-run, and hides whether the change itself is sound.

What should happen

  • A slow or hung install fails within minutes, says which step and why, and is retried, instead of burning the whole 30 minutes.
  • Most runs do not depend on the Ubuntu mirror at all for things that rarely change.
  • The environment under the suite does not change because a runner label moved.
  • When a stall does happen, the run's summary shows which setup step took the time, without reading logs.

Constraints

  • The e2e backend needs an ffmpeg with libmp3lame, aac, flac and libvorbis; the ogg export specs need Vorbis.
  • No unverified binary may be downloaded or run: a download is pinned by checksum.
  • Job names frontend-e2e (1) to (3), the spec selector and the scoped and full cadence stay as they are.
  • No change to specs, app code, uv.lock or package files.
  • Hosted runners only; the self-hosted runners are not involved here.

Activity

  1. added 2 commits that reference this issue on Oct 22, 2026
    ca231fb
    bcce70e
  2. thcp commented on Oct 22, 2026

    @thcp
    CollaboratorAuthor

    Fixed: the e2e jobs are pinned to ubuntu-24.04, and each install step has a 10 minute limit and runs through a retry wrapper (3 attempts, a timeout per attempt, every attempt logged). A retry drops the slow Azure package mirror, because apt only switches mirrors on an error, never on a slow download. The Playwright browsers, their system packages and a checksum-pinned static ffmpeg (with libvorbis) are cached, and each job summary lists the seconds each setup step took and whether the cache hit. The full run on big-release passed on all three shards after this change. On big-release (#839).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions