ci(test): fix the 6 flaky unit failures on GitHub-hosted runners - #1
Merged
Merged
Conversation
Windows (5x Ripgrep > ...): windows-latest ships no rg, so RipgrepBinary
downloaded and Expand-Archive'd ripgrep on first use, which alone blew the
5s per-test timeout and SIGTERM'd every Ripgrep test behind it. Install
ripgrep via choco before the suite so which("rg.exe") short-circuits the
download. Linux was unaffected: tar extraction fits inside the budget.
Linux (1x run-process "exits nonzero promptly when the model is unknown"):
the 15s wall-clock bound was tighter than the harness default and was
being consumed by cold `bun src/index.ts` startup under concurrent
siblings on ubuntu-latest, not by the anomalyco#27371 hang it guards against.
Use the harness default 30s bound (outer 60s, matching sibling tests);
a real hang still fails via the harness kill + duration assertion.
Backlog a4ef43f9.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
This PR doesn't fully meet our contributing guidelines and PR template. What needs to be fixed:
Please edit this PR description to address the above within 2 hours, or it will be automatically closed. If you believe this was flagged incorrectly, please let a maintainer know. |
|
Hey! Your PR title Please update it to start with one of:
Where See CONTRIBUTING.md for details. |
|
The following comment was made by an LLM, it may be inaccurate: |
…tep 60 min First CI pass on this branch moved the failures rather than removing them: Linux: the regression test now passes, but 4 sibling run-process tests hit the 30s bound instead. run-process.test.ts runs 11 CLI spawns concurrently (bun's default cap is 20) and each is a cold `bun run src/index.ts` - CPU bound, not I/O bound - so on a 4-vCPU ubuntu-latest they starve each other and which ones lose is a lottery. --max-concurrency 4 on packages/opencode's test script (the only package with CLI-subprocess tests; a turbo `--` passthrough is unsafe because app's script is compound) matches the width to the cores. Windows: core:test now passes all 1099 (the ripgrep install worked), which unmasked opencode:test - turbo had never reached it here because it stops at the first failing package. The harness runs its CLI tests serially on win32 by design, windows-latest spawns slowly, and the package alone blew the 20-minute step timeout. 60 minutes for the Windows matrix leg only. Backlog a4ef43f9. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
e2e/regression/session-todo-dock-navigation failed 3/3 attempts on GitHub-hosted ubuntu-latest at line 92 (`openSamples.length > 0`). The "returning to the source session" step opened a fixed 700ms rAF sampling window and then navigated back; when navigation + first paint took longer than 700ms the window closed before the dock existed, so there were zero present samples. The assertion is about the FIRST present frame being already fully open (no replayed animation), which needs "sample until present", not a clock. sampleDock gains an `untilPresent` option that stops five frames after the dock is first seen, capped at 10s; the other three call sites are unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
With core:test and the run-process lottery fixed, opencode:test completes on windows-latest (21 min, 3652 tests) with exactly 2 failures, both deterministic: external-directory "normalizes Windows path variants to one glob" and read "normalizes read permission paths on Windows". Each strips the drive letter from a temp path and expects it to resolve back to the same file - an assumption that temp and the checkout share a drive. On windows-latest the checkout is D:\a\opencode and %TEMP% is C:\Users\..., so the drive-less path resolves against the current drive to D:\users\runneradmin\appdata\... and the file is not there. Blacksmith kept both on C:. Point TMP/TEMP at runner.temp (D:\a\_temp) for the Windows leg so the tests' premise holds; Linux is untouched (empty value, reads TMPDIR). Backlog a4ef43f9. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The runner.temp override (f1758f3) traded two failures for five. The Windows path tests strip the drive letter and expect the file to resolve back; that needs temp on the checkout's drive (drive-less paths resolve against the current drive), which runner.temp gave - but the normalizer also reads Git-Bash style /x/... as drive X:, so /a/_temp from D:\a\_temp became A:\_temp and three shell-permission tests joined the two originals. Temp must be on D: and NOT under D:\a. A small step derives the drive from GITHUB_WORKSPACE, creates <drive>:\tmp, and exports TMP/TEMP via GITHUB_ENV; Linux is untouched. Backlog a4ef43f9. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…not C: With temp on the checkout's drive (e893afe) the Windows leg is down to these two: "asks for external_directory permission for drive-relative PowerShell paths [pwsh|powershell]". They hardcode `C:../outside.txt`, a path relative to C:'s current directory, and expect it to resolve against the test cwd - true only when the cwd is on C:. The other five Windows path tests need temp on the checkout's drive, which on windows-latest is D:, so the two constraints cannot both hold with a literal C:. Take the drive letter from tmp; the test still exercises drive-relative resolution. Backlog a4ef43f9. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Result
Run 34749489714 on
e3f2367e: unit (linux) ✅ · unit (windows) ✅ · e2e (linux) ✅ · e2e (windows) ❌ (Playwright flake, see below). First green unit matrix on the fork since the Blacksmith→GitHub-hosted runner sweep. Linux unit green 4 runs running; Windows unit went 7 → 5 → 2 → 0 failures over the iterations.Why
After the runner sweep,
testondevfailed on every push. None of it was random — every failure had a runner-shaped mechanism.What (6 commits)
Ripgrep > …—windows-latesthas norg; first-use download +Expand-Archiveblew the 5 s per-test budget.choco install ripgrepbefore the suite sowhich("rg.exe")short-circuits. Linux, 1× run-process regression test — 15 s bound eaten by cold CLI spawn under concurrent siblings; harness default 30 s. (d9b5972)--max-concurrency 4onpackages/opencodeonly. Windowsopencode:testhad never run here (turbo stops at the first failing package); it's serial on win32 by harness design and takes ~20 min, so the Windows step gets 60. (21915a4)session-todo-dock-navigationfailed 3/3 on Linux — fixed 700 ms sampling window raced the navigation back.sampleDockgainsuntilPresent. Green every run since. (22e9326)%TEMP%is C:) and not under a single-letter dir (/a/_tempparses asA:). Temp atD:\tmpviaGITHUB_ENV, Windows only. (f1758f3,e893afe) The last two hardcodedC:../outside.txt; they now use the cwd's drive — still drive-relative, which is the point of the test. (e3f2367)Not fixed — your call
The app e2e suite is timing-marginal on GitHub-hosted runners:
e2e (windows)failed 3 of 6 runs on twotoBeVisibletimeouts (open-file-expansion … trailing Windows separator,file-browser … switching file tabs);e2e (linux)failed once onreview-line-commenthover with 4 others flaky-then-passed. Options: accept and re-run; raise Playwrightretriesfor CI inpackages/app; or stop gating the fork on e2e. I'd take the retries bump, but it's a posture choice.Backlog
a4ef43f9.🤖 Generated with Claude Code