Skip to content

fix(e2e): retain-on-failure traces + a globalTimeout under the 45m CI cap - #461

Merged
rubenvdlinde merged 2 commits into
developmentfrom
fix/playwright-trace-and-globaltimeout
Aug 8, 2026
Merged

fix(e2e): retain-on-failure traces + a globalTimeout under the 45m CI cap#461
rubenvdlinde merged 2 commits into
developmentfrom
fix/playwright-trace-and-globaltimeout

Conversation

@rubenvdlinde

Copy link
Copy Markdown
Contributor

Part of the fleet-wide Playwright instrument sweep for ConductionNL/.github#188. Neither change can alter a verdict — both change whether you can see why a verdict happened.

trace

trace: 'on-first-retry' only writes a trace when a retry actually happens, which makes the trace artifact a function of retries. retain-on-failure captures every test, keeps only the failures, and does not depend on the retry count.

Reproduced with two minimal Playwright projects differing only in the trace value, same deliberately failing test, retries: 0 in both:

config trace zips written
retries: 0 + on-first-retry 0
retries: 0 + retain-on-failure 1 (7.4 KB)

globalTimeout: 38 * 60_000

The shared quality.yml Playwright job is timeout-minutes: 45. A job cancelled by that cap yields no verdict and no artifacts — the trace upload is if: failure(), the report upload is if: always(), and neither runs on a cancelled job, while gh pr checks still renders it as "fail". Runs cancelled at ~45m16s have been observed in this fleet.

Margin, measured rather than assumed: overhead before Run Playwright tests starts is 2.0-2.4 min (openconnector run 31257480415 = 2m20s; opencatalogi, doriath, openregister in the same band); uploads after it take seconds. 38m + ~2.5m setup + uploads sits ~7 min under the cap. Verified that a fired globalTimeout exits with a tally (3 did not run / 1 passed) plus an HTML report and a trace zip on disk — exactly what a cancelled job does not give you.

Matches the value already landed in nldesign.

… cap

Fleet-wide Playwright instrument sweep, ConductionNL/.github#188. Neither
change can alter a verdict; both change whether you can see why a verdict
happened.

`trace: 'on-first-retry'` only writes a trace when a retry actually happens, which makes the trace artifact a function of `retries`. `retain-on-failure` captures every test, keeps only the failures, and does not depend on the retry count.

No repo in the fleet set `globalTimeout`. The shared quality.yml Playwright
job is `timeout-minutes: 45`, and a job cancelled by that cap produces no
verdict and no artifacts: the trace upload is `if: failure()` and the report
upload is `if: always()`, and neither runs on a cancelled job, while
`gh pr checks` still renders it as "fail". Runs cancelled at ~45m16s have
been observed in this fleet. Measured overhead in that job before the
`Run Playwright tests` step starts is 2.0-2.4 min, so 38m leaves ~7 min of
margin while guaranteeing a tally and its artifacts.
The shared quality.yml resolves its config as
`${playwright-test-path}/playwright.config.ts` and only falls back to the
app-root `playwright.config.ts` when that file is absent (quality.yml
~L2218). This repo ships tests/e2e/playwright.config.ts, so THAT is the
file every CI run has been using — the app-root config fixed in the previous
commit is the one developers load by hand, not the one the gate reads.

Applies the identical `retain-on-failure` + `globalTimeout: 38 * 60_000`
change here. ConductionNL/.github#188.
@rubenvdlinde

Copy link
Copy Markdown
Contributor Author

Second commit added: this repo ships a nested Playwright config, and that is the one CI actually loads.

quality.yml (~L2218) resolves CONFIG="${playwright-test-path}/playwright.config.ts" and only falls back to the app-root playwright.config.ts when that file does not exist. Measured across the fleet: 16 of 23 repos with a Playwright config have such a nested file, so the app-root config the original issue tabulated is not the one the gate reads in any of them.

Both files are now fixed — the app-root one (what a developer gets from npx playwright test at the repo root) and the nested one (what CI runs).

@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Quality Report — ConductionNL/softwarecatalog @ 8fec479

Check PHP Vue Security License Tests
lint
phpcs
phpmd
psalm
phpstan
phpmetrics
eslint
stylelint
build
check-manifest
check-vue-demi
test-l10n
composer ✅ 128/128
npm ✅ 718/718
PHPUnit
Newman ⏭️
Playwright
Hydra gates

Quality workflow — 2026-08-08 13:43 UTC

Download the full PDF report from the workflow artifacts.

@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Quality Report — ConductionNL/softwarecatalog @ 3d013f5

Check PHP Vue Security License Tests
lint
phpcs
phpmd
psalm
phpstan
phpmetrics
eslint
stylelint
build
check-manifest
check-vue-demi
test-l10n
composer ✅ 128/128
npm ✅ 718/718
PHPUnit
Newman ⏭️
Playwright
Hydra gates

Quality workflow — 2026-08-08 14:02 UTC

Download the full PDF report from the workflow artifacts.

@rubenvdlinde
rubenvdlinde merged commit d826781 into development Aug 8, 2026
29 checks passed
@rubenvdlinde
rubenvdlinde deleted the fix/playwright-trace-and-globaltimeout branch August 8, 2026 14:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant