fix(tests): isolate process.env to stop flaky auth/pairs failures - #160
therealbibson wants to merge 2 commits into
Conversation
Root cause: shared process.env across reused vitest workers (and request-time ADMIN_API_KEY reads after pairs.buildApp restored/deleted it). Snapshot/restore env per test; keep ADMIN_API_KEY for request life; raise testTimeout under parallel import load. Closes Miracle656#151
|
@therealbibson Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits. You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀 |
|
Flagging rather than closing, since it is yours — but I have asked for #161 instead, and the two cannot both land: they conflict with each other (both add The deciding factor is when That is also why this PR had to convert This PR is the better one in every other respect, and I have asked #161 to take three things from it: the 20s Worth knowing: the flake is real. I reproduced it three times, always in a full run, always passing in isolation, with rotating victims. An earlier check of mine got seven green runs and wrongly called it fixed — that was luck. Neither PR reaches the root hazard, though: |
…eilings Review follow-up on Miracle656#151. - vitest.setup.ts: the header blamed Vitest's `threads` pool running files concurrently in one process. Vitest 4's default pool is already `forks`, so files never share a process; what actually leaks is `process.env` *within* a reused fork, which runs several files sequentially. Comment corrected. - vitest.config.ts: `pool: 'forks'` pins the existing default rather than changing it, and the comment now says so. `testTimeout` 60_000 -> 20_000 so a genuine hang fails the run instead of passing slowly. - tests/aggregator.property.test.ts: property timeout 120_000 -> 30_000. - src/__tests__/usage.test.ts, src/__tests__/x402-quota.test.ts: adopt the per-file env restores from Miracle656#160 (additive, compatible with the shared setup).
) * fix(test): isolate process.env to stop flaky auth/pairs suite (#151) Root cause: Vitest threads pool shares one process across parallel files, so process.env mutations (ADMIN_API_KEY, ADMIN_TOKEN, venue flags, etc.) race mid-assertion. Switch to forks, enforce env snapshot/restore setup, and keep ADMIN_API_KEY set through pairs inject. * test(env): correct the stated root cause and tighten the regression ceilings Review follow-up on #151. - vitest.setup.ts: the header blamed Vitest's `threads` pool running files concurrently in one process. Vitest 4's default pool is already `forks`, so files never share a process; what actually leaks is `process.env` *within* a reused fork, which runs several files sequentially. Comment corrected. - vitest.config.ts: `pool: 'forks'` pins the existing default rather than changing it, and the comment now says so. `testTimeout` 60_000 -> 20_000 so a genuine hang fails the run instead of passing slowly. - tests/aggregator.property.test.ts: property timeout 120_000 -> 30_000. - src/__tests__/usage.test.ts, src/__tests__/x402-quota.test.ts: adopt the per-file env restores from #160 (additive, compatible with the shared setup).
Miracle656
left a comment
There was a problem hiding this comment.
This one lands awkwardly and I want to be straight about why: #161 shipped as 8b66b44 and it cherry-picked most of this PR. Your diagnosis was right and it's now on main — but the credit landed in another PR's commit, and that's on us, not you.
Here is the file-by-file comparison against current origin/main:
| Your change | Status on main |
|---|---|
src/__tests__/usage.test.ts — ORIGINAL_ADMIN_TOKEN + afterEach delete/restore |
already there, identical |
src/__tests__/x402-quota.test.ts — ENV_KEYS snapshot/restore |
already there, identical |
src/__tests__/auth.test.ts — delete instead of assigning undefined |
already there; your diff is now a comment reword only |
src/__tests__/pairs.test.ts — drop the restore in buildApp, add afterEach delete |
already there, same rationale comment |
vitest.config.ts — setupFiles + testTimeout: 20_000 |
already there |
vitest.setup.ts — env snapshot/restore |
already there, different snapshot point (below) |
src/__tests__/pairIssuerMatch.test.ts — beforeAll → beforeEach |
only genuinely new line |
So the unique remainder is one line, and the two differences that are left both go the wrong way:
1. vitest.setup.ts — snapshot timing. You snapshot once at module load (const ENV_SNAPSHOT = {...process.env}); main snapshots in a beforeEach. main's version is the more robust of the two, because a global beforeEach registered from setupFiles runs after a file's beforeAll, so env a file sets once for the whole file is captured and survives. With a load-time snapshot it gets wiped after the first test — which is exactly why you then had to convert pairIssuerMatch's beforeAll to beforeEach. main doesn't need that conversion at all. Your own comment ("suites that need a sticky env for the whole file should set it in beforeEach, not only beforeAll") is the constraint this choice imposes, and it's avoidable.
2. vitest.config.ts — pool: 'forks'. You remove it. main pins it deliberately (it's already Vitest's default; pinning stops a future default change from silently reintroducing cross-file sharing). Relatedly, your comment says "with --pool=threads files can even share one process concurrently" — that reads as the diagnosis, but Vitest 4's default is forks, so the leak that was actually biting was sequential reuse of one fork across files, not concurrent sharing. main's comment says that.
3. pairIssuerMatch.test.ts:18. This is the only new line, and it doesn't do what it looks like. src/config.ts:245-253 memoises each NetworkConfig in a module-level Map, so once test 1 calls getNetworkConfig('testnet') the result is cached and later tests don't read process.env at all — only vi.resetModules() invalidates it. That's why all 5 tests pass on main with beforeAll (5 passed, verified locally both standalone and in the full run). Converting to beforeEach makes vi.resetModules() + await import('../config') run five times instead of once, inside a hook — and hooks still use the 10s hookTimeout default, because #161 raised testTimeout only. I actually caught that beforeAll hook failing on one full local run of main, so this makes the exposure worse, not better.
My read: no unique value left, and the two remaining deltas are small regressions. I'd rather not merge it in this shape.
If you want something real to land from this, there is a genuine open bug sitting right next to your work and you're the best-placed person to fix it: hookTimeout was never raised alongside testTimeout, so every beforeAll/beforeEach that does resetModules() + a cold module import is still on a 10s budget. A two-line vitest.config.ts change (hookTimeout: 20_000) plus a note explaining why hooks need the same headroom as tests would be a clean, correct PR, and I'd merge it. Repoint this branch at that if you're willing.
Verification I ran: origin/main baseline 538 passed, 1 skipped (56 files), tsc --noEmit clean; npx vitest run src/__tests__/pairIssuerMatch.test.ts on main → 5 passed; git diff of this branch's merge-base diff against each file on main.
Sorry this is the second of two in a row — the work was real and correct, main just moved under it.
…env-isolation Resolves the merge conflicts with main.
|
@Miracle656 I've resolved the merge conflicts with All other changes from Merge commit: Could you take another look when you have a moment? Thanks! |
|
| GitGuardian id | GitGuardian status | Secret | Commit | Filename | |
|---|---|---|---|---|---|
| 37768935 | Triggered | Generic High Entropy Secret | 67f8b41 | src/tests/toid.test.ts | View secret |
🛠 Guidelines to remediate hardcoded secrets
- Understand the implications of revoking this secret by investigating where it is used in your code.
- Replace and store your secret safely. Learn here the best practices.
- Revoke and rotate this secret.
- If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.
To avoid such incidents in the future consider
- following these best practices for managing and storing secrets including API keys and other credentials
- install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.
🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.
Overview
Fixes intermittent vitest failures (notably
auth.test.tsandpairs.test.ts, ~1 in 5 full runs) caused by sharedprocess.envacross reused workers / parallel files, plus a pairs-test anti-pattern that clearedADMIN_API_KEYbefore request-time auth checks.Related Issue
Closes #151
Changes
Root cause
process.envacross the suite. Vitest reuses worker processes between files. Suites such asx402-quota,usage,pairIssuerMatch, andauthmutate env vars (REQUIRE_API_KEY,ADMIN_TOKEN,WATCHED_PAIRS_*,STELLAR_NETWORK, …) and either never restore them or restore incorrectly (process.env.X = undefinedbecomes the string"undefined"). That cross-file state is why failures vanish in isolation and rotate between victims.pairs.test.tsrestored/deletedADMIN_API_KEYinsidebuildApp()afterapp.ready().routes/pairs.tsreadsprocess.env.ADMIN_API_KEYat request time, not registration time — so a race/clear left the happy-path POST returning 401 instead of 201.testTimeoutunder parallel import load. Cold Fastify boots andvi.resetModules()+ config re-imports occasionally exceeded 5s when many files transformed in parallel (same symptom class as the issue: full run flakes, isolation passes). Fixed by raising timeout without disablingfileParallelism.Fix
vitest.setup.ts— snapshotprocess.envonce per worker; restore after every test.vitest.config.ts— register setup file; settestTimeout: 20_000; keep file parallelism enabled.src/__tests__/pairs.test.ts— keepADMIN_API_KEYfor the request lifetime; restore inafterEach.src/__tests__/auth.test.ts— deleteADMIN_TOKENon restore when originally unset (avoid"undefined"string).src/__tests__/pairIssuerMatch.test.ts— move env + config import intobeforeEachso global restore cannot wipe stickybeforeAllstate.src/__tests__/x402-quota.test.ts/usage.test.ts— restore mutated env keys after each test.Verification Results
process.envvia worker reuse + pairsADMIN_API_KEYcleared before request-time read; timeout contention under parallel importsfileParallelismleft on; isolation via env snapshot/restoreprocess.envrestore itCloses #151