test: stabilize CPU simulation with isolated SSR workers and explicit V8 settings - #8248
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
View your CI Pipeline Execution ↗ for commit 6741406
☁️ Nx Cloud last updated this comment at |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Team Run ID: 📒 Files selected for processing (4)
💤 Files with no reviewable changes (1)
🚧 Files skipped from review as they are similar to previous changes (1)
Included review availability: Your plan provides up to 10 included reviews per hour; 7 remain after this review. 📝 WalkthroughWalkthroughThe benchmark suite now applies shared CPU-simulation flags, isolates SSR benchmark workers, splits SSR scenarios into separate files, widens benchmark TypeScript includes, updates streaming scheduling, and revises benchmark runtime documentation. ChangesBenchmark runtime and workload coverage
Estimated code review effort: 4 (Complex) | ~45 minutes Merge Risk: ⚪ Minimal · up to This change updates benchmark execution isolation, CPU-simulation settings, fixture scheduling, and benchmark cache configuration without changing production router behavior. The supplied validation and benchmark results indicate no remaining merge-blocking risk. Suggested reviewers: Sequence Diagram(s)sequenceDiagram
participant Vitest
participant cpuSimulationExecArgv
participant SSRBenchmark
participant BuiltServerHandler
Vitest->>cpuSimulationExecArgv: Resolve execution arguments
cpuSimulationExecArgv-->>Vitest: Return runtime flags
Vitest->>SSRBenchmark: Start isolated speed file
SSRBenchmark->>BuiltServerHandler: Load and validate handler
SSRBenchmark->>BuiltServerHandler: Run scenario request loop
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Full details: Docstring CoverageExplanation Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 3 functions across 51 files. (3 skipped: 3 unsupported.)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
🚀 Changeset Version Preview5 package(s) bumped directly, 18 bumped as dependents. 🟩 Patch bumps
|
Bundle Size BenchmarksThis pull request does not affect bundle size in any measured scenario. |
Merging this PR will degrade performance by 26.53%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | Simulation | ssr global-mw document (solid) |
352.4 ms | 282.6 ms | +24.69% |
| ⚡ | Simulation | ssr global-mw document (react) |
298.3 ms | 269.3 ms | +10.76% |
| ⚡ | Memory | mem client mount-unmount (solid) |
513.7 KB | 483.3 KB | +6.3% |
| ⚡ | Memory | mem server error-paths redirect (react) |
317.8 KB | 302.1 KB | +5.19% |
| ⚡ | Simulation | client-mount loop (react) |
73.8 ms | 70.4 ms | +4.88% |
| ⚡ | Memory | mem server error-paths not-found (solid) |
577.2 KB | 554.5 KB | +4.08% |
| 🆕 | Simulation | ssr assets linked-css control (solid) |
N/A | 208.2 ms | N/A |
| 🆕 | Simulation | ssr control-flow error 500 (solid) |
N/A | 211 ms | N/A |
| 🆕 | Simulation | ssr control-flow route headers (solid) |
N/A | 288.6 ms | N/A |
| 🆕 | Simulation | ssr control-flow unmatched 404 (solid) |
N/A | 212.8 ms | N/A |
| 🆕 | Simulation | ssr not-found (solid) |
N/A | 179.9 ms | N/A |
| 🆕 | Simulation | ssr global-mw server-fn (solid) |
N/A | 232.1 ms | N/A |
| 🆕 | Simulation | ssr global-mw server-route (solid) |
N/A | 205.6 ms | N/A |
| 🆕 | Simulation | ssr rewrite passthrough (solid) |
N/A | 263.4 ms | N/A |
| 🆕 | Simulation | ssr dehydrate plain control (solid) |
N/A | 223.7 ms | N/A |
| 🆕 | Simulation | ssr server-fn raw-response (solid) |
N/A | 196.7 ms | N/A |
| 🆕 | Simulation | ssr server-fn raw-stream (solid) |
N/A | 731.1 ms | N/A |
| 🆕 | Simulation | ssr server-fn during document ssr (solid) |
N/A | 218.6 ms | N/A |
| 🆕 | Simulation | ssr server-fn not-found (solid) |
N/A | 254.1 ms | N/A |
| 🆕 | Simulation | ssr server-fn POST (solid) |
N/A | 270.4 ms | N/A |
| ... | ... | ... | ... | ... | ... |
ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.
Tip
Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.
Comparing codex/stabilize-cpu-simulation (6741406) with main (28a5e45)
Footnotes
-
48 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
🎯 Changes
CPU simulation scenario benchmarks produced multi-percent changes when rerunning identical code. This PR gives each SSR workload a fresh worker, makes Node 24 compiler/GC settings explicit, and replaces the streaming fixture's elapsed-time delay with counted task turns.
The final configuration adds four flags, down from five in the initial proposal. Post-isolation experiments allow removal of
--no-incremental-marking-task. Eager baseline compilation and the 512 MiB heap budget remain: none of the tested default-heap configurations met the sub-1% repeatability target. The retained heap budget changes GC policy; repeatability does not establish representative production full-GC costs.This builds on #7987. Production router code, workload names, navigation/request counts, concurrency, payload sizes and coverage are preserved. All measurements below are CPU simulation; memory results are excluded.
Observed impact
The selected configuration completed five independent Linux CI runs at one commit, each with all 132 fresh CPU measurements (660 measurements). Its largest observed spread was 0.593%.
Spread is
100 × (max simulated duration / min simulated duration − 1): the worst observed pair, not a confidence interval or a router speedup. Every comparison within a configuration uses one exact commit. Inherited, partial and memory results are excluded, and all workload-name sets match. The sub-1% criterion is the practical repeatability target used for these experiments, not a universal significance threshold. Not every individual benchmark's observed spread decreases.Testing simpler configurations after worker isolation
These candidates differ only in the runtime helper and corresponding configuration tests. Worker isolation, benchmark code, builds, inputs, dependencies, pinned Node 24.8.0 and streaming fixtures remain the same. The last two candidates form a direct paired comparison: both keep the upstream Node fixes and eager baseline compilation, both use stock incremental-marking task behavior, and their runtime settings differ only in the heap budget.
Restoring stock incremental-marking task behavior passes when the heap budget and eager baseline compilation remain. Removing the heap budget in that same configuration fails the target. Removing eager baseline compilation with the heap retained also fails. These results justify removing the incremental-marking task override while retaining the two other project-specific controls in this PR.
A failing candidate can stop once an observed pair exceeds 1%, since further repetitions cannot reduce its max/min spread. The final default-heap candidate was stopped early on that basis; only its complete 132-measurement runs are included, and cancelled/partial results are excluded. Any retained configuration must complete all five repetitions.
The chosen configuration is the simplest stable one among the combinations tested; this is not an exhaustive search over every V8 flag combination. The stock-runtime result shows that file isolation alone does not establish repeatability.
Removing only the heap flag initially looked acceptable after two runs (0.980% worst spread), but the third exposed larger outliers. Across all five runs, Vue route headers reached 2.145% and Vue document middleware 2.109%; the other 130 benchmarks stayed below 0.80%. This is why the decisions use complete repeated suites rather than accepting the first promising pair.
Retained runtime controls
--no-maglev--no-opt, which disables TurboFan but leaves Maglev enabled on Node 24. A hot-loop reproduction confirmed this using the actual merged worker flags.--no-minor-gc-task--no-scavenge-taskflag, which the pinned integration omits on modern Node. This restores its intended allocation-driven minor-GC behavior.--always-sparkplug--initial-old-space-size=512The first two flags follow CodSpeed's upstream Node 24 fixes. The helper augments the integration's existing flags only when CodSpeed is active in CPU simulation or legacy instrumentation mode; its existing 4096 MiB maximum remains. These extra flags are absent from ordinary Vitest timing, walltime and memory execution.
Heap sizing and representativeness
The original
--initial-old-space-size=512did not make the first 512 MiB of cumulative allocation GC-free. It affected the old-generation budget. Minor collections still ran: a local SSR reproduction recorded 13 allocation-triggered scavenges inside the measured invocation, around 17–22 MiB of heap usage. CodSpeed's explicit pre-measurement full collection also still ran.However, V8 uses the explicit initial size as a minimum allocation budget and can skip automatic full collections below it. It does not preallocate 512 MiB, but it can exclude full-GC work that a default-heap process would perform. Increased object retention can therefore have part of its eventual GC CPU cost omitted from these short simulations. See V8's initial/minimum heap configuration.
The final configuration retains this policy deliberately, based on the comparisons above. The result establishes repeatability, not representative production full-GC frequency or proof of sensitivity to every possible regression. The default-heap results remain in the PR so this tradeoff is visible when reviewing the change.
Worker and fixture changes
speed.bench.ts; 48 additional variants move to descriptive filenames. An AST comparison verifies all 93 names and completebench()calls are unchanged. The 21 affected project TypeScript includes coverspeed*.bench.ts. All 39 client benchmark files already contained a single workload. Vitest isolation.setImmediatedirectly. This fixture change and file isolation apply whenever the CPU-suite fixtures run.Exactly 48 CodSpeed identities change because their file paths move; all 132 workload names match. The six original benchmark paths remain. Runtime and worker changes also alter absolute performance baselines. Compare code changes with the same configuration on both sides; shifts against the previous setup are not router optimizations or regressions.
Validation
pnpm test:unit,pnpm test:typesandpnpm test:eslintpass for affected projects and task dependencies. Nx reuses valid caches; the remaining SSR utility tests and affected type checks execute.git diff --checkand the offline workflow security check pass.setImmediatecalls, React, Solid and Vue builds pass. Each rebuilt handler passes the streaming response assertions for a sanity request and 16 concurrent requests. All six split React assets/serialization/rewrites cases passed locally; an AST comparison checked all 93 SSR workloads.Reproduce against an unchanged branch:
Wait for each run to finish before dispatching the next repetition at the same commit. Repeat five times, exclude memory results, verify complete fresh CPU-result coverage and calculate the per-benchmark spread defined above. The temporary CPU-only input and manual concurrency changes used during investigation are not retained in the PR; the workflow is unchanged from the implementation base.
Post-isolation experiment commits and CI runs
Four flags: remove only the heap override —
ef00e4d015a82e6264e1d0bc3cf8d30ae574461c.33973371004, 33973371983, 33973373355, 33973374620, 33973375866.
Stock CodSpeed runtime: no added flags —
0499502b9d54d2804d0474c0f4f7335f4ed18101.33973481240, 33973482899, 33973484373.
Two upstream Node fixes only —
8d3c986f505f3d970cf39253c51632d1a06be501.33973626220, 33973627252, 33973628425.
Upstream fixes + heap budget —
aa87dc713e838c70b8ddfec6ceee83f49f6a0b63.33975539435, 33975541195, 33975542465, 33975543777, 33975545229.
Upstream fixes + eager baseline + heap budget —
5dcdaa92015934633031479da015ef0edf67742b.33977041102, 33977042635, 33977044106, 33977045809, 33977047895.
Upstream fixes + eager baseline; stock heap and marking tasks —
fbf0ea15e507639acd9a8b352d65057c20464afd.33977180457, 33977181771, 33977183020 (cancelled), 33977184251 (cancelled), 33977185532 (cancelled).
Earlier experiments explaining the initial stabilization approach
These experiments preceded the flag-removal comparisons above and explain the initial five-flag proposal.
Early partial fixes were insufficient. Repeating them exposed outliers that were absent in their first two runs:
The targeted comparison selected 18 Solid/Vue SSR measurements in streaming, global middleware and control flow. Both candidates below have the same counted-turn streaming change and three runtime flags. Only the initial heap flag differs:
Disabling incremental marking entirely was rejected: its two complete targeted runs had a worst spread above 9%. A third attempt was partial because the CodSpeed installer download lost its connection; it is excluded. The counted-turn change alone did not remove the streaming outlier, so it is not credited with that improvement.
Native profiles of the streaming outlier had identical application call counts, with additional cost concentrated in allocation-heavy functions. Local GC tracing of a standalone Solid SSR workload retained 13 allocation-triggered scavenges in the measured invocation with and without the fixed initial budget. The fixed budget removed incremental-marking activity during warmup in that reproduction. These traces support the runtime/heap explanation; the repeated Linux simulations establish the observed repeatability. They do not isolate every internal V8 operation responsible for each historical outlier.
Fresh-worker controls were the next decisive experiment: selecting only the React linked-CSS and plain-dehydration controls, with all five flags unchanged, produced three-run spreads of 0.018% and 0.086%. Both shared-worker cases had previously been outliers. The full integration then applied one workload per file to every SSR case, rather than special-casing those two.
Three clean Vue middleware builds produced byte-identical emitted artifacts. A task turn before GC allowed more warmup objects to be reclaimed, but three targeted runs still left a 4.10% linked-CSS spread. Adding GC and another cleanup turn still left 1.34%. Those hooks were removed. Disabling Sparkplug entirely left 1.41%; eager baseline compilation held the same assets workload to 0.65% across five repetitions. These are observed repeatability results; tracing with additional measurement markers did not directly catch baseline compilation inside the CSS measurement, so the exact contribution of compilation and heap layout is not fully isolated.
The pinned Node version is 24.8.0. The original control commit differs from the implementation base only in the manual workflow support, so it measures the original benchmark runtime/workloads. The final worker-isolation comparison retains the same five runtime flags and benchmark calls as the preceding candidate. These experiments narrow the sources of instability, but do not establish a unique internal V8 explanation for every historical spike.
Original configuration — commit
002eda8a10b69426aad7ce0de4fc7860c31b6d16.GitHub: 33959196258, 33959198216, 33959200409.
Disable Maglev — commit
c3782a40360982521942417e4c3058d1a8707cbd.GitHub: 33959197229, 33959199431, 33959201353.
Also disable minor-GC tasks — commit
347ecfa724588983673b1d05652ad480908e37b3.GitHub: 33960606235, 33960607571, 33960608833, 33960997294, 33960998027.
Also disable incremental-marking tasks — commit
d44ba4c42a3708a69b8e7d0f90536e075e929179.GitHub: 33961681221, 33961682787, 33961684239, 33961685473, 33961686639.
Fixed initial heap and counted streaming turns — commit
3e0be7a0c67913ef476bb432636a32808df03d3b.GitHub: 33965107993, 33965109291, 33965110654, 33965111891, 33965113523.
Eager baseline; multiple workloads per worker — commit
085581bc115799d643fcf26e8d257735d6a110cf.GitHub: 33967663478, 33967664880, 33967666372, 33967667824, 33967668939.
Initial isolated-worker proposal: five flags — commit
504b6b4e549b2511a722c3d34f8a63f98c9e41f1.GitHub: 33970160374, 33970161802, 33970163178, 33970164414, 33970165694.
The isolated React linked-CSS/plain-dehydration control experiment used
1b286aae70af2f92dc603e7e6ffd29aac35fef94: 33968947657, 33968948792, 33968949736.All 132 benchmark spreads
Values are rounded to three decimal places; a displayed 0.000% need not be exactly zero.
✅ Checklist
🚀 Release Impact
Summary by CodeRabbit
Benchmarking
Documentation
Maintenance