Repository navigation
ci: 2-3x faster pipeline — parallel jobs, layer/build caches, parallel e2e - #47
Merged
Merged
Conversation
The workflow was still tuned for the old memory-constrained shared self-hosted runner. On GitHub-hosted runners every job gets its own isolated VM, so the serialization is pure wall-clock waste: - drop max-parallel: 2 from the go matrix (5 modules now run concurrently instead of in three waves) - drop needs: [go, helm] from e2e — it delayed the ~22 min e2e job by ~6.6 min for no resource benefit - drop the cross-run gameplane-kind-e2e concurrency group that queued e2e between unrelated PRs (clusters no longer share a machine) - split the single serial e2e job into four independent jobs, each with its own kind cluster: core, extended, web-live (playwright), and game-bot (advisory, unchanged continue-on-error) - the web-live job bootstraps the admin via TestAPI_BootstrapAndLogin first, which writes test/e2e/.tmp/admin-password for globalSetup.ts - drop the teardown and vite port-cleanup steps (ephemeral VMs) and the per-run cluster-name suffix (no shared docker daemon to collide in) - paths-ignore: keep "website" (matches the submodule gitlink) and add website/** for future in-tree files - port the web coverage commit-status step from the Forgejo API path to the GitHub API (github.api_url); it was silently 404ing - add timeout-minutes to every job Buckets are unchanged: same tests, same regexes, same -timeout values. Measured baseline: 28.5 min wall. Expected after this change: ~12.5 min (critical path becomes the extended e2e job). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: valgulnecron <39313199+ValgulNecron@users.noreply.github.com>
Move the two giant inline RUN_REGEX lists out of ci.yaml into test/e2e/buckets.sh, which prints each bucket's anchored regex and — the point of the exercise — verifies that the buckets are pairwise disjoint and exhaustive against the actual Test functions in test/e2e/*_test.go. A new cheap e2e-buckets CI job runs that check, so a newly added e2e test that isn't assigned to a bucket now fails CI instead of silently never running. The verify pass found five tests the old inline regexes had silently omitted (they have NEVER run in CI): TestGameServer_HeartbeatReachesRunning, TestModuleSourceUpload, TestAPI_CustomRole_Lifecycle, TestAPI_BuiltinRole_Immutable, TestAPI_PerNamespaceBinding_GrantsScopedAccess. They are parked in an explicit, documented 'unbucketed' allowlist here and get bucketed in the next commit, keeping that change revertable on its own. Also adds 'make test-e2e-bucket BUCKET=...' so local runs use the same bucket definitions as CI. Bucket contents are byte-identical to the old inline lists. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: valgulnecron <39313199+ValgulNecron@users.noreply.github.com>
TestGameServer_HeartbeatReachesRunning, TestModuleSourceUpload, TestAPI_CustomRole_Lifecycle, TestAPI_BuiltinRole_Immutable, and TestAPI_PerNamespaceBinding_GrantsScopedAccess existed in the suite but were missing from the old inline ci.yaml regexes, so they had never run in CI. Assign them to buckets (GameServer/roles → core, module upload → extended) and empty the unbucketed allowlist. Isolated commit on purpose: if any of these turns out red in CI, revert this without touching the pipeline restructure. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: valgulnecron <39313199+ValgulNecron@users.noreply.github.com>
Nothing in the pipeline was cached, so every run re-downloaded modules, recompiled from scratch, re-fetched the envtest apiserver binaries, and re-downloaded chromium: - new .github/actions/e2e-images composite action builds the three e2e images through buildx with cache-from/to type=gha (same pattern as publish-edge.yaml), replacing plain 'make e2e-images' in all four e2e jobs — warm builds skip the module download and most of the compile - explicit per-module go module+build caches in the go matrix (setup-go's built-in cache keys off a repo-root go.sum this go.work layout doesn't have, and one shared key would only ever save the first leg's cache); the e2e jobs share one 'e2e' cache for the test-binary compile - pin setup-envtest to release-0.19 (matches controller-runtime v0.19.0; @latest was unpinned) in ci.yaml and the Makefile, and cache the ~/.local/share/kubebuilder-envtest assets so the K8s 1.31 binaries aren't re-downloaded per run - cache ~/.cache/ms-playwright in web-e2e-mock and e2e-web-live keyed on web/package-lock.json Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: valgulnecron <39313199+ValgulNecron@users.noreply.github.com>
The suite was strictly sequential, yet most tests are ~all waiting: cron windows (1-min floor), restic round-trips, agent mTLS startup, module registry indexing. t.Parallel() on every test overlaps those waits instead of summing them. Two tests stay serial by design: the login rate-limit test (below) and the advisory Minecraft bot. Shared state that parallelism exposes, and how it's handled: - the four module tests delete-and-recreate the fixed-name oras-push Job (and the signed test the cosign keypair) — a new ociPushMu mutex serializes them against each other while they still overlap with the rest of the suite - concurrent first backups race 'restic init' on the empty shared repo; the loser's Job pod fails once and the Backup controller reports Failed (Failed>0 is checked before Active), which waitBackupSucceeded treats as terminal. New ensureResticRepo(t) helper (sync.Once) seeds the repo via a one-shot warm-up Job before any backup test runs, replacing the per-test fixture applies - TestHelmInstall_AllPodsReady demanded Ready on EVERY gameplane-system pod; one-shot helper pods (healthz probes, oras/cosign jobs) in Succeeded phase would flake it. Scoped to the operator/api pods, which is what its doc comment promises anyway - the API login rate limiter (per-IP 5/min burst 10, per-user 3/min burst 6) is the real bound on API-test wall clock: every test shares one client IP through the port-forward and logs in as e2e-admin. The CI buckets are re-cut by login pressure — operator (32 tests, zero logins, -parallel 8), api-auth (9, -parallel 4), api-servers (14, -parallel 4) — instead of core/extended - TestAPI_LoginRateLimit deliberately drains those buckets. Go runs non-parallel tests FIRST, so keeping it serial in-bucket would starve everyone else's logins — it runs as a separate final go test step - APIClient's 429 retry ceiling raised 5→7 attempts (capped 30s step) for logins queued behind the refill; Eventually polls at 500ms instead of 1s; the e2e API memory limit goes 512Mi→1Gi since up to 4 argon2id logins (64Mi each) can now hash concurrently Same tests, same assertions — only the scheduling changed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: valgulnecron <39313199+ValgulNecron@users.noreply.github.com>
consistently() sleeps its entire window unconditionally on every call — ~19.5s of guaranteed serial sleeping across 11 call sites, plus two 500ms settling sleeps. Every misbehavior these windows guard against is event-driven: a wrong reconcile fires within milliseconds of the triggering write in envtest (poll interval is 200ms), so a 750ms-1.5s window observes the same failures the old 1-3s windows did. The cron-shaped guards (suspend honored, invalid cron, Forbid) could only ever catch the immediate-fire bug class anyway — a fires-at-the-next- minute bug is out of reach of any sane window, shortened or not. agent_certs' window is already at the 1s floor and stays put. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: valgulnecron <39313199+ValgulNecron@users.noreply.github.com>
Every HashPassword in tests ran at production cost — 64 MiB and t=3 per call — multiplied across seedUser in the whole handler suite, the auth round-trip tests, and each bootstrap-admin invocation. On top of that, dummyHash was computed at package init, so every test binary that so much as links the auth package paid one full-cost hash before running anything. - argonTime/argonMemory become package vars; new auth.SetFastHashParams (takes just the Cleanup-bearing subset of testing.TB, so no testing import in prod code) drops them to t=1/8MiB for a test and restores on cleanup. VerifyPassword decodes its parameters from the PHC hash string, so cheap hashes verify cheap and production hashes are untouched. - dummyHash moves to sync.OnceValue: computed on first VerifyDummy, not at init. - Wired into the hash-heavy helpers: auth's and handlers' seedUser, the cmd runBootstrap helper, and the salt/round-trip tests. TestHashAndVerify deliberately stays on production parameters so the real cost path keeps getting exercised. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: valgulnecron <39313199+ValgulNecron@users.noreply.github.com>
First parallel CI run confirmed the login rate limiter as the binding constraint, two ways: - TestAPI_LoginPrivacy observes RAW login status codes (it bypasses APIClient's 429 retry on purpose) and got a 429 where it asserts 401 because parallel neighbors had drained the per-IP bucket. Drop its t.Parallel(): Go runs non-parallel tests to completion before the parallel phase, so it now sees a fresh limiter by construction. - the api-servers bucket made ~14 admin logins against a per-username budget of burst 6 + 3/min refill; two tests exhausted even the 7-try retry ceiling, and the retry storm's argon2id load knocked on into TestAPI_LifecycleRestart's restart budget. Split it into api-rbac (7 tests) and api-agent (7 tests) so each cluster's limiter only absorbs ~7 admin logins — about one burst — per job. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: valgulnecron <39313199+ValgulNecron@users.noreply.github.com>
The composite action ran three sequential build-push-action builds and every e2e job exported the same cache scopes. Measured on the first cached run: ~6 min per job — WORSE than the uncached 3:18 baseline — because the compile layers can never cache across commits (the source COPY invalidates them), each job paid ~1 min re-uploading identical cache, and the three compiles ran back to back. Switch to docker buildx bake (new docker-bake.hcl, shared with local use) so the three images build concurrently, and add an export-cache input to the composite action so exactly one job per run (e2e-web-live, the shortest) writes the cache while everyone else only imports the dependency-download layers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: valgulnecron <39313199+ValgulNecron@users.noreply.github.com>
New e2e tests must land in a bucket (buckets.sh verify gates CI), call t.Parallel(), and budget API logins against the per-IP/per-user rate limiter — none of which is discoverable without reading the CI history. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: valgulnecron <39313199+ValgulNecron@users.noreply.github.com>
The test POSTed :restart as soon as pod-0 existed, without waiting for the agent sidecar. The restart handler patches suspend=true, waits up to 45s for the StatefulSet to drain, then patches suspend=false; the operator's graceful stop needs the agent to run the stop sequence and otherwise waits out the 30s stop-grace period. Under the parallel suite's load the agentless drain exceeds the 45s window, the resume patch races the reconcile, the operator coalesces the two suspend patches, and the restart becomes a silent no-op — pod UID never changes and the test times out. Reproduced twice in CI, both times as the bucket's only failure. Wait for the agent (requireAgentReady, same helper the other agent tests use) so the test restarts what the dashboard restarts: a running server. The drain-window race itself is a real product wart under load — flagged in PR #47 as an operator-side follow-up (rule 10: a restart primitive belongs in the reconciler, not in handler patch choreography). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: valgulnecron <39313199+ValgulNecron@users.noreply.github.com>
Two more parallel-suite findings from CI runs 28704030433/28704239141: - api-auth at 9 tests still overran the per-IP login bucket (~14 logins vs burst 10 + 5/min): TestAPI_PerNamespaceBinding_GrantsScopedAccess exhausted the retry ceiling and its siblings passed only after 60-90s of 429 queueing. Move the three role-management tests to their own api-roles bucket/cluster. - TestAPI_LifecycleRestart kept failing even after waiting for the agent: restart is two suspend patches around a bounded drain-wait, and with the reconcile queue deep (parallel bucket) both patches land between two reconciles of the GameServer — the operator coalesces them and the restart is silently lost (pod UID never changes). Run it without t.Parallel(): Go executes non-parallel tests before the parallel phase, giving it the idle operator its contract currently needs. The lost-restart race itself is an operator-side follow-up (restart should be a reconciler primitive, not handler patch choreography). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: valgulnecron <39313199+ValgulNecron@users.noreply.github.com>
ValgulNecron
pushed a commit
that referenced
this pull request
Oct 9, 2026
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Cuts CI wall-clock from a measured 28.5 min toward ~10 min, running the same suites (plus five e2e tests the old regexes had silently dropped). Coverage gates are untouched.
The workflow was still tuned for the old memory-constrained shared Forgejo runner. On GitHub-hosted runners every job is an isolated VM, so all of that serialization was pure wall-clock waste.
Commits
max-parallel: 2on the go matrix, dropneeds: [go, helm]on e2e (started ~6.6 min late), drop the cross-run kind concurrency group (queued PRs behind each other), split the single 21.8-min serial e2e job into independent jobs each with its own kind cluster. Also ports the web coverage commit-status step off the dead Forgejo API path.-runregex lists move from ci.yaml intotest/e2e/buckets.shwith averifysubcommand (new cheap CI job) proving buckets are disjoint + exhaustive, so a new test can never silently not run in CI again.TestGameServer_HeartbeatReachesRunning,TestModuleSourceUpload,TestAPI_CustomRole_Lifecycle,TestAPI_BuiltinRole_Immutable,TestAPI_PerNamespaceBinding_GrantsScopedAccesshad never run in CI. Isolated commit → cleanly revertable if any is red.type=ghalayer cache for the three e2e images (composite action, pattern from publish-edge), per-module go module+build caches (setup-go's default cache is broken by the go.work layout), pinnedsetup-envtest@release-0.19+ cached K8s 1.31 envtest assets, cached playwright chromium.t.Parallel()everywhere safe (waits overlap instead of summing);ociPushMuserializes the four module tests sharing the fixed-name oras-push Job; a one-shot restic warm-up Job (ensureResticRepo, sync.Once) kills therestic initrace on the shared repo;TestHelmInstall_AllPodsReadyscoped to chart pods so one-shot helper pods can't flake it. Buckets re-cut by login pressure (the per-IP/per-user login rate limiter, not CPU, bounds API-test wall time):operator(32 tests, no logins,-parallel 8) /api-auth(9,-parallel 4) /api-servers(14,-parallel 4), withTestAPI_LoginRateLimitas a separate final step since it deliberately drains the shared limiter. API pod memory 512Mi→1Gi for concurrent argon2id logins.consistently()windows — ~10s of unconditional sleep trimmed; every guarded misbehavior is event-driven.auth.SetFastHashParams(t=1/8MiB, restores on cleanup) wired into the hash-heavy helpers;dummyHashnow lazy (sync.OnceValue) so linking the package no longer costs a 64MiB hash at init. One round-trip test stays on production params.Measured baseline (main run 28681350157)
needs+ everything serial on one clustermax-parallel: 2Early results (run on commits 1–4, cold caches)
All go jobs concurrent and green; e2e web-live green in 10 min; the advisory Minecraft bot test passed (it consistently failed on the old shared runner — the port-forward corruption was load-induced).
🤖 Generated with Claude Code