Skip to content

[no-ci] ci: plan builds and artifact reuse per bindings root - #3321

Merged
rwgk merged 17 commits into
rwgk/cb12-staged-05-releasesfrom
rwgk/cb12-staged-06-planning
Nov 13, 2026
Merged

rwgk merged 17 commits into
rwgk/cb12-staged-05-releasesfrom
rwgk/cb12-staged-06-planning

Conversation

@rwgk

@rwgk rwgk commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

— codex GPT-6 on behalf of Ralf —

Layer 6 of 7 of the staged "CUDA 12 bindings on main" stack. Previous: #3319. Next: #3322. Stack map at the bottom.

What this PR does

Teaches CI to do only the work a change needs, per bindings root. Before this layer, PR CI built and tested both majors in full.

  • ci/tools/compute_ci_plan.py: planning targets become (bindings, root), (core, cuNN), (python, root), and pathfinder. A cuda_bindings_12/ edit rebuilds the CUDA 12 bindings and metapackage, reuses Core, Pathfinder, and CUDA 13 wheels from the baseline run at the exact merge base, and tests only CUDA 12 rows. Core edits rebuild both ABI halves. Shared or unknown paths build everything. A bare v12.* or v13.* tag builds only its own line and uses published Pathfinder.
  • ci/tools/select_test_matrix.py (new) replaces the jq program that was duplicated in both test-wheel workflows: per-row suite selection, local versus published floor bindings, runner-driver mapping, and a guard so a PR cannot silently lose a required local-wheel row. Job names now say what runs, for example pathfinder+bindings12+core12+meta12 / py3.12, CTK12.9.1 (wheels; bindings PR).
  • build-wheel.yml: ci/tools/ci_env.py turns the workplan into per-root build and test variables; every step is gated per root; reusable artifacts are downloaded from the baseline and re-uploaded under the new SHA. ci/tools/ci_artifacts.py shares artifact naming between producers, consumers, and baseline completeness checks. Inputs cuda-version / prev-cuda-version are replaced by bindings-config.
  • ci.yml: ci-vars is folded into detect-changes. One planner emits eight outputs: bindings-config and workplan JSON, five job gates, and the Windows ARM64 single-major flag. The final status check uses the same job gates and rejects unexpected skips.
  • Test workflows check out ci/tools from the workflow revision (.ci-control), so nightly can test an older artifact source with current tooling.
  • sdist jobs consume the planner’s package execution records. ci/tools/select_pixi_packages.py shares source-impact selection with the planner and also selects package-local Pixi manifests and documentation changes for the source-build smoke.

After this merges

  • PR CI work drops for single-root changes; pushes to main, scheduled runs, and non-bindings tags still build all packages.
  • ci.yml uses the PR wheel-test matrix, including for pushes to main, matching production. The separate ci-nightly.yml workflow retains production’s integration and standard-test schedule.
  • Artifact reuse engages only when the base branch has a successful push run at the exact merge base with the full artifact set. On main that becomes true after the first post-merge push run; before that, PRs build everything.
  • Scheduled GPU source tests cover both registered environments; manual dispatch selects one and defaults to the current environment.

Where to look

All changes (27 files, +1,970 / -1,130)

Workflows: build-wheel.yml, ci.yml, ci-nightly.yml, coverage.yml, ci-pixi-source-test.yml, test-sdist-linux.yml, test-sdist-windows.yml, test-wheel-linux.yml, test-wheel-windows.yml; .github/actions/fetch_ctk/action.yml.

Tools: compute_ci_plan.py (per-root impacts, execution records and scheduler outputs), select_test_matrix.py (new), ci_env.py (new: build/test environments; replaces env-vars), ci_artifacts.py (new: shared artifact identity and baseline completeness), select_pixi_packages.py (new), bindings_config.py (Windows ARM64 configuration), cuda_core_bindings_floor.py, run-tests (published Pathfinder for focused releases).

Tests: test_compute_ci_plan.py, test_select_test_matrix.py (new), test_select_pixi_packages.py (new), test_merge_cuda_core_wheels.py (new), test_bindings_config.py, ci/tools/tests/AGENTS.md (new: proportional test guidance).

Configuration and release guidance: ci/versions.yml, .github/RELEASE-bindings.md.

Validation
Stack map: cumulative capabilities by layer
Layer PR Capability added
1 #3314 behaves exactly as before; CI has one reader for the two bindings roots; CUDA 13 versions can no longer match a v12.* tag
2 #3315 lets a developer build, test, and document cuda_bindings_12/; the PR-time CPU source-build smoke covers both roots; three CUDA 12 runtime fixes are in
3 #3317 builds the CUDA 12 bindings wheel itself and runs its GPU tests; cuda.core and the metapackage still consume 12.9.x artifacts
4 #3318 uses same-run CUDA 12 wheels everywhere; PR CI no longer reads the 12.9.x branch
5 #3319 can release CUDA 12 from a v12.9.N tag on main; CUDA 13 releases are unchanged
6 #3321 builds and tests only what a change needs, per bindings root, reusing baseline artifacts
7 #3322 adopts the schema-2 registry, removes backport automation, adds metadata checks and maintenance docs; 12.9.x is frozen

Layers #3314 and #3315 have merged. The remaining five migration PRs (#3317, #3318, #3319, #3321 and #3322) will land together using GitHub’s atomic stack merge, with full CI on #3322 as described in the landing plan. #3434 follows separately, after a successful post-migration main CI run supplies complete artifacts for both CUDA majors.

@copy-pr-bot

copy-pr-bot Bot commented Oct 31, 2026

Copy link
Copy Markdown
Contributor

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@rwgk
rwgk added this pull request to stack #3316 October 31, 2026 05:45
@github-actions github-actions Bot added the CI/CD CI/CD infrastructure label Oct 31, 2026
@rwgk rwgk added this to the cuda.bindings next milestone Oct 31, 2026
@rwgk rwgk added enhancement Any code-related improvements cuda.bindings Everything related to the cuda.bindings module labels Oct 31, 2026
@rwgk rwgk self-assigned this Oct 31, 2026
@rwgk
rwgk force-pushed the rwgk/cb12-staged-06-planning branch 2 times, most recently from fdc9d18 to ba05a1d Compare October 31, 2026 07:49
@rwgk
rwgk force-pushed the rwgk/cb12-staged-06-planning branch from ba05a1d to 463bd2c Compare October 31, 2026 08:03

rwgk commented Oct 31, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 463bd2c

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor
Doc Preview CI
Preview removed because the pull request was closed or merged.

rwgk commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 88a2ab3

@rwgk
rwgk force-pushed the rwgk/cb12-staged-06-planning branch 4 times, most recently from 7418aad to c1b95b6 Compare November 6, 2026 21:38

rwgk commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

— codex GPT-6 on behalf of Ralf —

Major issue #1: Increase in nightly jobs

Addressed with 4a0207a. This commit restores production’s nightly scope in #3321 and propagates it into #3322.

The eight integration callers no longer run an additional standard-mode pass, and the dedicated standard caller is ARM64-only again. Additional coverage, including H200, can be considered separately from this migration.

Expanded jobs per scheduled run Before Restored / production
GPU tests 44 25
Matrix computation 20 11
Other jobs 5 5
Total 69 41

These are configured job counts, not GPU-hours. The selected rows, modes and platforms match production; #3322 retains its source-aware artifact routing and security fixes. The comparison was checked locally, and all pre-commit checks passed.

rwgk commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

— codex GPT-6 on behalf of Ralf —

Major issue #2: Change in nightly platform coverage

Clarified the PR description to distinguish package scope from matrix coverage.

ci.yml uses the PR wheel-test matrix for pushes to main, matching production’s existing configuration. The change identified in the review is relative to #3319’s intermediate state; it preserves production’s matrix-selection policy.

The separate nightly workflow retains production’s integration and standard-test schedule following the correction for Major issue #1.

@rwgk

rwgk commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 29ca7e0

@mdboom mdboom left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this part of the simplification still hasn't been done. Let's do it (or explain why we can't):

Phase 2: planner and registry emit scalars where if: needs them.

Extend compute_ci_plan.py --github-output to also write flat outputs for platforms.linux, platforms.windows, windows_arm64_build, sdist_tests and core_api_checks.
In the same call, emit current-cuda-version and windows-arm64-*, so the three-mechanism registry step collapses into one.
Keep workplan and bindings-config as the JSON outputs for reusable workflows.
The checks job then reads needs.detect-changes.outputs.platform-linux and similar, with no fromJSON(...) || false chains.

Now that env-vars is just a simple wrapper around ci-env.py, let's remove it and update all users of env-vars to use the Python script.

@rwgk

rwgk commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

— codex GPT-6 on behalf of Ralf —

Major issue #3: Accidental complexity converting between multiple forms of the same data

A further simplification pass is implemented in 2676e38 and af6cd64 on #3321, and propagated into #3322 at a26b60e. Local validation passed. Full CI remains pending while we address the remaining review feedback.

Changes and rationale
  • Removed env-vars; all four callers invoke the existing Python helper directly.
  • Consolidated registry and planner outputs into one producer, removing the separate registry writer and unused CLI modes.
  • Moved build-interpreter selection into a host-keyed workplan map, removing the separate python-versions input. The build workflow now takes three inputs instead of four.
  • Moved scheduler decisions out of workplan JSON into flat outputs. Job conditions and final checks consume those values directly.
  • Made missing scheduler outputs fail the final check, preventing silently skipped coverage.
  • Removed four redundant conditions inside the reusable test workflows.

The resulting interface has eight outputs: two JSON payloads, five scheduler gates, and one existing single-major policy flag. Previously it had six. The simplification removes separate producers, serialized lists, wrappers, and repeated parsing; it does not reduce the total output count.

The established baseline resolver and single-major build sequence were deliberately retained to keep this pass focused.

Incremental diff statistics

Net changes on #3321, relative to its previous head:

Scope Added Removed Net
Production/configuration 96 159 −63
Tests 22 5 +17
Total 118 164 −46

Ordinary Git line counts, including comments and blank lines. One focused Windows ARM64 regression test was added; existing outcome assertions were adapted.

Review and local validation

An independent Codex pass was followed by two review rounds with Fable 5.2, invoked with --effort max. Its final patch review found no blocking issues; the recommended missing-output safeguard was included.

These are local results. Full hosted CI has not yet been requested for these revisions.

@rwgk rwgk changed the title ci: plan builds and artifact reuse per bindings root [no-ci] ci: plan builds and artifact reuse per bindings root Nov 12, 2026
@rwgk
rwgk force-pushed the rwgk/cb12-staged-06-planning branch from af6cd64 to 4d9f5ed Compare November 12, 2026 21:18
@rwgk
rwgk force-pushed the rwgk/cb12-staged-06-planning branch from 4d9f5ed to e6dde0a Compare November 13, 2026 03:04
@rwgk

rwgk commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test e6dde0a

@rwgk
rwgk merged commit 68a68b8 into main Nov 13, 2026
29 checks passed
@rwgk
rwgk deleted the rwgk/cb12-staged-06-planning branch November 13, 2026 05:25
github-actions Bot pushed a commit that referenced this pull request Nov 14, 2026
Removed preview folders for the following PRs:
- PR #3176
- PR #3182
- PR #3317
- PR #3318
- PR #3319
- PR #3321
- PR #3322
- PR #3430
- PR #3434
- PR #3456
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI/CD CI/CD infrastructure cuda.bindings Everything related to the cuda.bindings module enhancement Any code-related improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants