Skip to content

Take CUDA 12 PyTorch from the cu128 index in the cu12 training requirements - #101

Merged
ivanbasov merged 1 commit into
mainfrom
fix/cu12-train-requirements-torch-index
Nov 16, 2026
Merged

ivanbasov merged 1 commit into
mainfrom
fix/cu12-train-requirements-torch-index

Conversation

@ivanbasov

@ivanbasov ivanbasov commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

pip install -r code/requirements_public_train-cu12.txt against PyPI alone now produces an environment where the cuStabilizer backend cannot be imported:

ImportError: cuda.core 1.3.0 requires cuda-bindings >= 12.9.8 for CUDA 12, but cuda-bindings 12.9.4 is installed

This is the README training install, and it is what mid-gpu-tests, multi-gpu-tests and gpu-coverage run. All three fail on current main (manual run); the gpu-tests matrix passes because it installs through the cu128 PyTorch index.

Cause

  • PyPI torch 2.11 and later require cuda-bindings 13.x; cuquantum-python-cu12 requires <13. pip backtracks to torch 2.10.0, which pins cuda-bindings==12.9.4.
  • cuda-core 1.3.0 (PyPI, 2026-11-11) refuses to import with cuda-bindings older than 12.9.8. Before that release the same resolution worked.

Change

  • code/requirements_public_train-cu12.txt declares --extra-index-url https://download.pytorch.org/whl/cu128.
  • One comment line in the README install example.

No workflow changes: the three jobs keep their plain pip install -r, so they test the same command users run.

Verification

pip install --dry-run --report in fresh venvs:

Requirements file Python torch cuda-bindings cuda-core
main 3.12 2.10.0 12.9.4 1.3.0
this PR 3.11, 3.12, 3.13 2.11.0+cu128 12.9.10 1.3.0
this PR, --index-url .../cu121 (Dockerfile default) 3.12 2.11.0+cu128 12.9.10 1.3.0

The resolved set is the one the gpu-tests cu128 matrix already runs green.

gpu-coverage runs on this PR with the plain pip install -r and is the end-to-end check of the fix. mid-gpu-tests and multi-gpu-tests run only on main, so they are confirmed by the push run after merge. The dry-run checks resolution only; no local install or GPU run was done.

Notes

  • An alternative, a cuda-bindings>=12.9.8,<13 lower bound without the index, resolves to torch 2.9.1 from PyPI. Not taken: it silently moves users to an older torch than the one CI tests.
  • With the extra index declared in the file, a different CUDA 12 index passed on the command line (for example cu121) no longer selects the torch build; pip takes the highest version, 2.11.0+cu128. On main that case already resolves to PyPI torch 2.10.0, so the cu121 build was not being selected before either.

🤖 Generated with Claude Code

…ements

pip install -r code/requirements_public_train-cu12.txt against PyPI alone
now resolves torch 2.10.0 with cuda-bindings 12.9.4: newer PyPI torch
requires cuda-bindings 13.x, which cuquantum-python-cu12 (<13) rejects, so
pip backtracks to the last torch that pins a 12.x cuda-bindings.
cuda-core 1.3.0 refuses to import with cuda-bindings older than 12.9.8,
so the cuStabilizer backend fails with ImportError. This breaks the README
install and the mid-gpu-tests, multi-gpu-tests and gpu-coverage CI jobs.

Declare the cu128 PyTorch index in the requirements file. The same file
then resolves torch 2.11.0+cu128 with cuda-bindings 12.9.10, the set the
gpu-tests matrix already runs.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Ivan Basov <ibasov@nvidia.com>
@ivanbasov
ivanbasov marked this pull request as ready for review November 16, 2026 20:59
@ivanbasov
ivanbasov requested a review from bmhowe23 November 16, 2026 20:59
@ivanbasov

Copy link
Copy Markdown
Member Author

@anjbur could you take a look as well? The functional change is the single --extra-index-url line in code/requirements_public_train-cu12.txt; the rest is comments.

@ivanbasov
ivanbasov merged commit a04c9fd into main Nov 16, 2026
17 checks passed
@ivanbasov
ivanbasov deleted the fix/cu12-train-requirements-torch-index branch November 16, 2026 21:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants