Skip to content

EPIC: Port Rapids projects from pynvml to cuda.core.system #1915

Open

Description

@mdboom

This is an EPIC to track porting usage of pynvml to cuda.core.system across the RAPIDS family of projects. In order of importance / level of NVML usage.

Naming fixes prereq: #1946

Activity

  1. self-assigned this
    on Apr 15, 2026
  2. added this to the cuda.core v1.0.0 milestone on Apr 15, 2026
  3. added
    cuda.coreEverything related to the cuda.core module
    on Apr 15, 2026
  4. added theissue type on Apr 15, 2026
  5. jakirkham commented on Apr 23, 2026

    @jakirkham

    Thanks Mike! 🙏

    In many cases we run the benchmarks and examples on CI. So we may still want to migrate them if the goal is dropping PyNVML

    That said, think the prioritization here is reasonable. Also we are very appreciative on the RAPIDS side of you picking this up and pushing it forward

  6. added
    P1Medium priority - Should do
    EPICSoul of a release
    on May 1, 2026
  7. mdboom commented on Jun 10, 2026

    @mdboom
    ContributorAuthor

    Merging many of these Rapids PRs above is dependent on changing how dependencies are specified: rapidsai/build-planning#285

  8. jameslamb commented on Jul 7, 2026

    @jameslamb
    Member

    Merging many of these Rapids PRs above is dependent on changing how dependencies are specified: rapidsai/build-planning#285

    I've just finished rapidsai/build-planning#285. Across RAPIDS, we no longer depend directly on cuda-python in any libraries (even in testing environments), and instead have these:

    cuda-bindings >=12.9.2,<13.0  # (CUDA 12)
    cuda-bindings >=13.0.1,<14.0  # (CUDA 13)
    cuda-core>=0.5.1
    

    Most projects only depend on cuda-bindings, cuda-core is only used in a handful of libraries today.

    I went with those ranges because they most closely matched what cuda-python was pulling in for us. @mdboom looks like your series of PRs is proposing moving those floors higher, we'd appreciate your input on why: NVIDIA/cudf#22305 (comment)

  9. kkraus14 commented on Jul 8, 2026

    @kkraus14
    Collaborator

    @jameslamb I would recommend tightening the cuda-core pin if possible. There was a number of breaking changes introduce in cuda-core 1.0, after which we are supporting strict semantic versioning until we release a 2.0 which would be aligned with CUDA 14. At the very least would recommend pinning to <2.0 as well.

  10. mdboom commented on Jul 14, 2026

    @mdboom
    ContributorAuthor

    Agree with @kkraus14 on moving to the API-stable cuda-core 1.0.0.

    Additionally, NVML support wasn't added to cuda-bindings until 13.2.0 (and 12.9.6 on the 12.x series).

  11. leofang commented on Sep 2, 2026

    @leofang
    Member

    I would like to transfer this issue to our private repo, but it's in a different org now 🤔

  12. vyasr commented on Oct 13, 2026

    @vyasr
    Contributor

    @mdboom @kkraus14 we're no longer seeing major issues here and we just bumped our cuda-core requirement to 1.0 across RAPIDS. The main outstanding concern is that this forces us to require numba-cuda 0.30.2, but since we're actively working on our migration to numba-cuda-mlir my hope is that by our next release (26.12) we will no longer carry a numba-cuda dependency and so requiring such a new version of it will not cause any issues.

    @mdboom would you be able to revive your pynvml PRs now? Apologies that we took so long to unblock you.

  13. mdboom commented on Oct 14, 2026

    @mdboom
    ContributorAuthor

    It's important note that we have to /both/ move to cuda-core >= 1.0 /and/ cuda-bindings >= 12.9.6 or >= 13.3.0 (depending on CTK series). Once cuda-core 1.3.0 is released (which will have a proper floor on the cuda-bindings version), that second dependency will no longer be needed. So there is a decision to make:

    1. Add the complex cuda-bindings version specifier now
    2. Wait until cuda-core 1.3.0 and move directly to that (which may be ok given the stable cuda-core dependency is brand-new anyway).

    I have updated all of the linked PRs to take approach (1) and done another sweep for any pynvml usage that may have creeped in since the last time I filed the PRs.

  14. mdboom commented on Oct 14, 2026

    @mdboom
    ContributorAuthor

    Closing the loop from a Slack discussion:

    We can't yet ship these changes to rapids because pytorch 2.10 still pins cuda_bindings at version 13.0 (which does not have NVML support). More recent versions of torch change this to a version floor so things will work once the older versions of pytorch no longer need to be supported.

    Given that we need to wait on that, we can also wait for cuda-core 1.3.0 which will specify a cuda_bindings version floor that includes all of the NVML functionality. This will remove the need for RAPIDS to specify /both/ a dependency on cuda-core and cuda-bindings, and can specify the direct dependency alone (on cuda-core).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

EPICSoul of a releaseP1Medium priority - Should docuda.coreEverything related to the cuda.core module

Type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions