Skip to content

[release/cuda-core-v1.3.x] Backport the cuda.core 1.3.1 changes - #3474

Draft
Andy-Jost wants to merge 8 commits into
NVIDIA:release/cuda-core-v1.3.xfrom
Andy-Jost:backport-1.3.1-canary
Draft

Andy-Jost wants to merge 8 commits into
NVIDIA:release/cuda-core-v1.3.xfrom
Andy-Jost:backport-1.3.1-canary

Conversation

@Andy-Jost

@Andy-Jost Andy-Jost commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Backports the cuda.core 1.3.1 changes from main to release/cuda-core-v1.3.x, in the order they merged on main:

Each commit is a git cherry-pick -x. Please keep the (cherry picked from commit ...) lines in the squash message.

More 1.3.1 changes will be added here as they merge on main. Part of #3459.

🤖 Generated with Claude Code

Andy-Jost and others added 4 commits November 16, 2026 11:38
…t pinning cuda-bindings (NVIDIA#3441)

* docs(cuda.core): drop the Breaking Changes section name from the floor policy

The release notes no longer have a Breaking Changes section for a floor
move; 1.3.0 lists it under Highlights. Say only that the release notes
list every move.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(cuda.core): advise against pinning cuda-bindings next to cuda-core

Users who pin cuda-bindings themselves can conflict with the floor of a
later cuda-core release. Tell them to depend on cuda-core with the cu12
or cu13 extra and let cuda-core choose the cuda-bindings version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 83a3422)
Signed-off-by: Ralf Juengling <rjuengling@nvidia.com>
(cherry picked from commit 1a22eec)
…NVIDIA#3436)

* fix(cuda.core): serialize pinned-memory kernel-access test on Windows

closes NVIDIA#3426

Add @thread_unsafe_on_windows to the pinned-memory kernel-access test.
The test accesses host memory while parallel workers may execute GPU
work; synchronizing its own stream does not coordinate those workers.

This follows the guard used by similar tests on Windows and WSL.
Windows CI is needed to confirm it resolves the reported crashes.
The separate OOM failures remain under investigation.

Validation: Ruff and git diff --check pass.

* fix(cuda.core): report CUDA synchronization errors during test teardown

Check the cuCtxSynchronize result in init_cuda so asynchronous errors
surface during the triggering test's teardown instead of going unnoticed
until a later test. Always attempt context cleanup via finally.

Related to NVIDIA#3426.

* fix(cuda.core): synchronize the bound context during test teardown

Use device.sync() because the test wrapper may pop the current context
before init_cuda teardown. This avoids spurious INVALID_CONTEXT errors
while preserving synchronization error reporting and context cleanup.

Add regression coverage for a missing current context and cleanup after
synchronization failures.

Related to NVIDIA#3426.

* fix(cuda.core): serialize memory-pool watermark reset tests

Mark test_mempool_watermark_reset thread_unsafe, matching test_mempool_trim.
Both exercise the pool teardown and synchronization sequence associated
with the suspected driver race in NVIDIA#3350.

* fix(cuda.core): bypass access revocation when destroying host pools

Skip peer-access revocation for locally created HOST and HOST_NUMA pools
to investigate pinned-pool teardown OOMs and crashes. Continue destroying
the pools and preserve cleanup for device, managed, and imported pools.

This is a diagnostic workaround pending Windows CI validation.

Related to NVIDIA#3426.

* fix(cuda.core): preserve the first exception during fixture cleanup

Keep setup or synchronization failures primary and attach subsequent
cleanup failures as exception notes. Always attempt to pop the context.

Add regression coverage for combined failures, including interrupts.

* fix(cuda.core): limit host-pool cleanup workaround to pinned allocations

HOST and HOST_NUMA also describe managed-pool preferences. Check the allocation type so the pinned-memory workaround preserves managed-pool peer-access revocation.

Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>

* test(cuda.core): expose pinned allocation OOM and retain its traceback

Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>

* ci: diagnose PR3436 pinned pool teardown on Windows

Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>

* ci: move watermark investigation out of PR3436

Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>

* test(cuda.core): restore expected pinned allocation OOM handling

Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>

* revert(cuda.core): defer pinned host pool cleanup workaround

Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>

---------

Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>
Co-authored-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>
(cherry picked from commit b715078)
…3471)

* ci: exclude unavailable historical GitHub references

Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>

* ci: require manual copy-pr-bot approval

Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>

---------

Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>
(cherry picked from commit 3991a69)
@Andy-Jost Andy-Jost added this to the cuda.core 1.3.1 milestone Nov 16, 2026
@Andy-Jost Andy-Jost added P0 High priority - Must do! test Improvements or additions to tests cuda.core Everything related to the cuda.core module labels Nov 16, 2026
@Andy-Jost Andy-Jost self-assigned this Nov 16, 2026
@github-actions github-actions Bot added the CI/CD CI/CD infrastructure label Nov 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor

@Andy-Jost
Andy-Jost marked this pull request as draft November 16, 2026 20:16
@copy-pr-bot

copy-pr-bot Bot commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

…IA#3359)

_smv_dtype_numpy_to_dlpack dispatched on numpy dtype kind, but
ml_dtypes.bfloat16 has kind "V", so bf16 views fell through to the
unsupported-dtype BufferError and could not be re-exported, even though
the DLPack import path already accepts kDLBfloat and _tensor_map.pyx
maps bfloat16. Detect bfloat16 up front and export it as kDLBfloat
(16 bits).

The new round-trip test goes through StridedMemoryView.from_dlpack so
it cannot skip on stacks where NumPy's own DLPack support for extended
dtypes is unavailable, unlike the existing parametrized round-trip
test's NumPy-consumer probe.

Signed-off-by: Sainath-Reddy7 <180950590+Sainath-Reddy7@users.noreply.github.com>
Co-authored-by: Sainath-Reddy7 <180950590+Sainath-Reddy7@users.noreply.github.com>
Co-authored-by: Sebastian Berg <sebastianb@nvidia.com>
(cherry picked from commit d1d1457)
@Andy-Jost Andy-Jost changed the title [release/cuda-core-v1.3.x] Backport #3441, #3433, #3436, #3471 for cuda.core 1.3.1 [release/cuda-core-v1.3.x] Backport #3441, #3433, #3436, #3471, #3359 for cuda.core 1.3.1 Nov 16, 2026
…me descriptor (NVIDIA#3450)

* fix(cuda.core): make IPC imports wait for a pending release of the same descriptor

deviceptr_import_ipc() shares one handle per descriptor because the
driver's first free of an imported pointer unmaps every import of it
(nvbug 5570902). The handle's deleter took ipc_import_mutex only after
the last reference was gone. In that gap an import found the cache entry
expired, re-imported the still-mapped pointer and registered a new entry.
The deleter then erased that new entry and freed the pointer, which
unmapped it under the new Buffer.

The cache entry now outlives the last reference and only its own deleter
erases it, under the mutex, together with the free. An import that finds
an expired entry waits on a condition variable until that release is
done, then imports fresh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Use CTAD for the ipc_import_mutex locks

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 2f1185e)
@Andy-Jost Andy-Jost changed the title [release/cuda-core-v1.3.x] Backport #3441, #3433, #3436, #3471, #3359 for cuda.core 1.3.1 [release/cuda-core-v1.3.x] Backport the cuda.core 1.3.1 changes Nov 17, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI/CD CI/CD infrastructure cuda.core Everything related to the cuda.core module P0 High priority - Must do! test Improvements or additions to tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants