Repository navigation
Open
Labels
P1Medium priority - Should doMedium priority - Should dobugSomething isn't workingSomething isn't workingcuda.coreEverything related to the cuda.core moduleEverything related to the cuda.core module
Milestone
Description
#3426 was closed by the merge of #3436, because the squash message carried closes #3426. #3436 fixed only the first half of #3426: the kernel-access test is now serialized on Windows. The rest is still open and has no issue.
Still open:
- OOM cascade on Windows. Three jobs in [BUG]: Windows pinned-memory crashes and resource failures following #3405 #3426 report
CUDA_ERROR_OUT_OF_MEMORYin pinned-memory tests and then cascade to access violations. In the run analysed on fix(cuda.core): serialize pinned-memory kernel-access test on Windows #3436, the cascade starts in the teardown oftest_mempool_watermark_reset[PinnedMR-used]. fix(cuda.core): serialize pinned-memory kernel-access test on Windows #3436 serialized that test. Whether that removes the cascade is not confirmed. - Default host pool allocation on Windows. Allocating from
PinnedMemoryResource()without options fails withCUDA_ERROR_OUT_OF_MEMORYon several Windows configurations, also before fix(cuda.core): grant the current device access to PinnedMemoryResource pools #3405 (see the analysis by @juenglin on fix(cuda.core): serialize pinned-memory kernel-access test on Windows #3436)._allocate_pinned_buffer_or_xfailturns this into an xfail under the placeholderTODO(#9999), sotest_pinned_memory_resource_kernel_access[current]never reaches the kernel launch there. - Host pool teardown. fix(cuda.core): serialize pinned-memory kernel-access test on Windows #3436 carried a workaround that skipped access revocation when a locally created pinned host pool is destroyed. It was backed out before the merge, because it changes teardown for users on every platform and the root cause is not known.
Tasks:
- Confirm from nightly CI whether the cascade still occurs after fix(cuda.core): serialize pinned-memory kernel-access test on Windows #3436.
- Find the cause of the default-host-pool OOM on Windows, and replace
TODO(#9999)with this issue's number. - Bring back the teardown workaround only with evidence that revocation before
cuMemPoolDestroyis the cause. If it changes library behavior, it needs a release note. - Remove the
thread_unsafemarker ontest_mempool_watermark_resetwhen [BUG]: test_mempool_trim crashes intermittently on Windows (suspected driver race) #3350 is resolved.
Metadata
Metadata
Assignees
Labels
P1Medium priority - Should doMedium priority - Should dobugSomething isn't workingSomething isn't workingcuda.coreEverything related to the cuda.core moduleEverything related to the cuda.core module
Activity