Skip to content

cuda.core: Windows pinned-memory OOM cascade and host-pool teardown are untracked since #3426 closed #3472

Open

Description

@Andy-Jost

#3426 was closed by the merge of #3436, because the squash message carried closes #3426. #3436 fixed only the first half of #3426: the kernel-access test is now serialized on Windows. The rest is still open and has no issue.

Still open:

  1. OOM cascade on Windows. Three jobs in [BUG]: Windows pinned-memory crashes and resource failures following #3405 #3426 report CUDA_ERROR_OUT_OF_MEMORY in pinned-memory tests and then cascade to access violations. In the run analysed on fix(cuda.core): serialize pinned-memory kernel-access test on Windows #3436, the cascade starts in the teardown of test_mempool_watermark_reset[PinnedMR-used]. fix(cuda.core): serialize pinned-memory kernel-access test on Windows #3436 serialized that test. Whether that removes the cascade is not confirmed.
  2. Default host pool allocation on Windows. Allocating from PinnedMemoryResource() without options fails with CUDA_ERROR_OUT_OF_MEMORY on several Windows configurations, also before fix(cuda.core): grant the current device access to PinnedMemoryResource pools #3405 (see the analysis by @juenglin on fix(cuda.core): serialize pinned-memory kernel-access test on Windows #3436). _allocate_pinned_buffer_or_xfail turns this into an xfail under the placeholder TODO(#9999), so test_pinned_memory_resource_kernel_access[current] never reaches the kernel launch there.
  3. Host pool teardown. fix(cuda.core): serialize pinned-memory kernel-access test on Windows #3436 carried a workaround that skipped access revocation when a locally created pinned host pool is destroyed. It was backed out before the merge, because it changes teardown for users on every platform and the root cause is not known.

Tasks:

Related: #3350, #3405, #3436, #3458.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P1Medium priority - Should dobugSomething isn't workingcuda.coreEverything related to the cuda.core module

Type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions