Skip to content

test(cuda.core): isolate pending-call saturation cleanup#2406

Open
Andy-Jost wants to merge 2 commits into
NVIDIA:mainfrom
Andy-Jost:fix-pending-call-saturation-test
Open

test(cuda.core): isolate pending-call saturation cleanup#2406
Andy-Jost wants to merge 2 commits into
NVIDIA:mainfrom
Andy-Jost:fix-pending-call-saturation-test

Conversation

@Andy-Jost

@Andy-Jost Andy-Jost commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Summary

Isolate the CPython pending-call saturation probe so its process-global queue cannot contaminate other tests running concurrently under free-threaded Python. Explicitly drain the synthetic no-op callbacks before exercising cuda.core's existing retry path.

Changes

  • Run test_pending_call_queue_saturation_preserves_cleanup in a subprocess using the current Python executable.
  • Call Py_MakePendingCalls after saturating the queue and before creating the second graph attachment that retries cleanup scheduling.
  • Keep the cleanup-completion and main-thread-finalization assertions inside the isolated process.

Test Coverage

  • python -m py_compile cuda_core/tests/graph/test_graph_definition_lifetime.py
  • pre-commit run --all-files
  • Full CI passed all ten Python 3.14t rows across Linux x86-64, Linux arm64, and Windows.
  • Targeted Python 3.14t reruns passed on Linux x86-64 (T4), Linux arm64 (L4), and Windows (A100).

Related Work

Run the process-global queue probe in a subprocess and explicitly drain its synthetic callbacks so parallel free-threaded tests cannot contaminate one another.
@Andy-Jost Andy-Jost added this to the cuda.core 1.2.0 milestone Jul 22, 2026
@Andy-Jost Andy-Jost added bug Something isn't working P0 High priority - Must do! cuda.core Everything related to the cuda.core module labels Jul 22, 2026
@Andy-Jost Andy-Jost self-assigned this Jul 22, 2026
@copy-pr-bot

copy-pr-bot Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@Andy-Jost

Copy link
Copy Markdown
Contributor Author

/ok to test

@github-actions

Copy link
Copy Markdown

@mdboom

mdboom commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Driveby comment: We could add a dependency on pytest-forked and it could handle running the test in a separate process.

Pass the finalization timeout into the child so process isolation does not depend on importing test helpers from the temporary working directory.
@Andy-Jost

Copy link
Copy Markdown
Contributor Author

/ok to test

@Andy-Jost

Copy link
Copy Markdown
Contributor Author

Driveby comment: We could add a dependency on pytest-forked and it could handle running the test in a separate process.

Oh, it looks like pytest-forked uses os.fork (not supported by CUDA) and also doesn't work on Windows.

@Andy-Jost
Andy-Jost marked this pull request as ready for review July 22, 2026 21:49
@Andy-Jost
Andy-Jost requested a review from mdboom July 22, 2026 21:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working cuda.core Everything related to the cuda.core module P0 High priority - Must do!

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI: graph-lifetime tests flaky on free-threaded 3.14t jobs since #2371

2 participants