Skip to content

Karaoke split dies with "operator torchvision::nms does not exist" after a cu128 install #614

Closed

Description

@thcp

Reported in #502 after the CUDA install finally completed.

What happens

vocal split failed: operator torchvision::nms does not exist
  File "app/pipeline/vocal_split.py", line 151, in split_vocals

Separation works. The karaoke split does not, on every attempt, permanently.

Why

torchvision ships compiled ops registered against one specific torch ABI. Loaded next to a different torch, the registration silently does not happen, and the failure surfaces much later from whatever first calls one of those ops.

The path there:

  1. The lockfile resolves torch 2.6.0 with torchvision 0.21.0, a matched pair.
  2. The CUDA setup force-installs torch and torchaudio from the PyTorch index with --no-deps, so pip touches only what is named.
  3. torchvision is not named, so it stays at 0.21.0.
  4. On the cu128 line, and only there, torch_version_for_tag moves torch to 2.8.0. Now torchvision 0.21.0 is sitting next to torch 2.8.0.
  5. audio-separator pulls in onnx2torch-py313, which requires torchvision. The karaoke split calls into it, hits nms, and finds nothing registered.

Separation is unaffected because Demucs never touches torchvision, which is why this looks like a karaoke-only bug rather than a broken install.

Who gets it

Only installs that land on cu128, because every other tag keeps torch at the locked 2.6.0 and stays paired.

That used to mean Blackwell alone. #602 widened it: a CUDA 13 driver now selects cu128 on any card, which is correct for the driver and is what put the reporter's machine on that line. So the change that fixed their install is what exposed this.

Constraints for a fix

  • --no-deps is deliberate on the first pass and must stay: it is what makes the swap fast and stops pip rebuilding the world. So torchvision has to be named explicitly rather than resolved.
  • The version has to track torch_version_for_tag. Two mappings that must agree is a thing that drifts, so they belong next to each other with the reason written down.
  • restore_cpu_torch has the same shape and the same gap. It happens to heal the mismatch today, because putting torch back to 2.6.0 re-pairs it with 0.21.0, but that is luck rather than intent.
  • Verify any version against the actual index before pinning it. torchvision 0.23.0+cu128 exists; do not assume a pairing from memory.

Activity

  1. added a commit that references this issue on Sep 9, 2026
    7d60953
  2. self-assigned this
    on Oct 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions