Repository navigation
The 20-minute pip cap kills a healthy CUDA download on a slow connection #602
Description
What happens
First-run CUDA setup appears to hang, then fails. Retrying sometimes works, which makes it look intermittent and machine-specific. It is neither: it is a bandwidth threshold nobody declared.
The mechanism
Each python -m pip install pass gets a flat 20 minutes from spawn (desktop/src-tauri/src/main.rs), enforced by a deadline that has no idea whether the child is making progress.
The Linux CUDA runtime pass moves roughly 3 GB (cudnn 707 MB, cublas 594 MB, nccl 322 MB, cusparse 288 MB, cusparselt 287 MB, cusolver 268 MB, cufft 193 MB, triton 156 MB, and more). Fitting that into 1200 seconds requires a sustained 2.5 MB/s. That requirement is written down nowhere and is not checked.
Evidence it is the cap and not a hang
From a reporter's two consecutive runs on the same machine, same wheel, nothing changed in between:
| run | cudnn | rate | outcome |
|---|---|---|---|
| 1 | 921s, unfinished | 0.77 MB/s | killed at the 1200s cap, still downloading |
| 2 | 193s | 3.66 MB/s | whole pass finished with ~170s to spare |
The second run did not fix anything. The connection was 4.7x faster. Anyone whose link resembles run 1 never completes setup, and no amount of retrying helps them.
Secondary problem in the same path
When every candidate wheel fails to install, the run is still recorded as cuda-verify-failed, even though verify was never reached. That string is the first thing anyone reads in a bug report and it points at the wrong stage.
Constraints
- Raising the number does not fix it. 20 minutes to 60 minutes just moves the threshold; the property that is wrong is measuring elapsed time rather than progress.
- The project already has the right pattern elsewhere.
.claude/rules/python-fastapi.mdrequires a stall watchdog on long subprocesses andapp/pipeline/separate.pyimplements one for Demucs. The Rust side is the one place that used a wall clock. - Something still has to bound a child that never finishes, so a stall budget needs a hard backstop behind it.
- Whatever replaces it must not disturb
GPU_VERIFY_TIMEOUTor the grandchild-pipe handling from [Bug]: the subprocess timeout is not a bound when the child has children of its own #583, which share the same wait helper.
Surfaced by the logs on #502.
Activity
thcp commentedon Sep 9, 2026
Fixed in #606, landed on main as 53a6c04.
Streamed pip passes now give up on silence rather than on elapsed time: five minutes with no output, bounded by a four hour backstop. pip emits a Progress line continuously while bytes move, so a gap in them is the only honest signal that it is wedged. This is the pattern the project already required of long subprocesses and that separate.py implements for Demucs; the Rust side was the one place still using a wall clock.
The old-pip retry path, which drops --progress-bar raw and then prints nothing, keeps a plain deadline. A stall budget there would have killed a healthy download at five minutes, which would have been worse than the cap it replaced. That was caught in review, not by the tests.
Also fixed on the same path: a run where every candidate wheel failed to install was recorded as cuda-verify-failed even though verify was never reached. Those now say cuda-install-failed.
Not reproducible locally, so this ships on unit tests covering both directions plus the logs on #502.
fix(setup): install a torchvision that matches the torch it ships bes…