Repository navigation
Cancellation does not reach several pipeline subprocesses; two workers never arm the parent-PID watchdog #519
Description
Several pipeline stages spawn subprocesses without registering them, so pressing Cancel does nothing until the stage finishes on its own. Separate from #514, which covers the Demucs worker specifically.
.claude/rules/python-fastapi.md is explicit: "Always register subprocess with set_proc(job_id, proc) immediately after Popen(); deregister in finally."
1. The local-upload ffmpeg calls are unregistered
app/pipeline/runner.py:79 and :130 (verified):
result = subprocess.run(cmd, capture_output=True, timeout=TIMEOUT_FFMPEG)Neither _extract_video_track nor _prepare_local_source calls set_proc. Cancel during "Preparing audio..." on a 400 MB .mp4 terminates nothing; both calls can run their full TIMEOUT_FFMPEG (300s each, 600s total on the mp4 path) while the UI shows "Cancelling".
collect._run_ffmpeg demonstrates the correct pattern for exactly this reason.
2. The beat-grid decode is unregistered and the stage is uncancellable
app/pipeline/beatgrid.py:625 calls the shared helper with the whole track:
loaded = _load_audio_ffmpeg(source, sr=BEATGRID_SR, duration=None, timeout=TIMEOUT_BEATGRID)and that helper (app/pipeline/analyze.py:212) is:
proc = subprocess.run(cmd, capture_output=True, check=True, timeout=timeout)No set_proc. Worse, compute_beat_grid(stems_dir) takes no Job, so it cannot check cancel_requested at all. Cancel during beatgrid on a 10-minute track terminates nothing, and the decode plus beat_this inference plus _refine_beats all run to completion -- minutes -- before _check_cancel at runner.py:228 finally fires.
3. Two of the three workers never arm the parent-PID watchdog
Verified: STEMDECK_PARENT_PID is set in exactly one place, app/pipeline/separate.py:66, and read in exactly one place, app/pipeline/demucs_worker.py:118.
app/pipeline/vocal_split.py:64 and app/pipeline/sections.py:217 build a child env (they set PYTHONIOENCODING) but do not set STEMDECK_PARENT_PID, and neither vocal_split_worker.py nor section_worker.py watches for it.
Force-Quit the desktop app during a vocal split and the audio-separator/onnxruntime child keeps running with no parent and no timeout of its own, holding the GPU until it finishes. section_worker has the same exposure for a CPU inference pass whose only bound was the now-absent parent's TIMEOUT_SECTIONS.
This is precisely the orphan scenario _watch_parent's own docstring describes, and .claude/rules/ml-pipeline.md says to preserve it when touching worker startup/env.
4. A running vocal split cannot be cancelled at all
app/api/jobs.py:328-330: cancel_job returns early when job.status in ("done", "error", "cancelled"). A vocal split only ever runs on a done job, so cancel_requested is never even set. split_vocals has no cancel check either, and start_vocal_split holds runner._pipeline_lock for the whole run.
So: start a vocal split on a long track, then import three songs. The queue is stalled behind it with no way to abort, and the cancel button does nothing.
Fix
- Route the
runner.pyandanalyze.pyffmpeg calls through the registered-subprocess helper. - Give
compute_beat_gridtheJobso it can checkcancel_requestedbetween phases. - Set
STEMDECK_PARENT_PIDinvocal_split.pyandsections.py, and arm_arm_parent_watchdog()in both workers. - Allow cancellation of a vocal split on a done job, and check
cancel_requestedinsidesplit_vocals.
Test
.claude/rules/testing.md requires a cancellation test per pipeline stage. None of the four above has one.
Activity
fix(pipeline): let cancellation reach the processes it is meant to st…
0.16.1: macOS song sections, settings data loss, songs coming back fr…
fix(pipeline): let cancellation reach the processes it is meant to stop
test: cover tearing down a child that will not go quietly