Skip to content

Cancellation does not reach several pipeline subprocesses; two workers never arm the parent-PID watchdog #519

Closed

Description

@thcp

Several pipeline stages spawn subprocesses without registering them, so pressing Cancel does nothing until the stage finishes on its own. Separate from #514, which covers the Demucs worker specifically.

.claude/rules/python-fastapi.md is explicit: "Always register subprocess with set_proc(job_id, proc) immediately after Popen(); deregister in finally."

1. The local-upload ffmpeg calls are unregistered

app/pipeline/runner.py:79 and :130 (verified):

result = subprocess.run(cmd, capture_output=True, timeout=TIMEOUT_FFMPEG)

Neither _extract_video_track nor _prepare_local_source calls set_proc. Cancel during "Preparing audio..." on a 400 MB .mp4 terminates nothing; both calls can run their full TIMEOUT_FFMPEG (300s each, 600s total on the mp4 path) while the UI shows "Cancelling".

collect._run_ffmpeg demonstrates the correct pattern for exactly this reason.

2. The beat-grid decode is unregistered and the stage is uncancellable

app/pipeline/beatgrid.py:625 calls the shared helper with the whole track:

loaded = _load_audio_ffmpeg(source, sr=BEATGRID_SR, duration=None, timeout=TIMEOUT_BEATGRID)

and that helper (app/pipeline/analyze.py:212) is:

proc = subprocess.run(cmd, capture_output=True, check=True, timeout=timeout)

No set_proc. Worse, compute_beat_grid(stems_dir) takes no Job, so it cannot check cancel_requested at all. Cancel during beatgrid on a 10-minute track terminates nothing, and the decode plus beat_this inference plus _refine_beats all run to completion -- minutes -- before _check_cancel at runner.py:228 finally fires.

3. Two of the three workers never arm the parent-PID watchdog

Verified: STEMDECK_PARENT_PID is set in exactly one place, app/pipeline/separate.py:66, and read in exactly one place, app/pipeline/demucs_worker.py:118.

app/pipeline/vocal_split.py:64 and app/pipeline/sections.py:217 build a child env (they set PYTHONIOENCODING) but do not set STEMDECK_PARENT_PID, and neither vocal_split_worker.py nor section_worker.py watches for it.

Force-Quit the desktop app during a vocal split and the audio-separator/onnxruntime child keeps running with no parent and no timeout of its own, holding the GPU until it finishes. section_worker has the same exposure for a CPU inference pass whose only bound was the now-absent parent's TIMEOUT_SECTIONS.

This is precisely the orphan scenario _watch_parent's own docstring describes, and .claude/rules/ml-pipeline.md says to preserve it when touching worker startup/env.

4. A running vocal split cannot be cancelled at all

app/api/jobs.py:328-330: cancel_job returns early when job.status in ("done", "error", "cancelled"). A vocal split only ever runs on a done job, so cancel_requested is never even set. split_vocals has no cancel check either, and start_vocal_split holds runner._pipeline_lock for the whole run.

So: start a vocal split on a long track, then import three songs. The queue is stalled behind it with no way to abort, and the cancel button does nothing.

Fix

  • Route the runner.py and analyze.py ffmpeg calls through the registered-subprocess helper.
  • Give compute_beat_grid the Job so it can check cancel_requested between phases.
  • Set STEMDECK_PARENT_PID in vocal_split.py and sections.py, and arm _arm_parent_watchdog() in both workers.
  • Allow cancellation of a vocal split on a done job, and check cancel_requested inside split_vocals.

Test

.claude/rules/testing.md requires a cancellation test per pipeline stage. None of the four above has one.

Activity

  1. added
    bugSomething isn't working
    reliabilityStability, resource management, race conditions
    on Aug 31, 2026
  2. added a commit that references this issue on Aug 31, 2026
    3e89b45
  3. self-assigned this
    on Oct 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingreliabilityStability, resource management, race conditions

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions