Persistent demucs worker: eliminate per-job model reload on GPU (35-42% of separate stage) #309

Background

Follow-up to #288, which instrumented and measured separate_startup (subprocess spawn + demucs/torch model load, up to the first progress line) on real hardware. Full numbers in the closing comment.

Summary: on an RTX 3080, startup is 35-42% of the entire separate() stage (steady-state: ~5s of ~14s). On CPU it's ~8% (~7s of ~85s) -- comfortably a rounding error, per the plan's original "keep as-is" expectation. GPU crossed the >20% trigger the plan set for opening this follow-up.

Why this happens

Every job spawns a fresh python -m demucs subprocess (app/pipeline/separate.py::_demucs_cmd), which re-imports torch, re-loads the htdemucs_6s model weights (~/~80MB, 6-source model) onto the GPU, and initializes CUDA context -- all before the first sample gets processed. On GPU this fixed cost barely shrinks while the actual separation work drops dramatically (GPU compute is ~5-6x faster than CPU for this model), so the fixed cost dominates proportionally more, not less, on faster hardware.

Constraint: keep what the subprocess boundary buys us

The current design is deliberate, not accidental (see separate()'s docstring and #276/#282's history) -- subprocess isolation gives us:

• Kill-on-cancel: POST /cancel calls proc.terminate() directly and the job dies within ~1s, no cooperative cancellation needed inside torch's inference loop.
• Crash containment: a demucs/CUDA crash (OOM, driver fault) takes down the subprocess, not the FastAPI server -- separate() sees a nonzero exit code and falls back to CPU (No GPU→CPU fallback when separation fails #276) or fails the job cleanly, instead of the crash propagating into the process serving every other request.

Any fix must preserve both properties. A naive "run demucs in-process" refactor would lose them.

Proposed design: warm worker, cold-killed only on cancel/crash

Since _pipeline_lock = asyncio.Semaphore(1) already means only one separation runs at a time, there's no pooling problem to solve -- just keep-alive for a single worker:

1. A long-lived worker subprocess (spawned once, on first job or app startup) loads the demucs model and then blocks reading job requests from stdin (or a small local socket) -- one line in, one job's-worth of stderr-style progress lines out, same \r-delimited protocol the current reader loop already parses.
2. separate() sends {source, device, job_dir} to the worker instead of spawning a new subprocess; reads progress the same way it does today.
3. Cancel: unchanged behavior -- proc.terminate() kills the worker outright (mid-inference cancellation inside torch isn't realistically interruptible anyway). The next job after a cancel pays the cold-start cost again (a fresh worker spawns lazily), same as an ordinary crash-recovery path.
4. Crash containment: unchanged -- a worker crash (OOM, CUDA fault) is detected the same way (nonzero exit / EOF on the pipe), separate()'s existing GPU->CPU fallback and error classification apply unmodified. A fresh worker spawns for the next job.
5. Net effect: consecutive normal (non-cancelled, non-crashed) jobs on the same device reuse the warm worker and pay ~0 startup cost after the first. Only cancel and crash pay the full reload -- both already rare/exceptional paths, and both already pay a comparable cost today (a fresh process either way).

Scope check before starting

This is a real architecture change (worker lifecycle management, an IPC protocol beyond a one-shot subprocess, respawn-on-crash logic) -- bigger than any single Phase 3 PR. Recommend scoping as its own mini-project: prototype the worker protocol first (measure actual warm-job time to confirm the ~5s/job GPU win holds up under the IPC overhead), then wire it into separate() behind the existing device-fallback logic.

Not blocking anything currently shipped -- today's behavior (fresh subprocess per job) is correct and safe, just leaves GPU throughput on the table for back-to-back jobs.