Phase 4 — Separation SOTA: RoFormer + lead/backing vocal split #275

Activity

1. thcp

   added

   separationStem separation model/qualityStem separation model/quality

   on Jul 16, 2026
2. thcp commented on Jul 16, 2026

   

   thcp

   on Jul 16, 2026

   CollaboratorAuthor

   More actions

   Implementation plan — Phase 4: Separation SOTA (lead/backing vocal split)

   The feature phase. Architecture: second pass over the vocals stem with a karaoke/BVE model, on-demand, cached in the job dir. 6 PRs (last one optional/follow-on). Hard prerequisite: Phases 1–2 (quarantine, classifier, GPU fallback pattern) — a second ML pass must land on debuggable infrastructure.

   Decided architecture

   source ──demucs htdemucs_6s──▶ 6 stems ──(user asks)──▶ karaoke model on vocals.wav
                                                             ├─ lead_vocals.wav
                                                             └─ backing_vocals.wav
   

   • Engine: audio-separator (pip, actively maintained) as the model runner. The recommended checkpoint mel_band_roformer_karaoke_aufr33_viperx runs on torch (MDXC/roformer arch) → CUDA/MPS/CPU comes from the torch already shipped; no onnxruntime needed for the default path (ONNX only if we later enable the UVR_MDXNET_KARA_2 fallback model). This keeps portable-zip size impact ≈ 0 (models download at first use; GitHub's 2 GiB release-asset cap stays safe).
   • Trigger: on-demand (POST /api/jobs/{id}/vocal-split), not part of the main pipeline — most tracks never need it, CPU cost is minutes, and results are cached forever in the job dir.
   • Stem model: STEM_NAMES (the 6, demucs contract) stays untouched. New EXTRA_STEM_NAMES = ("lead_vocals", "backing_vocals") in config. The frontend already syncs names from /api/config (syncStemNamesFromAPI in constants.js) — extend that payload rather than hard-coding clients.
   • Mutual exclusivity: lead+backing are a decomposition of vocals. Backend mixdown guard: requesting vocals together with either derived lane → 422 (prevents doubled vocals in exports). UI enforces the same by lane grouping.
   • Models dir: MODELS_DIR / "audio-separator" via the library's model_file_dir — lands inside the portable data dir / TORCH_HOME sibling, consistent with demucs weights.

   ---

   PR-4A — Core: app/pipeline/vocal_split.py + dependency

   • Add audio-separator to pyproject (pinned exact version; document torch-version interplay — it must not force a torch upgrade in the lockfile; verify with uv lock diff before committing).
   • Module API:

   def split_vocals(job, stems_dir: Path) -> list[str]:
       """Run the karaoke model on stems/vocals.wav → lead_vocals.wav +
       backing_vocals.wav (16-bit 44.1k stereo, matching stem contract).
       Returns created names. Raises SeparationError on failure."""

   • Runs in a subprocess like demucs (same isolation rationale: kill-on-cancel via set_proc, crash containment, memory release). Thin runner script invoked as python -m app.pipeline._vocal_split_worker <in> <outdir> <model> <device>; parent parses progress from stdout lines (audio-separator logs percentages) into _set(job, stage=f"Splitting vocals {pct}%").
   • Device policy: reuse Phase 2's pattern — try get_demucs_device() resolution, on failure retry CPU once, loudly. Classifier from errors.py applies as-is.
   • Model download: first run downloads the checkpoint (~hundreds of MB) — surface as stage="Downloading vocal model (one-time)..." by watching the library's download progress output.
   • Output naming/format: enforce with an ffmpeg normalize step if the model emits anything but 44.1k/16-bit (same treatment as _prepare_local_source).
   • Env overrides: STEMDECK_KARAOKE_MODEL (default the mel-roformer checkpoint filename), enabling the ONNX fallback model without a code change.
   • Tests: worker interface mocked (stub script writes two WAVs) — split_vocals happy path, failure → SeparationError with tail, cancel kills proc. A real-model integration test exists but is @pytest.mark.manual (needs the checkpoint; run locally, never CI).

   PR-4B — API + job state + SSE

   • Job fields: vocal_split: Literal["none","running","done","error"] = "none" (+ in to_state, persisted in _JOB_FIELDS/metadata so restarts remember).
   • POST /api/jobs/{id}/vocal-split: 404 unknown/not-done job; 409 if already running; 202 if already done (idempotent). Acquires the same _pipeline_lock semaphore as full jobs (one heavy compute at a time, no starvation of separations — document that a queued separation wins by FIFO of the semaphore).
   • On success: append the two stems to job.stems, run the Phase-3 scan_stem on both and merge into peaks.json (read-modify-write via tmp+replace), update metadata.json, registry_persist.
   • On failure: vocal_split="error" + error_detail classified; job itself stays done (the base stems are untouched) — quarantine machinery not used here since the job dir must survive; instead write stems/vocal_split_error.txt.
   • SSE: no changes needed — _set bumps the version; clients see vocal_split transitions live.
   • registry._recover_done_job: add the two extra names to its scan list.
   • Tests: endpoint state machine (404/409/202/happy), idempotency, stems list + peaks merge, restart recovery includes the new lanes.

   PR-4C — Serving + export surface

   • app/api/stems.py: _ALLOWED_NAMES |= EXTRA_STEM_NAMES (serving, mp3 cache, zip); _MIXDOWN_NAMES |= EXTRA_STEM_NAMES + the exclusivity guard (vocals ⊕ derived lanes → 422 with a clear message).
   • /api/config: add extra_stem_names (keep stem_names as the 6 — existing clients unaffected; new clients merge).
   • Tests: mixdown with lead+backing gains; 422 on vocals+lead; zip includes new lanes when present; 404 before split has run.

   PR-4D — Studio + mobile UI

   • constants.js: merge extra_stem_names into lane vocab + STEM_DISPLAY ("Lead Vocals", "Backing Vocals") + two colors from the existing channel palette.
   • Studio: "Split vocals" action on the vocals lane (context: mixer row or stem strip). While running: progress from the job SSE (stage). On done: two new lanes appear (reuse the existing lane-add path from load); vocals lane and derived lanes are a radio-style group (enabling lead/backing mutes+disables vocals, and vice versa) — this mirrors how original already complements selected stems.
   • Mobile (static/mobile/app.js): same action in the Mixer tab; lanes render via the shared engine automatically.
   • Export panel: no changes needed — lanes flow through _effectiveMixGains(); the backend guard is the safety net.
   • Tests: JS is untested in repo (manual verification per /verify flow); backend contract tests in 4C carry correctness.

   PR-4E — License audit + model pinning (ship-gate, do in parallel)

   Clients depend on this app — this PR is documentation but blocks release, not merge:

   • Record in docs/models.md: exact checkpoint file, SHA256, source URL, license text/terms as published, and the decision. Known state: UVR's own models are MIT; several community roformer checkpoints (aufr33/viperx) are published freely but without an explicit commercial-use license.
   • Decision tree already made: if the mel-roformer karaoke terms are unclear/non-commercial → default STEMDECK_KARAOKE_MODEL flips to UVR_MDXNET_KARA_2.onnx (MIT, needs onnxruntime — add as extra stemdeck[karaoke-onnx]) and the roformer stays an opt-in env override. The abstraction in 4A makes this a config change.

   PR-4F (follow-on, separate design note before starting) — Vocals-boundary quality upgrade

   The report's §04 hybrid: run a RoFormer vocal model (e.g. model_bs_roformer_ep_317) for the vocals/instrumental boundary, keep htdemucs_6s for the instrument split, reconcile (instrumental_stems ×= (mix − roformer_vocals) energy match or simple replacement of the vocals lane + other residual adjustment). Meaningful SDR win but touches the core pipeline contract — needs its own design doc + A/B listening pass. Do not bundle into 4A–4E.

   ---

   Sequencing & estimates

   4A (2–3d) → 4B (1–2d) → 4C (0.5d) → 4D (2d) → 4E (0.5d, parallel) → release. 4F unscheduled until 4A–4E ship.
   Manual end-to-end gate: a track with clear harmonies (e.g. anything Beach-Boys-shaped) → split → solo backing lane and listen; export lead-only mix; restart app → lanes persist; delete job → all files gone.
3. thcp

   added the

   Feature

   issue type on Jul 25, 2026
4. thcp

   added 3 commits that reference this issue on Aug 21, 2026

   ci: install build-essential so diffq (audio-separator's dependency) c…

   ...

   0aeed27

   Lead/backing vocal split, stems relocation fixes, eager model pre-dow…

   ...

   bf561a6

   fix: pin librosa <1 and add audioread explicitly (#407) (#408)

   ...

   b50e221
5. thcp commented on Aug 21, 2026

   

   thcp

   on Aug 21, 2026

   CollaboratorAuthor

   More actions

   Shipped in 0.12.0: audio-separator adoption, on-demand lead/backing vocal split (PRs 4A-4E from the plan above), API, desktop and mobile UI, and the license audit in docs/models.md. Chose UVR-MDX-NET Karaoke 2 as the default per the plan's own decision tree, since the preferred mel-roformer karaoke checkpoint turned out to have no usable license.

   PR-4F (a RoFormer pass at the vocals/instrumental boundary for a general quality upgrade, separate from the lead/backing split) was always scoped out of this issue and was never started. The direct ask for a Roformer swap (#396) was already closed for the same licensing reason. Not tracking it further for now; closing this out.
6. thcp

   closed this as completedon Aug 21, 2026
7. thcp

   self-assigned this

   on Oct 1, 2026