Model weights reloaded per job #288

Activity

1. thcp

   added

   performanceSpeed, memory, and resource efficiencySpeed, memory, and resource efficiency

   on Jul 16, 2026
2. thcp

   added a commit that references this issue on Jul 17, 2026

   measure(pipeline): record demucs startup cost per attempt (#307)

   ...

   5e8caeb
3. thcp commented on Jul 17, 2026

   

   thcp

   on Jul 17, 2026

   CollaboratorAuthor

   More actions

   Measured on real hardware

   Machine: Windows, NVIDIA RTX 3080 (driver 595.79). Same 220s track (mix.wav from a real job), 3 runs per device via separate() directly.

   Device	Run	separate_startup	total separate()	startup % of stage
   CPU	1 (cold)	20.2s	90.9s	22%
   CPU	2 (warm)	6.9s	82.8s	8%
   CPU	3 (warm)	6.8s	80.8s	8%
   GPU (cuda)	1 (cold)	7.3s	17.5s	42%
   GPU (cuda)	2 (warm)	5.0s	14.0s	36%
   GPU (cuda)	3 (warm)	4.8s	13.7s	35%

   (Cold = first invocation in the process/environment, paying disk-cache warm-up on top of the fixed model-load cost; warm = steady state.)

   Decision

   Per the plan: keep if startup is 5-15s of a 1-15min stage (CPU: yes, ~7s of ~85s, comfortably in that range); open a follow-up with a concrete design if it's >20% of separate time on GPU.

   GPU is 35-42% — well over the >20% trigger. On fast hardware, subprocess spawn + demucs/torch model load is no longer a rounding error, it's over a third of the actual GPU-bound work. Opening a follow-up issue for a persistent-worker design (keeps subprocess isolation's kill-on-cancel/crash-containment properties, but the interesting part is figuring out how the model survives across jobs without giving that up -- proposal there).

   Closing this issue -- the follow-up is the actionable next step. Data below to inform it.
4. thcp

   closed this as completedon Jul 17, 2026
5. thcp

   added a commit that references this issue on Jul 17, 2026

   feat(pipeline): persistent demucs worker (#309) (#310)

   ...

   08b6abf
6. thcp

   self-assigned this

   on Oct 1, 2026