idea: attribute_voices - which reference singer sings each lyric line, by timbre (pitch mislabelled 6 lip-sync shots)

Idea: a task that says which reference singer sings each line of a song, by timbre, not pitch.

Source: field report from an Opus 5.5 session on lem, 2026-09-25/26. It made a 3:12 musical short with 18 lip-synced song shots from a Music3 take.

What went wrong without it:

Music3 doesn't honour "Singer X sings section Y" (ignored on all three seeds), so you must find out who sings what before staging lip-sync shots.
The session's pitch-based voice map (demucs vocal stem, median pYIN per lyric line) called the choruses female. The tenor melody sits at 220-293 Hz, which a mezzo also reaches, and the phrases open on high shouted ensemble attacks (535-790 Hz) that pull the median up.
Six chorus shots were staged with the female lead lip-syncing a male voice. The user caught it after delivery, and all six were re-rendered (~3 GPU-hours).

What worked: speaker embeddings, done locally (la-vela/tools/who_sings.py).

Separate the vocal stem (htdemucs, two stems).
Embed a known stretch of each singer (Resemblyzer).
Score every transcribed line by cosine similarity to each singer.
Roll the lines up per shot by time overlap.

It put the chorus lines at 61-100% the male singer: right on all six.

Shape it might take: a attribute_voices task taking the song, a timestamped transcript (depends on #483) and references: {name: [spans] | asset:}, returning per-line and per-window attribution. That could feed a music-video / shots-batch shot list directly.

fix(docs,plugin): #484 - fold 24-shot field report into H3/Music3 skills

Plan v1: `attribute_voices`, which reference voice sings each line (by timbre)

Lead: claude-opus-5-5 (anthropic), design session 2026-09-26.

Verdict: build smaller

Axis	Finding
Value	Don's musical-short sessions, whenever one song has more than one singer. The evidence is one field report (this one, from lem 2026-09-25/26). It cost ~3 GPU-hours: six chorus shots re-rendered because a pitch heuristic called a tenor female. The class is real and recurring. Music3 ignores per-section singer directions (#484 records this in the skill), so every duet or ensemble song needs this check before lip-sync staging. `list_workspaces` shows 11 of Don's workspaces are music or musical pieces (`encanto-uprising`, `shedding-skin-quartet`, `acorn-wars`, `inner-space`, …); `qa-` and `regression-` are not counted. The session had to leave dw and write `who_sings.py` by hand, and the next session would do the same.
Build cost	Stage A ~$6–8, stage B ~$1–2. Total ~$7–10.
Inertia	One task in `list_tasks` plus its `docs/TASKS.md` section, and no MCP tool, so the MCP surface budget doesn't move. One new dependency (demucs) plus its weights, which come from Meta's CDN via torch.hub, not HF, so `plan.downloads_required` can't see them. That is the same existing gap as transcribe_audio's Whisper and the speechbrain encoder (#6 below). One small engine change: the `returns: "json"` kind stops meaning "assessment probe". No new concept, and no security boundary is loosened.
Reversibility	High. A task is removed without a `breaking-change` unless a template names it, and none will.
Cheaper alternatives	(a) Do nothing / skill note only. The skill already warns that Music3 ignores per-section singers (#484), but a warning doesn't give the answer, and the pitch method it would push sessions toward is exactly what failed. (b) Pitch per line (no new dependency): this is the method that mislabelled six shots, so rejected. (c) Embeddings without source separation (no demucs): the accompaniment dominates a full mix's embedding, and the working recipe separated first. Offered as `separate: false` for already-dry stems, but not as the default.
Overlap	#483 (timestamped transcribe, parked with Don) is the natural source of lines, but it is not a hard dependency here (see design). #488 (VLM lip-sync-target probe) checks the picture side of the same failure, and this checks the audio side. They are complementary, with no shared work.

What's cut, and what would bring it back:

A music-video or shots-batch template field wired to the attribution (e.g. a per-shot singer chosen automatically). It would break the pinned entry shapes in tests/test_plugin_skills.py:279-305, and the agent reads the JSON and writes the shot list itself anyway. A second report of a session mis-wiring the output back into a shot list would bring it back.
Resemblyzer (the field-proven embedder). dw already carries speechbrain as a hard dependency (dw/tasks/speech_generation.py:60-66, pyproject.toml:90-95), so the plan uses ECAPA from it instead (Q2). If ECAPA fails the tester's duet case where Resemblyzer passed, that brings Resemblyzer back.
An automatic "who is singing" with no references (clustering/diarization). This plan attributes only to named references, which is what the lip-sync staging needs.

Problem and non-goals

Problem. A generated song's section-to-singer map is unknown, since Music3 doesn't obey one. Staging lip-sync shots needs it, and pitch can't separate a tenor from a mezzo in the shared range.

Non-goals:

diarization without references;
lyric transcription (that is transcribe_audio discards the timestamps Whisper already returns past 30 s; add opt-in segment/word timestamps #483);
changing any template;
judging whether a shot's picture lip-syncs the right person (that is idea: lip-sync target check - is the singer's mouth the one moving (attribute_voices + get_output_frames) #488).

The design

Corrections to the proposal in the issue:

Transcript input. The issue makes transcribe_audio discards the timestamps Whisper already returns past 30 s; add opt-in segment/word timestamps #483 a dependency. It isn't needed: the task takes line spans directly. The field accepts transcribe_audio discards the timestamps Whisper already returns past 30 s; add opt-in segment/word timestamps #483's proposed {start, end, text} chunk shape as-is, so transcribe_audio discards the timestamps Whisper already returns past 30 s; add opt-in segment/word timestamps #483's output drops straight in via previous_result:<step> once it lands. A JSON transcript can't be an asset: (uploads are media-only, dw/security.py:26-29), so lines come inline, from a variable, or from a previous step.
Return kind. Registering a second returns: "json" command silently files it under list_tasks' assessment bucket (dw/introspection.py:395-420). That breaks tests/test_assessment_rules.py:109-112 and the "three read-only commands" text (docs/TASKS.md:902). The fix is an explicit assessment=True on register_command, set on the three probes, with the bucket read from the flag. This is a small engine change and one of the things Don approves here.
Embedder. Use speechbrain ECAPA (already a dependency, HF-hosted, so it at least lives in the HF cache) rather than Resemblyzer (Q2).

Task: attribute_voices (new module dw/tasks/voice_attribution.py, with lazy imports; consumes_device=True; returns: "json", so the step saves content_type: application/json as the probes do).

Arguments:

audio: the song (a path, asset:, output:, or a previous result).
voices: a map of name → reference. Each reference is either:
- a list of {start_seconds, duration_seconds} spans into audio itself; or
- a path to a separate clip (asset:).
It needs at least 2 voices. There is a minimum total reference length per voice (default 3 s, min_reference_seconds); below it the task refuses and names the voice. Names follow validate_variable_name's pattern.
lines (optional): a list of {start, end, text?} (seconds). This is transcribe_audio discards the timestamps Whisper already returns past 30 s; add opt-in segment/word timestamps #483's chunk shape; start_seconds/duration_seconds is also accepted.
windows (optional): a list of {name, start, end} (e.g. shots) for roll-up.
window_seconds (optional): used when lines is absent, to cut the song into fixed windows (default 2.0).
separate (default true): htdemucs two-stem vocal isolation before embedding. false is for a stem that is already dry.
device.

Neither lines nor windows given means fixed windows. No model-name argument is exposed: htdemucs and ECAPA are fixed, so nothing free-form reaches torch.hub.load.

Returns (JSON):

voices
separated
lines: one entry per line: {start, end, text, scores: {name: cosine}, voice, margin, voiced_seconds, uncertain}.
- voice is the argmax.
- uncertain: true when margin is below a declared threshold, or when the line has too little voiced audio after separation. In that second case voice is null and the reason is given.
windows: one entry per window: {name, start, end, share: {name: fraction of voiced overlap}, voice, uncertain}. Lines are rolled up by time overlap, weighted by voiced seconds, as in the field recipe.
reference_similarity: the pairwise cosine between the voices' references. A high value is a warning (voices_too_similar) that the answer is weak whatever it says.

Each thresholds is a declared constant, named in the docs. The task decides nothing and reports scores, like the assessment probes.

Dependency. Add demucs to pyproject.toml as a hard dependency with a justification comment (Q1). A missing install is a clear error naming the package, and CI tests mock it through sys.modules (as tests/test_speech_generation.py:362-376 does for speechbrain). Models are cached through cached_model(("attribute_voices", …, device, dtype)), so clear_memory frees them. The dtype comes from preferred_task_dtype. demucs on MPS is unverified: if it fails there, separation runs on CPU with a warning (the tester runs lem, which is CUDA).

Surfaces touched:

engine: the register_command assessment flag, and introspection.py's bucket;
list_tasks / get_task: one new command;
docs/TASKS.md: a new section, and the assessment count text made flag-based;
the plugin skill line (stage B).

No REST route, no MCP tool, no syntax change.

Stages

Stage A: the task (server)

Builds: the assessment flag and bucket fix; dw/tasks/voice_attribution.py; the demucs dependency (and install.sh/install.ps1 if they pin audio packages); docs/TASKS.md section.
Unit tests:
- with separation and the embedder mocked: argument validation (fewer than 2 voices, reference too short, a span outside the audio, a bad voice name, lines in both shapes);
- scoring, argmax and margin from fixed fake embeddings;
- window roll-up by overlap;
- the uncertain and voice: null paths;
- voices_too_similar;
- list_tasks still lists exactly three probes under assessment, and attribute_voices is under commands and not under assessment.
Deploy: server.
Estimate: $6–8.
Acceptance intent (over MCP):
- list_tasks lists attribute_voices among the commands, and assessment is still exactly analyze_shots, analyze_seams, analyze_sync_drift. get_task shows its arguments.
- Known answer: the tester builds a clip whose singer at each moment is known, and passes a reference span for each voice. Two ways to build it:
  - a two-voice Music3 duet, with the take's structure checked by ear from get_output_audio;
  - two generate_speech voices concatenated with concat_audio/slice_audio, over a music bed so separation has work to do.
  Each line inside one voice's stretch is attributed to that voice, and each window straddling the join reports split share.
- With lines omitted, fixed windows come back covering the whole song.
- With windows given, each named window carries share summing to ~1 over its voiced overlap.
- Refused at validation or on the step:
  - one voice only;
  - a voice whose spans total under 3 s (the error names the voice);
  - a span past the end of the audio;
  - a voice name with a disallowed character.
- The two references are the same span under two names: voices_too_similar is reported, and lines are uncertain.
- A line over pure instrumental: voice: null with the reason.
- separate: false runs, and says separated: false.
- A step saving it without content_type: application/json fails validation, as the probes do.
- A second run on the same box doesn't re-download.

Stage B: skill note (plugin-only)

Builds: one short paragraph in the minimax-music3 skill (~360 B free). It says that per-section singers are ignored, so run attribute_voices with a reference span per singer before staging lip-sync, and don't infer the singer from pitch. minimax-h3's lip-sync bullet (6 B free) gets a pointer only if an equal cut is found. tests/test_plugin_skills.py:101 then pins the task name against list_tasks.
Deploy: plugin-only.
Estimate: $1–2.
Acceptance intent: the skill text names attribute_voices, and the plugin-skill tests pass. Verified by the tester reading the skill; nothing to observe over MCP beyond list_tasks naming the task.

Order: A then B. Neither is blocked by #483. Once #483 lands, lines: "previous_result:transcribe" works with no change here (the tester can add that case then).

Risks

Speaker encoders are trained on speech, not singing. Resemblyzer worked in the field, and ECAPA is unproven on sung vocals. The stage-A known-answer case is the check. If it fails, Q2 flips.
demucs: a new dependency and ~80 MB of weights from outside HF. plan.downloads_required won't report them. That is the same unreported gap transcribe_audio and speechbrain already have, and a fix for it is a separate follow-up (a per-task default-model table in _collect_sources, dw/plan.py:799-832), not part of this plan.
Ensemble lines (both singers at once) score between the two. The share and margin show it, but voice names only one.
demucs on MPS is unverified; the CPU fallback covers it.

Questions

Q1. demucs as a hard dependency? Default: yes, since separation is the default path and an optional extra would make the default path fail on a plain install. Otherwise it becomes an audio-analysis extra, the task errors clearly without it, and separate defaults to false.
Q2. Embedder: speechbrain ECAPA (default) or Resemblyzer? ECAPA adds no dependency; Resemblyzer is the one proven on this exact case. If you pick Resemblyzer, it becomes a second new dependency (its weights ship in the package) and the rest of the plan is unchanged.
Q3. returns: "json" needs an explicit assessment flag. Default: yes, add the flag. Otherwise the task returns a dict under the default artifact kind, and a result block saves it key by key. That is clumsier to read back and doesn't match the probes.

Reply in comments and hand back with owner:lead. Approve by adding status:plan-approved yourself. No agent may set it.

approve build smaller

Tester spec for #485's plan v1, written by claude-opus-5-5 via anthropic. The cases are written from the plan alone, and each is marked pending: on its stage.

#494, stage A (attribute_voices, the assessment flag, demucs):

regression-suite-complete.md:
- C-F137: registration. The task is in commands, and assessment still lists exactly the three probes. Argument surface, with no model-name argument. Only application/json is accepted as the result type.
- C-F138: the known answer, on a duet built with CPU tasks (Priya's voice, then HAL's, then the bed alone, mixed over a looped score).
  - Held-out lines name the right voice.
  - The bed-only line comes back voice: null and uncertain.
  - A window straddling the join splits its share, and voiced shares sum to about 1.
  - It also covers the start_seconds/duration_seconds line shape, lines with no text, and clip references given as asset: paths.
- C-F139: without lines, fixed windows cover 0 to the song's duration (default 2 s, and with window_seconds: 5). separate defaults to true, and separate: false runs on a dry stem.
- C-F140: the same span under two names gives the voices_too_similar warning, and every line is uncertain. The step completes, since this is a warning.
- C-F141: refusals, each accepted at validation or on the step:
  - one voice, or none;
  - references totalling 2.9 s, where the error must name the voice;
  - a span past the end, and a span straddling the end (not clamped);
  - bad voice names;
  - a missing asset;
  - window_seconds of 0 or below.
  - Boundary control: exactly 3.0 s is accepted.
- C-F142: a second run downloads no demucs or ECAPA weights.
regression-suite-security.md:
- SE-F038: the raw-path gate holds on audio and on each nested voices clip reference (absolute, missing and ../ forms), with no existence oracle and refusal before any job queues.

#495, stage B (the skill note):

regression-suite-complete.md:
- C-F143: dw:minimax-music3 names attribute_voices, says Music 3 ignores per-section singers, says to run the task per singer before lip-sync staging, and says not to infer the singer from pitch. Any argument the skill names must exist in get_task.

Non-blocking notes (the cases accept either reading):

The plan doesn't say what an empty lines: [] does, so no case tests it.
The plan doesn't name the field that gives the reason for voice: null. C-F138 accepts any stated reason.
The plan doesn't fix where fixed-window results go when lines is absent (in lines[], or in a separate list). C-F139 checks their coverage wherever the task's docs place them.
The plan doesn't name the refusal for window_seconds ≤ 0, but its default implies one. C-F141 arm (h) covers it.
The known answer uses the concatenated-voices variant rather than a Music 3 duet, so it runs without an LLM or music generation.
The previous_result:transcribe case waits for transcribe_audio discards the timestamps Whisper already returns past 30 s; add opt-in segment/word timestamps #483, as the plan says.

docs: #485 - attribute_voices design record

Close-out: design record filed. Lead: claude-opus-5-5 (anthropic), close-out session 2026-09-27.

Both stages are closed and verified:

#485 stage A: attribute_voices task + assessment flag + demucs dependency #494 (stage A): the attribute_voices task, the assessment flag and the demucs dependency. It bounced twice: a real fix on the first, and suite defects harnest#20 and sd_latents prompt weighting #21 on the second.
#485 stage B: minimax-music3 skill note naming attribute_voices #495 (stage B): the minimax-music3 skill note. No bounces.

The design record is docs/proposals/complete/attribute-voices-complete.md (and there's an entry in todo.md). It went in as a docs-only commit, merged to develop @ 2b3f748, with no deploy.

verify: run every case whose source: names one of this feature's stages, and close the parent if they all pass.

Verified: feature parent close-out by claude-opus-5-5 via anthropic (tester), against lem on develop @ 2b3f748, workspace regression-complete.

I ran every case whose source: names stage #494 or #495, over MCP:

C-F137 pass.
- list_tasks has attribute_voices, and its assessment is the three probes.
- get_task shows audio / voices / lines / windows / window_seconds (2.0) / min_reference_seconds (3.0) / separate / device, and no model argument.
- application/json validates. text/plain and audio/wav are refused at steps[0].result.
C-F138 passes, except shot_join.
- Priya/HAL in-ref and held-out lines are all attributed correctly (held-out margins 0.366 / 0.477), and none is uncertain. The bed-only line is null and uncertain, with a reason.
- shot_priya and shot_hal are 1.0, and shot_bed is null and uncertain.
- reference_similarity is 0.055, with no warning.
- attr_alt_shape and attr_clip_refs agree with attr.
- shot_join came back priya 0.081 / hal 0.919, outside the case's 0.2–0.8. This is the known case defect harnest#20 (Priya is nearly silent from 6–7 s; its proposed edit moves the start to 4.5), which is still with Don. #485 stage A: attribute_voices task + assessment flag + demucs dependency #494's verify treated it the same way. I count it as a case defect, not a code failure.
C-F139 pass.
- Default windows: 11 contiguous, 0–20.5 s = ceil(20.5/2).
- window_seconds: 5: 5 windows.
- separate: false: separated: false, held-out lines still right, and the bed reason no longer says "after separation".
C-F140 pass. Same-voice refs give cosine 1.0 with too_similar, plus a voices_too_similar warning on the output and the job. Every line is uncertain, and the step completes.
C-F141 pass.
- Refused at validate: (a) one voice and (b) {} ("needs at least 2 voices"); (c) 2.90 s total ("voice 'short' has 2.90 s … under the 3 s minimum"); (f) hal voice and hal/../x (the name rule); (g) a missing asset, at voices.other; (h) window_seconds 0 and −1.
- (d) and (e) validate and are refused only at run time. The step fails with "voice 'other' span 0 starts at 8.000 s, past the end of the 7.453 s audio" / "…ends at 8.000 s, past the end…". The end is not clamped. The case allows this; I'm noting it because a validate-time refusal is the preferred form.
- (i) priya_2 at exactly 3.0 s ran to completion.
C-F142 pass. Rerunning (i) (rerun_job f38a2fd89525 → 8b098e1517c7) succeeded in 0.7 s. Both runs' events hold no download/fetch event, and list_downloads was empty after each (warm box, so there was no cold start).
C-F143 pass. Skill dw:minimax-music3 (plugin tree at develop):
- It says Music 3 ignores per-section singer directions.
- It says to run attribute_voices with a reference span per singer before staging lip-sync.
- It says "Never infer the singer from pitch".
- It names no nonexistent argument.
SE-F038 pass.
- Every probe (P1, P2, P3) in audio, in voice b bare, and in voice b as asset: is valid: false, with the error at steps[0].task.arguments.audio / .voices.b.
- P1 and P2 get identical wording (only the probe differs). No resolved path is echoed beyond the probe and the workspace roots list.
- run_workflow on a P1 and a P2 probe was refused outright, with no job queued.
- The control validates.

Cleanup: every job output from this session is deleted. No new case, and no pending: lines remain.

Navigation Menu

Repository navigation

idea: attribute_voices - which reference singer sings each lyric line, by timbre (pitch mislabelled 6 lip-sync shots) #485

Plan v1: `attribute_voices`, which reference voice sings each line (by timbre)

Verdict: build smaller

Problem and non-goals

The design

Stages

Stage A: the task (server)

Stage B: skill note (plugin-only)

Risks

Questions

6 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

idea: attribute_voices - which reference singer sings each lyric line, by timbre (pitch mislabelled 6 lip-sync shots) #485

Description

Activity

dkackman commented on Sep 26, 2026

Plan v1: attribute_voices, which reference voice sings each line (by timbre)

Verdict: build smaller

Problem and non-goals

The design

Stages

Stage A: the task (server)

Stage B: skill note (plugin-only)

Risks

Questions

dkackman commented on Sep 27, 2026

6 remaining items

dkackman commented on Sep 27, 2026

dkackman commented on Sep 27, 2026

dkackman commented on Sep 27, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Plan v1: `attribute_voices`, which reference voice sings each line (by timbre)