idea: lip-sync target check - is the singer's mouth the one moving (attribute_voices + get_output_frames)

Idea: a VLM probe over sampled frames for the faults that took a human viewer to catch.

Source: field report from an Opus 5.5 session on lem, 2026-09-25/26. After the frame checks and assess_output passed, user review of a 24-shot H3 musical still found:

Anatomy vs props: a crowd told to clap while holding candles rendered with three hands.
Direction of travel: "march up to the house" rendered as walking toward camera, away from the house.
Lip-sync target: with two people in frame, the wrong person's mouth moved to the vocal (6 shots). In another shot the singer's mouth barely moved.

Ask: an opt-in assess_output / analyze_shots rule that runs a VLM (the judge machinery from #119) over a few sampled frames with targeted questions:

hands and props counts per visible person;
whether the motion heads toward a named landmark;
for a shot with a vocal slice, whether the intended subject's mouth is open during windows with vocal energy.

Related to #465, which is the audio side of "assess_output passes what a viewer catches".

Plan v2: lip-sync target check, from shipped parts (#488)

Written by the lead agent, model claude-opus-5-5 via anthropic. Rescoped per Don's 2026-10-03 comment: this covers the lip-sync target only. The VLM probe, the anatomy check and the travel-direction check stay parked under v1's Q1/Q2 decisions.

Verdict: build smaller. One stage, about $3-4. Stage B (`vocal_windows`) is cut.

Answer to Don's first question: yes, the loop can already be run with what has shipped, apart from one bug and one missing convenience. Stage B is not needed. Its voiced spans would come from the mixed track and would carry no speaker identity, so they could not say who sings. attribute_voices already gives each line a voice, and with windows= it gives each shot a voice and share.

The loop today (the sweep at develop b2f4714):

transcribe_audio(timestamps="segment") returns {text, chunks:[{start, end, text}]}. Times are in seconds from the start of the file (dw/tasks/audio_transcription.py:125-136).
attribute_voices(audio, voices, lines=<those chunks>) accepts the whole transcript dict as lines (dw/tasks/voice_attribution.py:275-300, tested at tests/test_voice_attribution.py:203-207). For each line it returns {start, end, text, voice, margin, voiced_seconds, uncertain, reason}.
get_output_frames(name=<final cut>, at=[...], crop=<face>, hear=1.0):
- at is absolute time in the cut (dw/media_frames.py:189-219);
- hear returns audio centred on each tile (dw_mcp/media.py:235-266);
- a call takes at most 32 moments (dw/server/routes/media.py:293).
The caller judges each tile: is the open mouth the line's voice?

What is broken or missing in that loop:

Bug: Whisper can return end: null on the last chunk. transcribe_audio passes it through (audio_transcription.py:131), and attribute_voices._span then refuses the whole call on float(None) (voice_attribution.py:135-142). So the chain from step 1 to step 2 fails on exactly the transcripts where a song ends mid-line. This alone is a one-fix item.
No template. transcribe-audio.json has no timestamps variable and saves text/plain (workflows/templates/transcribe-audio.json:18-20). A literal timestamps requires application/json (dw/scalar_result_validation.py). No template runs attribute_voices, because idea: attribute_voices - which reference singer sings each lyric line, by timbre (pitch mislabelled 6 lip-sync shots) #485 deferred template wiring (docs/proposals/complete/attribute-voices-complete.md:92-93,104). Each session therefore has to write and validate a two-step workflow by hand.
No written loop. No doc describes a check after generation that the right mouth is moving. The doc also needs to state which timeline the line times are on (below).
Skill room. minimax-h3 has 26 B of headroom and minimax-music3 has 144 B (cap 12,288 B, tests/test_plugin_skills.py:22). Neither can hold the loop. script-to-video has about 6.4 KB.

Axes

Value. The field report shows 7 of its 9 content faults were the lip-sync target (6 shots with the wrong mouth, 1 with a barely moving mouth). This is the one fault that recurs within a single run. The gain goes to Don's sessions driving multi-singer H3 pieces. Don has already called the demand (2026-10-03), so I have not re-measured it.
Build cost. One stage, $3-4.
Inertia. Small:
- one catalog template;
- one doc subsection;
- one line in a skill;
- a tightened output contract (end is never null).
It adds no MCP tool or parameter (SURFACE_BUDGET has about 40 tokens left, tests/test_mcp_server.py:1533). It adds no syntax, no engine concept and no ML judge, so v1's non-goals all hold.
Reversibility. High. A template and a doc section can be removed without a breaking-change. Filling a null end cannot break a reader, since null was never usable.
Cheaper alternatives:
- The null-end fix alone (one implementer fix, about $2). This loses the template and the written loop, so every session rebuilds both. That was v1's diagnosis: the session looked in the wrong places.
- Doing nothing. This keeps the bug, so the obvious chain breaks.
Overlap. transcribe_audio discards the timestamps Whisper already returns past 30 s; add opt-in segment/word timestamps #483 and idea: attribute_voices - which reference singer sings each lyric line, by timbre (pitch mislabelled 6 lip-sync shots) #485 are closed and supply the parts. assess_output doesn't flag in-shot dead air: H3 dialogue shots hold 0.5-2 s at -70..-84 dBFS between lines #465 (audio-side assessment) is built and does not overlap. No open feature covers this.

Problem and non-goals

Problem. A driving session has no ready-made way to check that the mouth moving during a lyric line belongs to the singer who sings it. The parts exist, but one link fails on a null end, and nothing joins them or tells the session where to look.

Non-goals (unchanged from v1):

no verdict and no auto-regeneration;
no ML model inside assess_output;
no change to judge;
no VLM probe;
no new MCP surface;
no automatic per-shot singer wiring (idea: attribute_voices - which reference singer sings each lyric line, by timbre (pitch mislabelled 6 lip-sync shots) #485's deferral stands).

The design

1. transcribe_audio: end is never null.

When timestamps is set, a chunk whose end is None gets the clip's duration, which is already known from the 16 kHz mono array.
A None start takes the previous chunk's end, or 0.
This is the engine side only. The result shape is unchanged.
The change goes in the producer, not in attribute_voices. Every reader of the transcript then gets numbers, and _span's strict check stays as it is (Q2).

2. A catalog template, workflows/templates/attribute-lines.json.

Variables:
- audio: the song;
- voices: name → reference spans into audio, the shape that attribute_voices already takes;
- timestamps: default "segment";
- optionally the Whisper model_name, as in transcribe-audio.json.
Steps: transcribe (transcribe_audio, saving application/json), then attribute (attribute_voices with lines: "previous_result:transcribe", saving application/json).
Shape and traits follow transcribe-audio.json. Its description names the follow-up loop.
Running it costs the Whisper model plus demucs and ECAPA, which is the existing per-task cost. The known plan.downloads_required gap for demucs (idea: attribute_voices - which reference singer sings each lyric line, by timbre (pitch mislabelled 6 lip-sync shots) #485 deferred) is not fixed here.
Wiring previous_result: from a JSON step into lines has never been run end to end. That is idea: attribute_voices - which reference singer sings each lyric line, by timbre (pitch mislabelled 6 lip-sync shots) #485's last deferred item, and this stage's tests and acceptance cover it.

3. The loop, written once, as a Checking the lip-sync target subsection under docs/TASKS.md Voice attribution:

Run attribute-lines on the song. These are the same references the music3 skill already has a session pick before casting.
For each line with a non-null voice, take frames at start + 0.3 s and at the line's midpoint. The first moment avoids a breath before the line.
Ask for those frames with one get_output_frames call per face framing, because one call takes one crop. Use hear: 1.0 and at most 32 moments per call.
Look at whose mouth is open. Lines with uncertain (ensembles) or a null voice are skipped and named.
Timelines, stated plainly:
- minimax/music-video.json lays the song over the cut from t = 0 (slice_audio at start_frame, concat_videos with trim_frames: 0, pair_audio fit: video). Song time equals cut time unless trim_frames > 0.
- A musical joined with join_into_song places the song at the end of the dialogue less cue_seconds. The offset is the first sung shot's start_frame / fps − cue_seconds (dw/tasks/join_into_song.py:132-142), added to every line time.
- For any other assembly, transcribe the final cut's own track instead. Expect weaker attribution there, because dialogue is ducked under the song.

4. One line in plugins/dw/skills/script-to-video/SKILL.md (Q3), pointing to that subsection as the review step for a sung multi-shot piece. minimax-h3 and minimax-music3 are not touched, since they have no room left.

Surfaces:

engine: the transcribe_audio output fix;
catalog: one template;
MCP, REST and syntax: none;
plugin: one skill line, with a plugin version bump.

Stages

Stage A, the only stage (server + plugin; about $3-4).

Builds: design items 1-4.
Tests:
- a unit test that an open-ended last chunk gets the clip's duration, and that a None start is filled;
- the template validates and appears in list_workflows;
- a test that the template's previous_result:transcribe resolves into attribute_voices' lines (mocked models, following tests/test_voice_attribution.py);
- the catalog and plugin pins that a new template and a skill edit trip (tests/test_plugin_skills.py, the catalog tests).
Docs:
- the docs/TASKS.md subsection;
- a note in the Transcription section of docs/TASKS.md that end is always a number;
- the template's entry wherever the catalog lists templates.
Deploy: server restart on lem, plus a plugin update.
Acceptance intent (over MCP):
- list_workflows(shape=...) lists attribute-lines, and validate_workflow on it is clean.
- Running it on a two-singer song, with a reference span per singer, completes. get_output_text on the attribute result returns JSON in which every line has numeric start and end and a voice that is one of the two names (or null with a reason). The lines match the known singers of that song (the idea: attribute_voices - which reference singer sings each lyric line, by timbre (pitch mislabelled 6 lip-sync shots) #485 known-answer song serves).
- Open-ended last line: run transcribe_audio with timestamps: "segment" on a clip cut mid-word (slice_audio ending inside a sung line). Its last chunk's end is a number no greater than the clip's duration, never null. The template run on that clip completes rather than failing on the last line.
- Refusals:
  - validate_workflow on the template with voices missing reports an error naming voices;
  - a reference span shorter than min_reference_seconds is still refused, with the existing message.
- The loop: get_guide("tasks", section=<the new subsection>) returns text naming attribute-lines, get_output_frames with at/crop/hear, the 32-moment limit, and both timeline rules.
- One real round: in a workspace holding a two-singer minimax/music-video cut, take two line start + 0.3 times from the template's output as at, with hear: 1.0. get_output_frames returns tiles whose audio clips carry that line's vocal. The tester records whose mouth is open. The case passes on the tool chain working end to end, not on H3 having rendered the right mouth.
- Plugin: the script-to-video skill names the subsection. Verified by the tester reading the installed plugin.

Order: a single stage, so none. No dependency on open issues.

Risks

Whisper segments can span two singers (a duet call and response). Such a line scores between the two voices and comes back uncertain, so the loop skips it and names it rather than guessing. timestamps: "word" is the escape: it gives finer lines at the cost of noisier attribution.
start + 0.3 s is a heuristic. A held note or a late entry can still land on a closed mouth. The doc says to look at the midpoint frame too, and the caller judges.
Timeline offsets. A session that skips the join_into_song rule will check the wrong moments. The doc leads with the rule, and transcribing the final track is the fallback.
The verdict rests on the caller's eye. This is by design, per v1's evidence and idea: Ref2VA reference-leak probe - flag shots that open on the reference portraits themselves #487's evidence that cheap heuristics miss content faults.

Questions

Q1. Which audio to transcribe?
- Default: the song, mapped onto cut time by the two timeline rules. It is clean, and it is the same audio whose references the session already chose when casting.
- Alternative: always the final cut's track. That is simpler, with no offsets, but attribution is weaker under ducked dialogue. The doc would lead with that instead, and the template would take a video.
Q2. Where does the null-end fix go?
- Default: transcribe_audio, so the producer never emits null.
- Alternative: attribute_voices reads a null end as end-of-audio. That leaves other transcript readers exposed but changes nothing in transcription.
Q3. Skill pointer in script-to-video?
- Default: yes, one line, which makes the stage server + plugin.
- Alternative: the guide only, and the stage becomes server-only. The cost is that a session reaches the loop only if it searches for it.

Reply in comments and hand back with owner:lead. Approve by adding status:plan-approved yourself. No agent may set it.

Q1: yes, accept the defer. Q2: no, not running the zero-code spike now. Q3: as scoped — revisit once #483 and #485 land.

Parking with the triggers you named: the Q2 spike showing a local VLM actually answers the three questions on the la-vela shots, a recurrence of any of the three faults after #484's notes, or #483+#485 landing (makes the lip-sync-only version small).

Cross-reference from #487 (reference-leak probe, now closed not planned, folded in here): the lead built and tested the cheap structural probe (opening_jump_ratio, a hard-cut-in-first-2s heuristic) against the actual known leak in encanto-uprising before giving up on it. It doesn't work — the leak is a ~3s morph, not a cut, and scores 1.06 against a clean-shot range of 0.37-2.01, with no separating threshold. A generic thumbnail-distance measure didn't separate it either. Only the reference backdrop's specific color did, which is overfit to one light backdrop and not a generalizable rule.

This is concrete evidence that a cheap structural/geometric heuristic can't catch this fault class — it needs actual content judgment (a frame model), reinforcing this issue's own reasoning for the defer. Filing here for whenever the Q2 spike or a trigger reopens this.

Spec for plan v2, written from the plan alone (tester, claude-opus-5-5 via anthropic). Every case carries pending: #617.

#617, stage A (null-end fix, attribute-lines template, lip-sync check loop):

regression-suite-complete.md:
- C-F184: catalog listing, step wiring and clean validation of templates/attribute-lines.
- C-F185: known answer on idea: attribute_voices - which reference singer sings each lyric line, by timbre (pitch mislabelled 6 lip-sync shots) #485's duet, with segment and word runs. Times are numeric, lines equal the transcript's chunks, and each line names the right voice.
- C-F186: a clip cut mid-line gets a numeric last end ≤ the clip duration, and the template completes on that clip.
- C-F187: refusals at validate time:
  - missing voices;
  - missing audio;
  - one voice;
  - a 2.9 s reference, named, with the existing min_reference_seconds message;
  - a 3.0 s reference as the boundary control, which must validate;
  - a missing asset.
- C-F188: the tasks guide subsection Checking the lip-sync target. It checks the call shape, the 32-moment limit, both timeline rules and the fallback, plus the Transcription note that end is always a number.
- C-F189: one round of the loop on a fixture stand-in cut (ep3 shots + duet, laid from t = 0). It checks that a tile's audio carries the line's voice, and probes the 32/33-moment limit.
- C-F190: the script-to-video skill's pointer, and that it resolves in the guide.
regression-suite-model-specific.md:
- M-F073: the real minimax/music-video round. It rides on an existing two-singer render, so no new H3 spend.

Notes (non-blocking):

I split the plan's "one real round" into C-F189 and M-F073. C-F189 is a CPU-built stand-in that always runs. M-F073 rides on an existing real render. Both pass on the chain working; whose mouth is open is recorded, never asserted, as the plan says.
Whisper does not emit a null end on every clip. C-F186 asserts the contract on clips cut mid-line whether or not that take produced a null. A None start can't be forced over MCP, so it is left to the plan's unit test.
C-F187 (b), C-F187 (e) and the 33-moment arm of C-F189 are edges the plan implies rather than names. C-F189 expects the 33-moment call to be refused, not silently truncated. If the existing get_output_frames behaviour already differs, that is a pre-existing contract and the arm should be read against it.
C-F184 expects kinds to report JSON/text output. The plan doesn't state the exact kinds value, so any non-media kind meets it.

Merge branch 'docs/488-close-out' into develop (#488)

Close-out: design record made (lead agent, model claude-opus-5-5 via anthropic).

Closed: #617, stage A (the only stage), verified on lem @ ea458b26 after 1 bounce (word-mode zero-length chunks and a confident verdict on a two-singer segment; fixed in attribute_voices).

Design record: docs/proposals/complete/lip-sync-target-check-complete.md (develop @ 22d7b989, merge of f599431b). todo.md lists it under Shipped since the ranking, and attribute-voices-complete.md marks its deferred previous_result:transcribe item done. No earlier doc to retire. Still open from verify: C-F185's harnest amendment and M-F073 (no two-singer music-video render to reuse), both left pending: #617.

verify: run every case whose source: names one of this feature's stages, and close the parent if they all pass.

Verified. Tester: claude-opus-5-5 via anthropic, against lem on develop @ 22d7b98.

I ran every suite case whose source: names stage #617, over MCP this session.

Case	Result	What I ran
C-F184	PASS	`templates/attribute-lines` is listed and correctly wired (transcribe_audio → attribute_voices). Segment- and word-granularity validates are both clean.
C-F186	PASS	Every chunk `end` is numeric (tx_hal 0–11.5 s, tx_priya 0–5.0 s, tx_whole 0–5.0 s). The template on the HAL cut (job 7e0492077277) succeeded; its last line is voice `hal`, `uncertain: true`.
C-F187	PASS	(a) refused at `steps[1].task.arguments.voices`. (b) refused, asset not found. (c) refused, "needs at least 2 voices". (d) refused at 2.90 s, under the 3 s minimum. (e) valid at exactly 3.0 s. (f) refused, asset not found.
C-F188	PASS	The tasks guide has the "Checking the lip-sync target" subsection with both timeline rules (cut time unless `trim_frames > 0`; add `start_frame / fps − cue_seconds` for join_into_song) and the always-numeric `end` note.
C-F189	PASS	Built a 248-frame, 24 fps two-voice cut. Segment and word runs succeeded: L1 priya 4.38–7.26 s, L2 hal 7.42–9.2 s. `get_output_frames` crop+hear at 4 moments returned 4 tiles, each with about 1 s of WAV. 32 moments (0.5–9.83 s, crop `[0,0,960,544]`) were accepted with 32 tiles. 33 moments were refused: "`at` names 33 moments; the most is 32 - ask for a contact sheet (`count`) to see more of the clip at once". The input was not truncated.
C-F190	PASS	The pointer in script-to-video section 6 resolves to the guide section.

Not run:

C-F185 and M-F073 still carry pending: #617, per the lead's close-out.
C-F185 is waiting on its harnest amendment.
M-F073 is waiting on a two-singer render.

Notes (not failures):

C-F187(b): the refusal path names variables.audio, not the argument. That is acceptable because the variable is what carried the bad asset.
C-F189: during the hal line L2 (7.71 s, 8.29 s), the shot shows Priya with her mouth open. That is a visible mismatch, which is what the check is meant to surface. At 4.67 s, HAL's mouth is out of frame.

All runs were cleaned up with delete_output(job_id=…): 850539be2e32, 7e0492077277, f0c5cf9ae29f, 2727a36948bf, 83a06b94275f.

docs(proposals): dkackman#488 - close-out record for the lip-sync tar…

Navigation Menu

Repository navigation

idea: lip-sync target check - is the singer's mouth the one moving (attribute_voices + get_output_frames) #488

Plan v2: lip-sync target check, from shipped parts (#488)

Verdict: build smaller. One stage, about $3-4. Stage B (`vocal_windows`) is cut.

Axes

Problem and non-goals

The design

Stages

Risks

Questions

18 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

idea: lip-sync target check - is the singer's mouth the one moving (attribute_voices + get_output_frames) #488

Description

Activity

dkackman commented on Sep 27, 2026

Plan v2: lip-sync target check, from shipped parts (#488)

Verdict: build smaller. One stage, about $3-4. Stage B (vocal_windows) is cut.

Axes

Problem and non-goals

The design

Stages

Risks

Questions

dkackman commented on Sep 27, 2026

dkackman commented on Sep 27, 2026

18 remaining items

dkackman commented on Oct 5, 2026

dkackman commented on Oct 5, 2026

dkackman commented on Oct 5, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Verdict: build smaller. One stage, about $3-4. Stage B (`vocal_windows`) is cut.