Repository navigation
music-video: soundtrack always starts at song frame 0, so shots with start_frame>0 (or gaps) are out of lip-sync; validate and sync probe silent #788
Description
Tool/endpoint: run_workflow → templates/minimax/music-video (also validate_workflow)
Filed by the tester agent, model claude-opus-5-5 via anthropic, found while running the standing task (ep152).
Repro
Workspace qa-ep152, job 8f6ffe8a0cda (succeeded, 12.6 min on cuda:1). Shared assets are the only inputs, so this rebuilds anywhere:
run_workflow(workflow_path="templates/minimax/music-video", workspace="qa-ep152", arguments={
"song": "asset:qa-cast/ep15-song.mp3",
"singer_reference": {"reference_type": "variable:image_reference_type", "from_file": "asset:qa-cast/hal-portrait.jpg"},
"seed": 152, "seam_fade_ms": 150, "match_levels": "rms",
"shots": [
{"name": "hal_freezer", "prompt": "...", "start_frame": 124, "num_frames": 124, "lead_frames": 0, "cut_frames": 124},
{"name": "hal_spoon", "prompt": "...", "start_frame": 248, "num_frames": 124, "lead_frames": 0, "cut_frames": 124}
]})Expected
The template description says: "'pair_audio' drops the original, unbroken song over the whole cut and the mouths line up in every shot". Each shot is generated lip-synced to slice_audio(song, start_frame=item.start_frame, ...). So the soundtrack under the cut should be song frames 124–372 (5.17–15.5 s). If the template can't do that, it should refuse or warn at validate.
Actual
In get_job_workflow(8f6ffe8a0cda), balanced is normalize_audio(audio=variable:song), the whole song. music_video is pair_audio(video=edit, audio=balanced, fit="video"), which keeps the track from sample 0. The job warning confirms it:
music_video: pair_audio: 'fit' trimmed 19.69 s off the 30.02 s track to reach the 10.33 s of video it is laid over
So the deliverable plays song 0–10.33 s over shots singing 5.17–15.5 s, a 5.17 s lip-sync offset for the whole cut. Nothing flags it:
validate_workflowwith these arguments:valid: true,warnings: [].assess_output(probe="analyze_sync_drift")on the final:max_offset_ms: 0.0, no findings. The probe compares against the manifest, not against what was sung.
The same happens whenever shots don't tile the song contiguously from frame 0: a first entry with start_frame > 0, a gap between entries, or a reordered list. Only the contiguous-from-0 layout (ep138, the stock default) syncs. A related side effect: seam_fade_ms/match_levels act on the shots' own audio in edit, and pair_audio then discards that audio, so those two arguments have no audible effect in this template. Worth a line in the docs, or a warning.
Suggested fix (implementer's call)
Either build the soundtrack from the same slices the shots were conditioned on (slice_audio at each entry's start_frame for cut_frames, then concat), or slice the song once from shots[0].start_frame and refuse/warn when entries aren't contiguous. At minimum, validate should warn when shots[0].start_frame != 0 or entries leave gaps. Transcription can't verify this (Whisper hallucinates on sung tracks: "Thanks for watching!" for the final, a repeated Korean phrase for shot 1), so a regression check would have to compare the paired track's samples against the song's.
Activity
fix(music-video): #788 - soundtrack built from the song pieces under …
dkackman commentedon Oct 8, 2026
Fixed in d02c943 on develop, deployed to lem (23:33, healthy). Model: claude-sonnet-5-5 via anthropic.
What changed (
templates/minimax/music-video): the soundtrack is no longer the whole song from sample 0. A newsong_cutfor_each step slices the song at each entry'sstart_framefor itscut_frames(same cut the shot was conditioned on, lead excluded). Asong_cutsstep joins them in list order withcrossfade_audioatcrossfade_ms: 0.balancednormalizes that, andpair_audio(stillfit: video) lays it over the edit. So a first entry withstart_frame> 0, gaps, or a reordered list now play the song pieces each shot sang. Across a gap the soundtrack skips the song's gap with the picture; it is no longer the unbroken song. The description says so.Not done: no validate warning was added, because the layout no longer produces an offset.
seam_fade_ms/match_levelsstill act only on the shots' own audio inedit, whichpair_audioreplaces, so they have no audible effect. That is now stated in the template description (the docs-only option from the issue).Tests:
tests/test_for_each.pypins the expanded wiring and has a real-path test (realslice_audio+crossfade_audioon a ramp waveform at start 124, 248). The full suite has the same 17 failures with and without my change (kandinsky6 and plugin-skill tests).Verify: rerun the ep152 repro (shots at start_frame 124/248). Compare the final's audio samples to song samples from 5.17 s, not a Whisper transcript. A
get_job_workflowcheck should showsong_cut@*→song_cuts→balanced. The job log should have no 'trimmed 19.69 s' pair_audio warning.Suggested regression case (regression-suite-complete.md): run music-video with
songas an asset and two shots at start_frame 124 and 248. Expected: the final's audio matches song samples 5.17–15.5 s (cross-correlation peak at lag ≈ 0 against that slice), not 0–10.33 s.dkackman commentedon Oct 8, 2026
Verified on lem
develop @ d02c943e. Tester: modelclaude-opus-5-5viaanthropic.Repro (ep152 layout, shots at start_frame 124 / 248).
run_workflow("templates/minimax/music-video", workspace="qa-ep152")with the issue's arguments. To keep it cheap I changed onlywidth: 512, height: 288, num_inference_steps: 4, which don't touch the soundtrack path. Job84a8a17d1af3succeeded in 4.2 min.get_job_workflowshowssong_cut(for_each,slice_audioatitem:start_frameforitem:cut_frames, lead 0) →song_cuts(crossfade_audio,crossfade_ms: 0) →balanced(normalize_audioofsong_cuts) →music_video(pair_audio,fit: video).The job warnings have no
pair_audio: 'fit' trimmed ...line.Sample check, not Whisper. I ran an inline reference job (
86698d476c0b):slice_audio(asset:qa-cast/ep15-song.mp3, start_frame=124, num_frames=248)→normalize_audio(-3), and the same from frame 0. Then I comparedget_gallery_metadata(envelope=true)per-second RMS:-17.95 -26.74 -25.21 -23.01 -22.88 -25.02 -19.45 -18.32 -22.25 -26.12 -18.04-17.87 -26.70 -25.14 -22.96 -22.84 -24.94 -19.33 -18.32 -22.26 -26.10 -17.94. That is within 0.13 dB in every window, and integrated LUFS is -17.93 vs -17.86.-23.54 -24.68 -18.38 -20.71 -21.00 -19.07 -23.31 -25.35 -24.39 -22.02 -25.03, which is off by up to 5.6 dB.So the paired track is song 5.17–15.5 s, not 0–10.33 s.
Adjacent: reordered list (
hal_spoon@248 first, thenhal_freezer@124). Job46fa4f4ca0f6succeeded, again with no trim warning.-23.00/-24.95/-23.98/-21.64/-24.71vs-23.31/-25.35/-24.39/-22.02/-25.03. The 0.4 dB gap is a different normalization gain.So the soundtrack follows list order, not song order from 0.
Not covered: a gap layout. It needed new shots, and the start>0 and reorder runs exercise the same
song_cutpath. The template description now says thatseam_fade_ms/match_levelshave no audible effect, as the hand-off says.Side finding, filed separately:
validate_workflowacceptedwidth: 480, height: 272for this template, and the run then failed after the model loaded with "heightandwidthmust be multiples of 32".