Repository navigation
Test lightx2v's 4-step Ref2VA turbo LoRA (v0.1) for music-video renders #824
Description
Field report
Lip-synced music videos use Ref2VA, and those renders ran about 1 h per card at 9 steps on the 8-step Ref2VA turbo LoRA. lightx2v ships minimax_h3_ref2v_turbo_4step_v0.1_bf16.safetensors, but the dw docs only say there is "no tested 4-step Ref2VA file" (plugins/dw/skills/minimax-h3/references/checkpoints-and-loras.md:16). We have known the file exists since the 2026-09-07 H3 audit (docs/proposals/audits/2026-09-07-minimax-h3-audit.md:240).
Proposal
- A/B on lem: the same Ref2VA job (124 frames, reference + audio) on the 8-step v1.0 LoRA at 9 steps vs the 4-step v0.1 at 5 steps. Compare wall clock, faces, lip-sync and identity hold.
- Use eval: H3 community LoRAs - Realism People stacked wins; 4-step v1.2 viable draft; Acc/HyperFlow/FastH3 need loaders #585 for comparison: the 4-step FL2VA v1.2 was ~25% faster with good faces. A v0.1 is more likely to fall short on quality.
- Check separately whether the field renders used
sage_hubattention. The 2026-10-10 A/B measured ~28% per step on H3, which is a safer speed-up available now. - If v0.1 doesn't hold up, write that in the skill reference and watch lightx2v for a v1.x.
Outcome
A catalog LoRA entry (loras/minimax-h3/turbo-4step-reference.json) with status and evidence, plus an updated skill line, or a recorded "tested, not good enough".
Activity
dkackman commentedon Oct 12, 2026
Disposition: one fix for the implementer (lead design session,
claude-opus-5-5viaanthropic).This is an eval like #585, which went implementer → tester. It needs no engine, syntax, MCP or REST change.
music-video.json(and every otherref2vatemplate) already exposeslora_weight_name,video_shift,audio_shiftandnum_inference_stepsas variables (workflows/templates/minimax/music-video.json:86-93), so the A/B needs arguments only. Not a duplicate: #149 added the 8-step Ref2VA file, and #585 covered the FL2VA files only.Scope
templates/minimax/music-video(or a single Ref2VA shot of it): 124 frames, the same reference image(s), audio, prompt and seed.minimax_h3_ref2v_turbo_8step_v1.0_768p_bf16.safetensors, 9 steps, shift 12/3 (the current default).minimax_h3_ref2v_turbo_4step_v0.1_bf16.safetensors, 5 steps. Read the file's safetensors header for alpha and training canvas and its HF model card for shift before running, and never state an alpha the header doesn't record. If the card gives no shift, try 12/3 first and then 6/3.get_job_events, and judge faces, lip-sync and identity hold against the reference withget_output_framesandget_output_audio. A warning: the 544p FL2VA 4-step v0.1 rendered noise (checkpoints-and-loras.md:17). If B is noise at both shifts, stop there and record it.music-video.jsonhas noattention_backendvariable. Useget_job_workflowon Don's recent music-video jobs (notqa-*/regression-*workspaces) to find which backend they ran, and report it on the issue. Don't change the template here. If the field runs used native attention, say that the measured ~28%/step from sage_hub (2026-10-10 A/B) is the safer win, and offer it as a separate idea.loras/minimax-h3/turbo-4step-reference.jsonmodelled onturbo-reference.json. Useworkflow: "ref2va"andstatus: "trial"(one prompt is not "proven"), with the job ids as evidence, and update the Ref2VA sentence inplugins/dw/skills/minimax-h3/references/checkpoints-and-loras.md:15-16. Keep the 8-step file as every template's default: changing templates is a separate decision.status: "rejected"and the reason, solist_lorassays why. Change the skill line to "4-step Ref2VA v0.1 tested, not good enough (Test lightx2v's 4-step Ref2VA turbo LoRA (v0.1) for music-video renders #824)".tests/test_plugin_skills.pyand any catalog-schema tests green.Deploy path: catalog + plugin only, with no server restart.
How the tester sees it work:
list_lorasreturns the new entry withworkflow: ref2vaand the stated status and evidence. The serveddw:minimax-h3skill's checkpoint paragraph names the outcome. Both cited jobs exist and their frames bear out the judgement.validate_workflowonmusic-videowith the newlora_weight_nameraises no partition warning (B's file is a ref2v file).Estimate: ~$3–5 of session, plus about 2 h of lem GPU time (two ~1 h renders, less if B is quick).
feat(loras): #824 - record the 4-step Ref2VA v0.1 turbo LoRA as a trial
Merge feat/824-4step-ref2va (#824)
dkackman commentedon Oct 12, 2026
Worked as
claude-sonnet-5-5(anthropic). Merged todevelopas 6f47153, 2026-10-12T00:33:33Z. Catalog + plugin only, no server restart.A/B (same 124-frame Ref2VA shot, 960x544, reference image + audio, seed 11, int4 sdnq, shift 12/3):
Denoise down ~42%; per-step cost unchanged. The jobs ran in parallel on two cards, so the timings are approximate.
Quality: B is a clean render, not noise. The identity holds against the reference and the mouth moves. The framing differs from A: B is tighter, with a flag in place of A's crowd. Judged from the contact sheet and a 2.0s close-up only. I did not audition the audio, so lip-sync is a visual read of the mouth. One shot, one seed.
Header / card: B's safetensors header records
alpha: 8,floating_dtype: bfloat16, and no training canvas. The HF model card names only the FL2V 8-step 768p file and gives no shift. 12/3 rendered fine, so I did not try 6/3.Shipped:
loras/minimax-h3/turbo-4step-reference.json(status: trial,workflow: ref2va, job ids as evidence) and the Ref2VA sentence inplugins/dw/skills/minimax-h3/references/checkpoints-and-loras.md. The 8-step file stays every template's default; the template is unchanged. Catalog-schema tests pass.tests/test_plugin_skills.pyhas 5 Kandinsky6 failures that also fail ondevelopwithout this change.Tester: confirm
list_lorasshows the new entry (lem's checkout may need the next deploy to see the file). Then a spot-check: validatetemplates/minimax/music-videowithlora_weight_name=minimax_h3_ref2v_turbo_4step_v0.1_bf16.safetensorsandnum_inference_steps=5, expecting no partition warning.Attention backend: the field music-video jobs (bd2751a06863, f57e3e100260, e996a3b9a8bc) set no
attention_backend, so they ran native attention. Separate idea, not done: offer sage_hub's measured ~28%/step (2026-10-10 A/B) as a template option.No regression case proposed.
dkackman commentedon Oct 12, 2026
Architecture review queued by the loop driver: this hand-off changed
dw/ordw_mcp/ondevelop(3f9eaca352..68ba49b77d: dw/disk_caches.py dw/quantized_cache.py dw/server/jobs.py dw/server/routes/system.py dw/tasks/cuts.py). The reviewer checks it againstdocs/ARCHITECTURE.mdbefore the tester verifies.dkackman commentedon Oct 12, 2026
Architecture review: PASS (reviewer, claude-opus-5-5 via anthropic). Read the #824 commits in 3f9eaca..68ba49b (ca730fd and its merge 6f47153; the merge diff vs its first parent is the same) against docs/ARCHITECTURE.md and stage-c-guardrails.md item 3. #824 touches no file under dw/ or dw_mcp/. It adds only catalog data (loras/minimax-h3/turbo-4step-reference.json) and plugin skill prose (plugins/dw/skills/minimax-h3/references/checkpoints-and-loras.md). The dw/ files that triggered this review (disk_caches.py, quantized_cache.py, server/jobs.py, server/routes/system.py, tasks/cuts.py) belong to #838-#843 and are out of scope here. Second owner: none. The entry is data read by the owner in the LoRA catalog matching row (dw/lora_catalog.py), with an exact base_models repo id and workflow ref2va. Build vs buy: n/a, because the change adds no code. Map: no new owner, so no row is needed. Context: no CLAUDE.md change. Nothing waived (no arch-approved label). Continues to the tester.
dkackman commentedon Oct 12, 2026
Verified (tester,
claude-opus-5-5viaanthropic), against lemdevelop@ 68ba49b.What I ran over MCP:
list_loras(model="MiniMaxAI/MiniMax-H3", workflow="ref2va", status="trial")returnsminimax-h3/turbo-4step-reference. It hasweight_name: minimax_h3_ref2v_turbo_4step_v0.1_bf16.safetensors,workflow: ref2vaandstatus: trial. Itsuse_whensays to pair it with 5 steps and shift 12/3 and to keep the 8-step file as the default. The evidence cites jobsec9f5b3b92e5(B) andd65cb4586b9c(baseline A).get_jobon both: eachsucceededwith one 124-framefinalmp4, on cuda:1 and cuda:0. B's arguments arelora_weight_name=…4step_v0.1…,num_inference_steps=5.get_output_frames(count=4)on B's mp4 (124 frames, 24 fps): a clean render, not noise. The subject's face and costume stay the same across the frames at 0.0, 1.7, 3.4 and 5.1s, and the mouth moves. I didn't audition the audio, the same limit as the hand-off.validate_workflow(name="templates/minimax/music-video", arguments={lora_weight_name: "minimax_h3_ref2v_turbo_4step_v0.1_bf16.safetensors", num_inference_steps: 5})returnsvalid: truewith no warnings. There's no partition warning.templates/minimax/first-and-last-frame(fl2va) are refused atarguments.lora_weight_namewith thetransformer_refpartition message. That's correct.dw:minimax-h3skill'sreferences/checkpoints-and-loras.md:16-17now reads: "4-step Ref2VA v0.1 (…, 5 steps, shift 12/3) is a trial (Test lightx2v's 4-step Ref2VA turbo LoRA (v0.1) for music-video renders #824): clean on one shot, ~42% less denoise time, not yet a default."Side note, not part of this fix:
list_loras(workflow="ref2va")withoutmodelreturns the whole catalog (flux, z-image, …, about 72 KB). It filters only whenmodelis given too. That might be intended, becausemodelresolves the partition, but the schema doesn't say so.No regression case added: this is catalog data, not behavior that's likely to regress.