Skip to content

feat: add Wan2.2 S2V (audio+img-to-video) support - #1925

Merged
leejet merged 6 commits into
leejet:masterfrom
noctrex:wan22-s2v-support
Sep 13, 2026
Merged

leejet merged 6 commits into
leejet:masterfrom
noctrex:wan22-s2v-support

Conversation

@noctrex

@noctrex noctrex commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

Summary

Audio-driven video generation for Wan2.2-S2V-14B: a reference portrait plus a driving WAV produce a lip-synced video whose output container carries the driving audio.

  • detect S2V checkpoints by audio-module key presence; uses the wan2.1 VAE
  • wav2vec2 audio encoder model class, backend module, --audio-encoder flag
  • S2V modules in the Wan DiT: causal audio encoder, motion encoder, audio injector (AdaLN + cross-attention), reference-latent tokens
  • audio windowing/bucketing to the latent frame count; uncond uses silence
  • new --audio flag; mux the driving track into avi/webm (wav sidecar)
  • skip ComfyUI fp8_scaled .scale_input keys at parse time
  • docs: downloads, S2V example, sampler and 24 GB VRAM guidance

Additional Information

Verification

wav2vec2 audio encoder parity (vs ComfyUI torch reference):

  • Final hidden state max abs error 0.0082 against the fp32 torch reference, inside the torch-fp16-vs-fp32 envelope (torch's own fp16 execution error: 0.0221).
    Per-layer error ratio 0.35-0.94 across layers, max 1.02 at layers 22/23.
    Includes the grouped positional conv (weight-norm, drop-last) and stable-LN conv frontend.
    A deliberately broken negative control fails the gate (0.547 vs 0.02 tolerance), so the harness is not vacuous.

End-to-end (RX 7900 XTX 24 GB, ROCm, GGML_HIP_GRAPHS=OFF, 640x368, 13 frames):

checkpoint VRAM s/it
wan2.2_s2v_14B_int8_convrot.safetensors 17.6 GB 5.05
wan2.2_s2v-14B-Q8_0.gguf 18.4 GB 6.19
wan2.2_s2v-14B-Q4_K_M.gguf 13.4 GB 10.99
  • Full 77-frame run (int8_convrot + lightx2v low-noise lightning LoRA, 4 steps, cfg 1.0): ~164 s wall including ~50 s VAE decode.
  • Baseline quality run (int8_convrot, 20 steps, cfg 6.0, no LoRA, 33 frames): 14.3 s/it.
  • Output audio muxing validated end-to-end on ROCm and Vulkan (driving track copied and truncated to video duration, e.g. 236147 -> 13000 samples for a 13-frame clip, muxed into .avi).

Note: int8_convrot only works on CUDA out of the box with the current ggml revision.
ROCm and Vulkan support comes from my ggml PRs (leejet/ggml#6-#9, currently open).
The ROCm/Vulkan numbers above were measured with that stack applied. GGUF checkpoints (Q8_0 / Q4_K_M) are unaffected by any of this.

Regression (existing Wan models unchanged):

  • Wan2.1 T2V 14B (Q8_0 GGUF, ROCm)
  • Wan2.1 I2V 14B 480p (Q8_0 GGUF + clip_vision, Vulkan)
  • Wan2.2 TI2V 5B T2V + I2V (fp16, 48-ch VAE, ROCm)
  • Wan2.2 T2V A14B + I2V A14B (int8_convrot MoE, ROCm)

All report the correct Version: lines and never detect as S2V.

Known limitations

  • One generation covers the first S2V chunk window (--video-frames, 77-81 frames at 16 fps); long-video chunked extend mode is not implemented yet.
  • bf16 and fp8_scaled checkpoints do not fit 24 GB GPUs. int8_convrot, Q8_0, Q4_K_M tested ok.
  • On ROCm, tiled VAE decode of longer clips currently needs GGML_HIP_GRAPHS=OFF (HIP graph capture issue for that graph shape, not specific to this PR's code paths).

Sample outputs

77 frames, 512x512, 56.86s/it, ROCm, generate_video completed in 1176.76s

s2v_test_q22_h264.mp4

Checklist

LLM Disclosure: Yes, used GLM-5.3-Flash

@noctrex noctrex changed the title feat: add Wan2.2 S2V (speech-to-video) support feat: add Wan2.2 S2V (audio+img-to-video) support Sep 8, 2026
Audio-driven video generation for Wan2.2-S2V-14B: a reference portrait
plus a driving WAV produce a lip-synced video whose output container
carries the driving audio.

- detect S2V checkpoints by audio-module key presence; uses the wan2.1 VAE
- wav2vec2 audio encoder model class, backend module, --audio-encoder flag
- S2V modules in the Wan DiT: causal audio encoder, motion encoder,
  audio injector (AdaLN + cross-attention), reference-latent tokens
- audio windowing/bucketing to the latent frame count; uncond uses silence
- new --audio flag; mux the driving track into avi/webm (wav sidecar)
- skip ComfyUI fp8_scaled .scale_input keys at parse time
- docs: downloads, S2V example, sampler and 24 GB VRAM guidance
@leejet
leejet merged commit 0bd72f0 into leejet:master Sep 13, 2026
9 checks passed
@leejet

leejet commented Sep 13, 2026

Copy link
Copy Markdown
Owner

Thank you for your contribution.

@noctrex

noctrex commented Sep 13, 2026

Copy link
Copy Markdown
Contributor Author

Thank you for the amazing program

danielhanchen added a commit to unslothai/stable-diffusion.cpp that referenced this pull request Sep 21, 2026
* fix: preserve "token_refiner" token for MiniMax H3 LoRAs (leejet#1864)

* fix: fail with a message when MiniMax-H3 is run in img_gen mode (leejet#1863)

* feat: support INT8 ConvRot safetensors (leejet#1857)

* fix: replace free_compute_buffer with runner_done in vae (leejet#1872)

* sync: update ggml (leejet#1873)

* fix(ci): trigger builds for ggml updates

* feat: add taeh3 support (leejet#1874)

* fix: prevent gallocr hash overflow in tiny graph-cut segments (leejet#1880)

* fix: re-clamp streaming VRAM budget to currently free memory (leejet#1878)

* fix: mark graph cuts with both a prefix and a suffix (leejet#1883)

* fix: make max_order of lms sampler configurable (leejet#1885)

* fix: guard against missing sampler/scheduler names (leejet#1887)

* chore: format code

* fix: use sd_get_preview_interval() (leejet#1907)

* feat: configurable image / video compression (leejet#1909)

* feat: support standard Qwen3-VL weights for MiniMax-H3 (leejet#1910)

* feat: load scaled FP8 weights without upfront conversion (leejet#1913)

* fix: match exact weights in LLM config detection (leejet#1923)

* feat: add LTX-2.5 support (leejet#1893)

Co-authored-by: leejet <leejet714@gmail.com>

* fix: correct MiniMax H3 reference audio encoding (leejet#1886)

* fix: correct MiniMax H3 audio Euler steps (leejet#1908)

* feat: use backend-native FP8 matmul when supported (leejet#1916)

* sync: update ggml

* feat: additional `--preview-interval` values (leejet#1915)

Co-authored-by: leejet <leejet714@gmail.com>

* feat: support numbering for preview images (leejet#1895)

* fix: use carrier sampling for MiniMax H3 audio (leejet#1924)

* feat: generalize temporal tiling across video VAEs (leejet#1926)

* feat: prefetch streamed layers during compute (leejet#1905)

Co-authored-by: leejet <leejet714@gmail.com>

* refactor: unify runner lifecycles and weight residency (leejet#1940)

* feat: add verbose logging and log-level selection (leejet#1941)

* feat: enable single-GPU auto-fit with tiered parameter placement (leejet#1942)

* fix: reuse graph cut plans across CFG passes (leejet#1943)

* refactor: split ggml extensions and move implementations to cpp files (leejet#1945)

* fix: preserve K-quantized embedding weights (leejet#1936)

* fix: correct SDXL embeddings loading (leejet#1939)

* refactor: unify model source and weight lifecycle management (leejet#1956)

* docs: reflect GGML_MAX_NAME value change in rpc docs (and in ggml_extend assert) (leejet#1950)

* refactor: split generation pipeline out of stable-diffusion.cpp (leejet#1957)

* fix: enable VAE decode tiling fallback without auto-fit (leejet#1932)

* fix: preserve BF16 embedding weights for get_rows (leejet#1959)

* fix: handle invalid option numbers (leejet#1961)

* feat: expose the loaded model version name through the public API (leejet#1962)

* feat: add SenseNova U1.5 support (leejet#1935)

* fix: reuse graph plans when scale parameters change (leejet#1963)

* feat: add linear and attention scale overrides (leejet#1964)

* feat: preserve explicit backend assignments during auto-fit (leejet#1967)

* fix: guard GPU memory capacity and propagate encoding failures (leejet#1958)

* fix: bound plain-text runs in parse_prompt_attention regex (leejet#1919)

* feat: add Wan2.2 S2V (audio+img-to-video) support (leejet#1925)

Co-authored-by: leejet <leejet714@gmail.com>

* fix: validate vision projector output dim against LLM hidden size (leejet#1918)

* fix: resolve MSVC narrowing conversion warnings (leejet#1969)

* feat: Add generation parameters into video metadata (leejet#1901)

Co-authored-by: leejet <leejet714@gmail.com>

* feat: support external Hugging Face tokenizer JSON files (leejet#1973)

* refactor: require external Gemma 2 and GPT-OSS tokenizers (leejet#1974)

* feat: support Brownian tree noise in all noise injection samplers (leejet#1899)

* fix: use tokenizer-specific pre-tokenization rules (leejet#1975)

* fix: remove vision_model. from ununsed tensors (leejet#1983)

* perf: eliminate temporary allocations in Philox rounds (leejet#1982)

* fix: honor flash attention flag in LLM text encoder attention (leejet#1987)

* refactor: remove obsolete unused tensor filtering (leejet#1984)

* perf: pad small attention heads to 64 for MMA Flash Attention (leejet#1992)

* perf: update ggml for faster direct convolutions (leejet#1993)

* perf: accelerate VAE direct 3D convolutions (leejet#1996)

* fix: propagate CUDA driver dependency to shared library consumers

* fix: prevent clip_preprocess center crop from exceeding the resized image (leejet#1995)

* perf: reduce CPU overhead in graph execution and sampling (leejet#1997)

* perf: parallelize host tensor elementwise and broadcast ops (leejet#1998)

* feat: support building with upstream ggml (leejet#1999)

* feat: add Qwen Image 2.1 support (leejet#1994)

* feat: restore legacy fp8 handling when building with upstream ggml (leejet#2001)

* fix: avoid passing ggml logs as format strings (leejet#2002)

* feat: add LLaDA-Image support (leejet#1968)

Co-authored-by: leejet <leejet714@gmail.com>

* feat: add native CUDA SageAttention support (leejet#2005)

* fix: avoid narrowing conversion in SigVQ patch embedding and format code

* docs: update CONTRIBUTING.md

---------

Co-authored-by: stduhpf <stephduh@live.fr>
Co-authored-by: leejet <leejet714@gmail.com>
Co-authored-by: LostRuins Concedo <39025047+LostRuins@users.noreply.github.com>
Co-authored-by: fszontagh <51741446+fszontagh@users.noreply.github.com>
Co-authored-by: Wagner Bruna <wbruna@users.noreply.github.com>
Co-authored-by: vmobilis <75476228+vmobilis@users.noreply.github.com>
Co-authored-by: Piotr Wilkin (ilintar) <ilintar@gmail.com>
Co-authored-by: jk212h20 <101200018+jk212h20@users.noreply.github.com>
Co-authored-by: assouan <750048+assouan@users.noreply.github.com>
Co-authored-by: nan <zjn32202153@gmail.com>
Co-authored-by: Hmission <62598659+Hmission@users.noreply.github.com>
Co-authored-by: LED-M <105789115+xledx@users.noreply.github.com>
Co-authored-by: Maphist0 <28743569+Maphist0@users.noreply.github.com>
Co-authored-by: George <35490284+noctrex@users.noreply.github.com>
Co-authored-by: Санька Четвёртый <CAHbKA-IV@mail.ru>
Co-authored-by: Lin Xuhao <linxuhao84@gmail.com>
Co-authored-by: Fabrice Aneche <akhenakh@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants