Repository navigation
Kandinsky6 #14949
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Kandinsky6 #14949
Changes from all commits
Commits
Show all changes
65 commits
Select commit
Hold shift + click to select a range
81f3b12
implement Kandinsky 6
c892279
update docs for Kandinsky 6
6b8a080
remove redundant bigvgan code and manual checkpoint loading inside model
bf4158a
add missing _no_split_modules
1f1e6ee
use attention processor in SR
efa2b56
fix unguarded dependencies
3dc5959
remove dead manual weight loading code; flattent MMAudioVAE structure
d5a34b6
move attention processors to transformer files
334d639
remove duplicate code in K6 latent_upscaler
9d7dc71
address 1st batch of comments
leffff 0eb5211
patch transformer k6
leffff c3012d0
patch SR
leffff 1f7f234
patch pipeline style
leffff 01bcb64
replace TimestepEmbedding and fix batching
leffff 8a57f57
refactor ti2va dit and pipeline
leffff d5f8e58
refactor piflow scheduler
leffff 7038c5a
refactor SR
leffff 089a5ae
patch sr transformer autocast
leffff 3cc8f88
major refactor
leffff 65b00fd
untrack notebooks directory
leffff 66a0d44
patch mmaudio VAE
leffff 0ec1bd1
patch mmaudio VAE
leffff c17252c
fix inits
leffff d6e1f65
fix flex attn
leffff b558b74
refactoring
leffff 9debf90
remove __all__
leffff 1cf55a8
separate MMAudio VAE and Vocoder
leffff 525281e
fix naming
leffff 695caec
refactor checkpointing, magcache, transformer, sr vae for K6
leffff 3d9c95c
refactor
leffff 002f4fb
address apply_scale_shift_norm problem
leffff 0e5c599
update docs
leffff 40fe659
apply style chanes
leffff 28c5326
address review feedback on transformer, SR VAE, and flex attention
leffff 372f290
address comments
leffff cc703db
fix critical bugs
leffff 9ae71f0
Revert checkpoint-breaking module renames from 28c5326da
leffff a7e596c
Revert checkpoint-breaking latent-upscaler renames from 372f29088
leffff 8fadd97
move vocoder and LU
leffff 0b2ca95
add coppied from resolve i2v cond mode
leffff 6f035c4
rename variables
leffff 1cfe98c
fix lu rename bug
leffff f16156a
Update src/diffusers/pipelines/kandinsky6/pipeline_kandinsky6_ti2va.py
leffff 348c341
Update src/diffusers/models/transformers/transformer_kandinsky6.py
leffff 0ef584f
address yiyixuxu's comments
leffff dcbe7b0
Merge remote-tracking branch 'origin/Kandinsky6' into Kandinsky6
leffff 4c0ad12
fix audio channels bug
leffff 1f9b471
fix tests
leffff 60cad43
Fix CI failures on the Kandinsky6 PR
leffff 35e6cc4
Avoid staging LFS dedup race in the org push test
leffff 6d8a643
Revert "Avoid staging LFS dedup race in the org push test"
leffff 0b4439b
Revert "Fix CI failures on the Kandinsky6 PR"
leffff fee0b42
Merge branch 'main' into Kandinsky6
yiyixuxu 82f46ef
Apply suggestion from @yiyixuxu
yiyixuxu 3bc67ad
Apply suggestion from @yiyixuxu
yiyixuxu ebea53a
Apply batched suggestions from code review
yiyixuxu 82d3eb5
remove review-only files from the Kandinsky6 PR
yiyixuxu bbab023
Apply suggestion from @yiyixuxu
yiyixuxu 34f329f
restore .gitignore to match main
yiyixuxu 1ea5fb3
simplify the mono audio handling in _write_audio
yiyixuxu 871acef
remove the no-op set_attention_backend calls from the Kandinsky6 docs
yiyixuxu 9c78058
give the Kandinsky6 transformer its own output class instead of reusi…
yiyixuxu c8a0c1f
fix the stale pipeline exports and add return annotations for the Kan…
yiyixuxu 39138ba
Merge branch 'main' into Kandinsky6
sayakpaul 7245998
address feedback from @leffff
sayakpaul File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| <!--Copyright 2026 The Kandinsky Team and The HuggingFace Team. All rights reserved. | ||
|
|
||
| Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with | ||
| the License. You may obtain a copy of the License at | ||
|
|
||
| http://www.apache.org/licenses/LICENSE-2.0 | ||
|
|
||
| Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on | ||
| an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the | ||
| specific language governing permissions and limitations under the License. | ||
| --> | ||
|
|
||
| # Kandinsky 6 Transformers | ||
|
|
||
| Kandinsky 6 uses a multimodal diffusion transformer that denoises video and audio latents together for | ||
| text/image-to-video-and-audio generation, and a text-free diffusion transformer for video super-resolution. | ||
|
|
||
| ## Kandinsky6Transformer3DModel | ||
|
|
||
| The multimodal transformer used by [`Kandinsky6TI2VAPipeline`]. | ||
|
|
||
| ```python | ||
| import torch | ||
| from diffusers import Kandinsky6Transformer3DModel | ||
|
|
||
| transformer = Kandinsky6Transformer3DModel.from_pretrained( | ||
| "kandinskylab/Kandinsky-6.0-Pro-distill-5s-Diffusers", subfolder="transformer", torch_dtype=torch.bfloat16 | ||
| ) | ||
| ``` | ||
|
|
||
| [[autodoc]] Kandinsky6Transformer3DModel | ||
| - all | ||
| - forward | ||
|
|
||
| ## Kandinsky6SRTransformer3DModel | ||
|
|
||
| The text-free transformer used by [`Kandinsky6SRPipeline`] to refine one tile of the upscaled video at a time. | ||
|
|
||
| ```python | ||
| import torch | ||
| from diffusers import Kandinsky6SRTransformer3DModel | ||
|
|
||
| transformer = Kandinsky6SRTransformer3DModel.from_pretrained( | ||
| "kandinskylab/Kandinsky-6.0-VSR-distilled2steps-5s-Diffusers", subfolder="transformer", torch_dtype=torch.bfloat16 | ||
| ) | ||
| # The transformer always runs NABLA sparse attention (`nabla_threshold`, 0.8 by default) on the `flex` backend. | ||
| # Compile it, otherwise flex falls back to an eager implementation that needs far more memory at video resolutions. | ||
| transformer.compile_repeated_blocks(fullgraph=True) | ||
| ``` | ||
|
|
||
| [[autodoc]] Kandinsky6SRTransformer3DModel | ||
| - all | ||
| - forward |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,66 @@ | ||
| <!--Copyright 2026 The Kandinsky Team and The HuggingFace Team. All rights reserved. | ||
|
|
||
| Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with | ||
| the License. You may obtain a copy of the License at | ||
|
|
||
| http://www.apache.org/licenses/LICENSE-2.0 | ||
|
|
||
| Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on | ||
| an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the | ||
| specific language governing permissions and limitations under the License. | ||
| --> | ||
|
|
||
| # Kandinsky 6 VAEs | ||
|
|
||
| Kandinsky 6 uses a causal 3D K-VAE for video super-resolution and the MMAudio mel-spectrogram VAE, paired with a | ||
| separate BigVGAN [`MMAudioVocoder`], for synchronized audio generation. | ||
|
|
||
| ## Kandinsky6SRVAE | ||
|
|
||
| The causal 3D K-VAE used by [`Kandinsky6SRPipeline`]. It processes arbitrarily long videos in bounded-memory | ||
| segments while reproducing the exact output of a single, non-segmented pass. | ||
|
|
||
| ```python | ||
| import torch | ||
| from diffusers import Kandinsky6SRVAE | ||
|
|
||
| vae = Kandinsky6SRVAE.from_pretrained( | ||
| "kandinskylab/Kandinsky-6.0-VSR-5s-Diffusers", subfolder="vae", torch_dtype=torch.bfloat16 | ||
| ) | ||
| ``` | ||
|
|
||
| [[autodoc]] Kandinsky6SRVAE | ||
| - encode | ||
| - decode | ||
| - all | ||
|
|
||
| ## MMAudioVAE | ||
|
|
||
| The mel-spectrogram VAE used by [`Kandinsky6TI2VAPipeline`] when `sample_audio=True`. Its `decode` output is a mel | ||
| spectrogram; pass it through [`MMAudioVocoder`] to get a waveform. | ||
|
|
||
| The reference implementation can be found at [hkchengrex/MMAudio](https://github.com/hkchengrex/MMAudio) (MIT | ||
| license). | ||
|
|
||
| ```python | ||
| import torch | ||
| from diffusers import MMAudioVAE | ||
|
|
||
| audio_vae = MMAudioVAE.from_pretrained( | ||
| "kandinskylab/Kandinsky-6.0-Pro-distill-5s-Diffusers", subfolder="audio_vae", torch_dtype=torch.bfloat16 | ||
| ) | ||
| ``` | ||
|
|
||
| [[autodoc]] MMAudioVAE | ||
| - encode | ||
| - decode | ||
| - all | ||
|
|
||
| ## MMAudioVocoder | ||
|
|
||
| Adapted from the BigVGAN-v2 vocoder MMAudio bundles, itself from | ||
| [NVIDIA/BigVGAN](https://github.com/NVIDIA/BigVGAN) (MIT license), with the anti-aliased Snake activations of | ||
| [alias-free-torch](https://github.com/junjun3518/alias-free-torch) (Apache License 2.0). | ||
|
|
||
| [[autodoc]] MMAudioVocoder | ||
| - forward |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,150 @@ | ||
| <!--Copyright 2026 The Kandinsky Team and The HuggingFace Team. All rights reserved. | ||
|
|
||
| Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with | ||
| the License. You may obtain a copy of the License at | ||
|
|
||
| http://www.apache.org/licenses/LICENSE-2.0 | ||
|
|
||
| Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on | ||
| an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the | ||
| specific language governing permissions and limitations under the License. | ||
| --> | ||
|
|
||
| # Kandinsky 6 | ||
|
|
||
| Kandinsky 6 is a family of video generation models from [Kandinsky Lab](https://huggingface.co/kandinskylab). The | ||
| main model generates video and synchronized audio from text or a reference image with a single multimodal diffusion | ||
| transformer: video and audio latents are denoised together through fused blocks that cross-attend between the two | ||
| modalities, each conditioned on its own Qwen2.5-VL text branch and a CLIP pooled embedding. A separate | ||
| super-resolution model upscales the generated video tile by tile in the latent space of a causal 3D K-VAE. | ||
|
|
||
| > [!TIP] | ||
| > Check out the [Kandinsky Lab](https://huggingface.co/kandinskylab) organization on the Hub for the full set of | ||
| > official checkpoints, including flow-matching and distilled variants of both the base and super-resolution models. | ||
| > | ||
| > Distilled checkpoints ship with the few-step [`PiflowScheduler`] and must be run with `guidance_scale=1.0`. | ||
|
|
||
| ## Available models | ||
|
|
||
| | Model | Pipeline | Notes | | ||
| |---|---|---| | ||
| | [`kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers`](https://huggingface.co/kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers) | [`Kandinsky6TI2VAPipeline`] | Flow matching, `guidance_scale=5.0`, 50 steps | | ||
| | [`kandinskylab/Kandinsky-6.0-Pro-distill-5s-Diffusers`](https://huggingface.co/kandinskylab/Kandinsky-6.0-Pro-distill-5s-Diffusers) | [`Kandinsky6TI2VAPipeline`] | Distilled, `guidance_scale=1.0`, 10 steps | | ||
| | [`kandinskylab/Kandinsky-6.0-Lite-5s-Diffusers`](https://huggingface.co/kandinskylab/Kandinsky-6.0-Lite-5s-Diffusers) | [`Kandinsky6TI2VAPipeline`] | Flow matching, `guidance_scale=5.0`, 50 steps | | ||
| | [`kandinskylab/Kandinsky-6.0-Lite-distill-5s-Diffusers`](https://huggingface.co/kandinskylab/Kandinsky-6.0-Lite-distill-5s-Diffusers) | [`Kandinsky6TI2VAPipeline`] | Distilled, `guidance_scale=1.0`, 10 steps | | ||
| | [`kandinskylab/Kandinsky-6.0-Pro-pretrain-5s-Diffusers`](https://huggingface.co/kandinskylab/Kandinsky-6.0-Pro-pretrain-5s-Diffusers) | [`Kandinsky6TI2VAPipeline`] | Flow matching, `guidance_scale=5.0`, 50 steps | | ||
| | [`kandinskylab/Kandinsky-6.0-Lite-pretrain-5s-Diffusers`](https://huggingface.co/kandinskylab/Kandinsky-6.0-Lite-pretrain-5s-Diffusers) | [`Kandinsky6TI2VAPipeline`] | Flow matching, `guidance_scale=5.0`, 50 steps | | ||
| | [`kandinskylab/Kandinsky-6.0-VSR-5s-Diffusers`](https://huggingface.co/kandinskylab/Kandinsky-6.0-VSR-5s-Diffusers) | [`Kandinsky6SRPipeline`] | Flow matching super-resolution | | ||
| | [`kandinskylab/Kandinsky-6.0-VSR-distilled2steps-5s-Diffusers`](https://huggingface.co/kandinskylab/Kandinsky-6.0-VSR-distilled2steps-5s-Diffusers) | [`Kandinsky6SRPipeline`] | Distilled super-resolution, 2 steps | | ||
|
|
||
| ## Text/image-to-video-and-audio | ||
|
|
||
| ```python | ||
| import torch | ||
| from diffusers import Kandinsky6TI2VAPipeline | ||
| from diffusers.utils import encode_video | ||
|
|
||
| pipe = Kandinsky6TI2VAPipeline.from_pretrained( | ||
| "kandinskylab/Kandinsky-6.0-Pro-distill-5s-Diffusers", torch_dtype=torch.bfloat16 | ||
| ) | ||
| pipe.enable_model_cpu_offload() | ||
|
|
||
| output = pipe( | ||
| prompt="A cat and a dog baking a cake together in a kitchen.", | ||
| height=480, | ||
| width=864, | ||
| num_frames=121, | ||
| num_inference_steps=10, | ||
| guidance_scale=1.0, | ||
| ) | ||
| encode_video( | ||
| output.frames[0], | ||
| fps=24, | ||
| output_path="output.mp4", | ||
| audio=output.audio[0][None], | ||
| audio_sample_rate=pipe.audio_sample_rate, | ||
| ) | ||
| ``` | ||
|
|
||
| Pass `image=` to condition the first frame on a reference image, `sample_audio=False` to generate video only, and | ||
| `expand_prompts=True` to let the Qwen2.5-VL text encoder rewrite short prompts into detailed ones first. | ||
|
|
||
| ## Video super-resolution | ||
|
|
||
| [`Kandinsky6SRPipeline`] takes the frames produced by [`Kandinsky6TI2VAPipeline`] and upscales them by `2`, `4`, or | ||
| `2.25` (a 1.125x bilinear pre-upscale followed by the 2x path). The video is split into overlapping tiles, every tile | ||
| is refined at one of the tile sizes the SR transformer was trained on, and the tiles are blended back with Hann | ||
| windows. | ||
|
|
||
| ```python | ||
| # required: lets inductor pick flex-attention tiles that fit the SR block mask | ||
| torch._inductor.config.max_autotune = True | ||
|
|
||
| sr_pipe = Kandinsky6SRPipeline.from_pretrained( | ||
|
sayakpaul marked this conversation as resolved.
|
||
| "kandinskylab/Kandinsky-6.0-VSR-distilled2steps-5s-Diffusers", torch_dtype=torch.bfloat16 | ||
| ) | ||
| # The SR transformer always runs NABLA sparse attention on the `flex` backend. Compile it, otherwise flex falls | ||
| # back to an eager implementation that needs far more memory at video resolutions. | ||
| sr_pipe.enable_model_cpu_offload() | ||
| sr_pipe.transformer.set_attention_backend("flex") | ||
| sr_pipe.transformer.compile_repeated_blocks(fullgraph=True) | ||
|
|
||
| upscaled = sr_pipe(video=output.frames[0], resolution_scale=2.25, num_inference_steps=2).frames[0] | ||
| ``` | ||
|
|
||
| ## Memory optimization | ||
|
|
||
| Refer to the [Reduce memory usage](../../optimization/memory) guide for the general set of techniques. Both | ||
| [`Kandinsky6TI2VAPipeline`] and [`Kandinsky6SRPipeline`] support [model offloading](../../optimization/memory#model-offloading) | ||
| (used above) and, for a smaller footprint at the cost of speed, [sequential CPU offloading](../../optimization/memory#cpu-offloading): | ||
|
|
||
| ```python | ||
| pipe.enable_sequential_cpu_offload() | ||
| ``` | ||
|
|
||
| [`Kandinsky6TI2VAPipeline`]'s video VAE also supports [tiled decoding](../../optimization/memory#vae-tiling) for high | ||
| resolutions or long videos: | ||
|
|
||
| ```python | ||
| pipe.vae.enable_tiling() | ||
| ``` | ||
|
|
||
| ## Notes | ||
|
|
||
| - `height` and `width` must be divisible by the video VAE's spatial compression ratio times the transformer's patch | ||
| size — `16` with the default [`Kandinsky6TI2VAPipeline`] configuration (`AutoencoderKLHunyuanVideo` at a | ||
| compression ratio of `8`, `patch_size=(1, 2, 2)`). `480x864`, used in the example above, satisfies this. | ||
| - [`Kandinsky6SRPipeline`]'s input `video` must have `1 + k * vae_scale_factor_temporal` frames for some integer `k` | ||
| (a temporal compression ratio of `4` with the default K-VAE configuration, so `121` frames works but `120` doesn't) | ||
| — trim or pad a video that doesn't already satisfy this before upscaling it. | ||
| - `expand_prompts=True` reuses the already-loaded Qwen2.5-VL text encoder for an extra generation pass before | ||
| denoising, so it adds latency but no extra model weights. | ||
| - Compile the repeated transformer blocks for faster repeated inference: | ||
| ```python | ||
| pipe.transformer.compile_repeated_blocks(fullgraph=True) | ||
| ``` | ||
|
|
||
| ## Kandinsky6TI2VAPipeline | ||
|
|
||
| [[autodoc]] Kandinsky6TI2VAPipeline | ||
| - all | ||
| - __call__ | ||
|
|
||
| ## Kandinsky6SRPipeline | ||
|
|
||
| [[autodoc]] Kandinsky6SRPipeline | ||
| - all | ||
| - __call__ | ||
|
|
||
| ## Kandinsky6SRLatentUpscalerBank | ||
|
|
||
| [[autodoc]] Kandinsky6SRLatentUpscalerBank | ||
| - forward | ||
|
|
||
| ## Kandinsky6TI2VAPipelineOutput | ||
|
|
||
| [[autodoc]] pipelines.kandinsky6.pipeline_output.Kandinsky6TI2VAPipelineOutput | ||
|
|
||
| ## Kandinsky6SRPipelineOutput | ||
|
|
||
| [[autodoc]] pipelines.kandinsky6.pipeline_output.Kandinsky6SRPipelineOutput | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| <!--Copyright 2026 The Kandinsky Team and The HuggingFace Team. All rights reserved. | ||
|
|
||
| Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with | ||
| the License. You may obtain a copy of the License at | ||
|
|
||
| http://www.apache.org/licenses/LICENSE-2.0 | ||
|
|
||
| Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on | ||
| an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the | ||
| specific language governing permissions and limitations under the License. | ||
| --> | ||
|
|
||
| # PiflowScheduler | ||
|
|
||
| `PiflowScheduler` is the few-step scheduler of the distilled Kandinsky 6 checkpoints, both the | ||
| text/image-to-video-and-audio model and the video super-resolution model. It implements | ||
| [π-Flow](https://huggingface.co/papers/2510.14974): the transformer predicts `n_grid` denoised estimates per latent | ||
| channel at a small number of grid points, and the scheduler integrates a network-free policy between them. | ||
|
|
||
| The reference implementation can be found at [Lakonik/LakonLab](https://github.com/Lakonik/LakonLab). | ||
|
|
||
| ## PiflowScheduler | ||
|
|
||
| [[autodoc]] PiflowScheduler | ||
| - set_timesteps | ||
| - step | ||
|
|
||
| ## PiflowSchedulerOutput | ||
|
|
||
| [[autodoc]] schedulers.scheduling_piflow.PiflowSchedulerOutput |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.