Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
81f3b12
implement Kandinsky 6
Sep 16, 2026
c892279
update docs for Kandinsky 6
Sep 16, 2026
6b8a080
remove redundant bigvgan code and manual checkpoint loading inside model
Sep 16, 2026
bf4158a
add missing _no_split_modules
Sep 16, 2026
1f1e6ee
use attention processor in SR
Sep 16, 2026
efa2b56
fix unguarded dependencies
Sep 16, 2026
3dc5959
remove dead manual weight loading code; flattent MMAudioVAE structure
Sep 16, 2026
d5a34b6
move attention processors to transformer files
Sep 17, 2026
334d639
remove duplicate code in K6 latent_upscaler
Sep 17, 2026
9d7dc71
address 1st batch of comments
leffff Sep 25, 2026
0eb5211
patch transformer k6
leffff Sep 25, 2026
c3012d0
patch SR
leffff Sep 27, 2026
1f7f234
patch pipeline style
leffff Sep 27, 2026
01bcb64
replace TimestepEmbedding and fix batching
leffff Sep 27, 2026
8a57f57
refactor ti2va dit and pipeline
leffff Sep 28, 2026
d5f8e58
refactor piflow scheduler
leffff Sep 28, 2026
7038c5a
refactor SR
leffff Sep 28, 2026
089a5ae
patch sr transformer autocast
leffff Sep 28, 2026
3cc8f88
major refactor
leffff Sep 28, 2026
65b00fd
untrack notebooks directory
leffff Sep 28, 2026
66a0d44
patch mmaudio VAE
leffff Sep 28, 2026
0ec1bd1
patch mmaudio VAE
leffff Sep 28, 2026
c17252c
fix inits
leffff Sep 28, 2026
d6e1f65
fix flex attn
leffff Sep 28, 2026
b558b74
refactoring
leffff Sep 29, 2026
9debf90
remove __all__
leffff Sep 29, 2026
1cf55a8
separate MMAudio VAE and Vocoder
leffff Sep 29, 2026
525281e
fix naming
leffff Sep 29, 2026
695caec
refactor checkpointing, magcache, transformer, sr vae for K6
leffff Sep 30, 2026
3d9c95c
refactor
leffff Sep 30, 2026
002f4fb
address apply_scale_shift_norm problem
leffff Sep 30, 2026
0e5c599
update docs
leffff Sep 30, 2026
40fe659
apply style chanes
leffff Sep 30, 2026
28c5326
address review feedback on transformer, SR VAE, and flex attention
leffff Oct 1, 2026
372f290
address comments
leffff Oct 1, 2026
cc703db
fix critical bugs
leffff Oct 1, 2026
9ae71f0
Revert checkpoint-breaking module renames from 28c5326da
leffff Oct 1, 2026
a7e596c
Revert checkpoint-breaking latent-upscaler renames from 372f29088
leffff Oct 1, 2026
8fadd97
move vocoder and LU
leffff Oct 2, 2026
0b2ca95
add coppied from resolve i2v cond mode
leffff Oct 2, 2026
6f035c4
rename variables
leffff Oct 3, 2026
1cfe98c
fix lu rename bug
leffff Oct 3, 2026
f16156a
Update src/diffusers/pipelines/kandinsky6/pipeline_kandinsky6_ti2va.py
leffff Oct 3, 2026
348c341
Update src/diffusers/models/transformers/transformer_kandinsky6.py
leffff Oct 3, 2026
0ef584f
address yiyixuxu's comments
leffff Oct 3, 2026
dcbe7b0
Merge remote-tracking branch 'origin/Kandinsky6' into Kandinsky6
leffff Oct 3, 2026
4c0ad12
fix audio channels bug
leffff Oct 4, 2026
1f9b471
fix tests
leffff Oct 5, 2026
60cad43
Fix CI failures on the Kandinsky6 PR
leffff Oct 5, 2026
35e6cc4
Avoid staging LFS dedup race in the org push test
leffff Oct 5, 2026
6d8a643
Revert "Avoid staging LFS dedup race in the org push test"
leffff Oct 5, 2026
0b4439b
Revert "Fix CI failures on the Kandinsky6 PR"
leffff Oct 5, 2026
fee0b42
Merge branch 'main' into Kandinsky6
yiyixuxu Oct 5, 2026
82f46ef
Apply suggestion from @yiyixuxu
yiyixuxu Oct 5, 2026
3bc67ad
Apply suggestion from @yiyixuxu
yiyixuxu Oct 5, 2026
ebea53a
Apply batched suggestions from code review
yiyixuxu Oct 5, 2026
82d3eb5
remove review-only files from the Kandinsky6 PR
yiyixuxu Oct 5, 2026
bbab023
Apply suggestion from @yiyixuxu
yiyixuxu Oct 5, 2026
34f329f
restore .gitignore to match main
yiyixuxu Oct 5, 2026
1ea5fb3
simplify the mono audio handling in _write_audio
yiyixuxu Oct 5, 2026
871acef
remove the no-op set_attention_backend calls from the Kandinsky6 docs
yiyixuxu Oct 5, 2026
9c78058
give the Kandinsky6 transformer its own output class instead of reusi…
yiyixuxu Oct 5, 2026
c8a0c1f
fix the stale pipeline exports and add return annotations for the Kan…
yiyixuxu Oct 6, 2026
39138ba
Merge branch 'main' into Kandinsky6
sayakpaul Oct 6, 2026
7245998
address feedback from @leffff
sayakpaul Oct 6, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs/source/en/_toctree.yml
Original file line number Diff line number Diff line change
Expand Up @@ -375,6 +375,8 @@
title: JoyImageEditPlusTransformer3DModel
- local: api/models/transformer_joyimage
title: JoyImageEditTransformer3DModel
- local: api/models/kandinsky6_transformer
title: Kandinsky 6 Transformers
- local: api/models/krea2_transformer2d
title: Krea2Transformer2DModel
- local: api/models/latte_transformer3d
Expand Down Expand Up @@ -501,6 +503,8 @@
title: AutoencoderSAME
- local: api/models/consistency_decoder_vae
title: ConsistencyDecoderVAE
- local: api/models/kandinsky6_vae
title: Kandinsky 6 VAEs
- local: api/models/ltx2_diffusion_decoder
title: LTX2VideoDiffusionDecoderModel
- local: api/models/autoencoder_oobleck
Expand Down Expand Up @@ -725,6 +729,8 @@
title: HunyuanVideo1.5
- local: api/pipelines/kandinsky5_video
title: Kandinsky 5.0 Video
- local: api/pipelines/kandinsky6
title: Kandinsky 6
- local: api/pipelines/latte
title: Latte
- local: api/pipelines/ltx2
Expand Down Expand Up @@ -816,6 +822,8 @@
title: LMSDiscreteScheduler
- local: api/schedulers/minimax_h3
title: MiniMaxH3Scheduler
- local: api/schedulers/piflow
title: PiflowScheduler
- local: api/schedulers/pndm
title: PNDMScheduler
- local: api/schedulers/repaint
Expand Down
53 changes: 53 additions & 0 deletions docs/source/en/api/models/kandinsky6_transformer.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
<!--Copyright 2026 The Kandinsky Team and The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.
-->

# Kandinsky 6 Transformers

Kandinsky 6 uses a multimodal diffusion transformer that denoises video and audio latents together for
text/image-to-video-and-audio generation, and a text-free diffusion transformer for video super-resolution.

## Kandinsky6Transformer3DModel

The multimodal transformer used by [`Kandinsky6TI2VAPipeline`].

```python
import torch
from diffusers import Kandinsky6Transformer3DModel

transformer = Kandinsky6Transformer3DModel.from_pretrained(
"kandinskylab/Kandinsky-6.0-Pro-distill-5s-Diffusers", subfolder="transformer", torch_dtype=torch.bfloat16
)
```

[[autodoc]] Kandinsky6Transformer3DModel
- all
- forward

## Kandinsky6SRTransformer3DModel

The text-free transformer used by [`Kandinsky6SRPipeline`] to refine one tile of the upscaled video at a time.

```python
import torch
from diffusers import Kandinsky6SRTransformer3DModel

transformer = Kandinsky6SRTransformer3DModel.from_pretrained(
"kandinskylab/Kandinsky-6.0-VSR-distilled2steps-5s-Diffusers", subfolder="transformer", torch_dtype=torch.bfloat16
)
# The transformer always runs NABLA sparse attention (`nabla_threshold`, 0.8 by default) on the `flex` backend.
# Compile it, otherwise flex falls back to an eager implementation that needs far more memory at video resolutions.
transformer.compile_repeated_blocks(fullgraph=True)
```

[[autodoc]] Kandinsky6SRTransformer3DModel
- all
- forward
66 changes: 66 additions & 0 deletions docs/source/en/api/models/kandinsky6_vae.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
<!--Copyright 2026 The Kandinsky Team and The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.
-->

# Kandinsky 6 VAEs

Kandinsky 6 uses a causal 3D K-VAE for video super-resolution and the MMAudio mel-spectrogram VAE, paired with a
separate BigVGAN [`MMAudioVocoder`], for synchronized audio generation.

## Kandinsky6SRVAE

The causal 3D K-VAE used by [`Kandinsky6SRPipeline`]. It processes arbitrarily long videos in bounded-memory
segments while reproducing the exact output of a single, non-segmented pass.

```python
import torch
from diffusers import Kandinsky6SRVAE

vae = Kandinsky6SRVAE.from_pretrained(
"kandinskylab/Kandinsky-6.0-VSR-5s-Diffusers", subfolder="vae", torch_dtype=torch.bfloat16
)
```

[[autodoc]] Kandinsky6SRVAE
- encode
- decode
- all

## MMAudioVAE

The mel-spectrogram VAE used by [`Kandinsky6TI2VAPipeline`] when `sample_audio=True`. Its `decode` output is a mel
spectrogram; pass it through [`MMAudioVocoder`] to get a waveform.

The reference implementation can be found at [hkchengrex/MMAudio](https://github.com/hkchengrex/MMAudio) (MIT
license).

```python
import torch
from diffusers import MMAudioVAE

audio_vae = MMAudioVAE.from_pretrained(
"kandinskylab/Kandinsky-6.0-Pro-distill-5s-Diffusers", subfolder="audio_vae", torch_dtype=torch.bfloat16
)
```

[[autodoc]] MMAudioVAE
- encode
- decode
- all

## MMAudioVocoder

Adapted from the BigVGAN-v2 vocoder MMAudio bundles, itself from
[NVIDIA/BigVGAN](https://github.com/NVIDIA/BigVGAN) (MIT license), with the anti-aliased Snake activations of
[alias-free-torch](https://github.com/junjun3518/alias-free-torch) (Apache License 2.0).

[[autodoc]] MMAudioVocoder
- forward
150 changes: 150 additions & 0 deletions docs/source/en/api/pipelines/kandinsky6.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,150 @@
<!--Copyright 2026 The Kandinsky Team and The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.
-->

# Kandinsky 6

Kandinsky 6 is a family of video generation models from [Kandinsky Lab](https://huggingface.co/kandinskylab). The
main model generates video and synchronized audio from text or a reference image with a single multimodal diffusion
transformer: video and audio latents are denoised together through fused blocks that cross-attend between the two
modalities, each conditioned on its own Qwen2.5-VL text branch and a CLIP pooled embedding. A separate
super-resolution model upscales the generated video tile by tile in the latent space of a causal 3D K-VAE.

> [!TIP]
> Check out the [Kandinsky Lab](https://huggingface.co/kandinskylab) organization on the Hub for the full set of
> official checkpoints, including flow-matching and distilled variants of both the base and super-resolution models.
>
> Distilled checkpoints ship with the few-step [`PiflowScheduler`] and must be run with `guidance_scale=1.0`.

## Available models
Comment thread
sayakpaul marked this conversation as resolved.

| Model | Pipeline | Notes |
|---|---|---|
| [`kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers`](https://huggingface.co/kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers) | [`Kandinsky6TI2VAPipeline`] | Flow matching, `guidance_scale=5.0`, 50 steps |
| [`kandinskylab/Kandinsky-6.0-Pro-distill-5s-Diffusers`](https://huggingface.co/kandinskylab/Kandinsky-6.0-Pro-distill-5s-Diffusers) | [`Kandinsky6TI2VAPipeline`] | Distilled, `guidance_scale=1.0`, 10 steps |
| [`kandinskylab/Kandinsky-6.0-Lite-5s-Diffusers`](https://huggingface.co/kandinskylab/Kandinsky-6.0-Lite-5s-Diffusers) | [`Kandinsky6TI2VAPipeline`] | Flow matching, `guidance_scale=5.0`, 50 steps |
| [`kandinskylab/Kandinsky-6.0-Lite-distill-5s-Diffusers`](https://huggingface.co/kandinskylab/Kandinsky-6.0-Lite-distill-5s-Diffusers) | [`Kandinsky6TI2VAPipeline`] | Distilled, `guidance_scale=1.0`, 10 steps |
| [`kandinskylab/Kandinsky-6.0-Pro-pretrain-5s-Diffusers`](https://huggingface.co/kandinskylab/Kandinsky-6.0-Pro-pretrain-5s-Diffusers) | [`Kandinsky6TI2VAPipeline`] | Flow matching, `guidance_scale=5.0`, 50 steps |
| [`kandinskylab/Kandinsky-6.0-Lite-pretrain-5s-Diffusers`](https://huggingface.co/kandinskylab/Kandinsky-6.0-Lite-pretrain-5s-Diffusers) | [`Kandinsky6TI2VAPipeline`] | Flow matching, `guidance_scale=5.0`, 50 steps |
| [`kandinskylab/Kandinsky-6.0-VSR-5s-Diffusers`](https://huggingface.co/kandinskylab/Kandinsky-6.0-VSR-5s-Diffusers) | [`Kandinsky6SRPipeline`] | Flow matching super-resolution |
| [`kandinskylab/Kandinsky-6.0-VSR-distilled2steps-5s-Diffusers`](https://huggingface.co/kandinskylab/Kandinsky-6.0-VSR-distilled2steps-5s-Diffusers) | [`Kandinsky6SRPipeline`] | Distilled super-resolution, 2 steps |

## Text/image-to-video-and-audio

```python
import torch
from diffusers import Kandinsky6TI2VAPipeline
from diffusers.utils import encode_video

pipe = Kandinsky6TI2VAPipeline.from_pretrained(
"kandinskylab/Kandinsky-6.0-Pro-distill-5s-Diffusers", torch_dtype=torch.bfloat16
)
pipe.enable_model_cpu_offload()

output = pipe(
prompt="A cat and a dog baking a cake together in a kitchen.",
height=480,
width=864,
num_frames=121,
num_inference_steps=10,
guidance_scale=1.0,
)
encode_video(
output.frames[0],
fps=24,
output_path="output.mp4",
audio=output.audio[0][None],
audio_sample_rate=pipe.audio_sample_rate,
)
```

Pass `image=` to condition the first frame on a reference image, `sample_audio=False` to generate video only, and
`expand_prompts=True` to let the Qwen2.5-VL text encoder rewrite short prompts into detailed ones first.

## Video super-resolution

[`Kandinsky6SRPipeline`] takes the frames produced by [`Kandinsky6TI2VAPipeline`] and upscales them by `2`, `4`, or
`2.25` (a 1.125x bilinear pre-upscale followed by the 2x path). The video is split into overlapping tiles, every tile
is refined at one of the tile sizes the SR transformer was trained on, and the tiles are blended back with Hann
windows.

```python
# required: lets inductor pick flex-attention tiles that fit the SR block mask
torch._inductor.config.max_autotune = True

sr_pipe = Kandinsky6SRPipeline.from_pretrained(
Comment thread
sayakpaul marked this conversation as resolved.
"kandinskylab/Kandinsky-6.0-VSR-distilled2steps-5s-Diffusers", torch_dtype=torch.bfloat16
)
# The SR transformer always runs NABLA sparse attention on the `flex` backend. Compile it, otherwise flex falls
# back to an eager implementation that needs far more memory at video resolutions.
sr_pipe.enable_model_cpu_offload()
sr_pipe.transformer.set_attention_backend("flex")
sr_pipe.transformer.compile_repeated_blocks(fullgraph=True)

upscaled = sr_pipe(video=output.frames[0], resolution_scale=2.25, num_inference_steps=2).frames[0]
```

## Memory optimization

Refer to the [Reduce memory usage](../../optimization/memory) guide for the general set of techniques. Both
[`Kandinsky6TI2VAPipeline`] and [`Kandinsky6SRPipeline`] support [model offloading](../../optimization/memory#model-offloading)
(used above) and, for a smaller footprint at the cost of speed, [sequential CPU offloading](../../optimization/memory#cpu-offloading):

```python
pipe.enable_sequential_cpu_offload()
```

[`Kandinsky6TI2VAPipeline`]'s video VAE also supports [tiled decoding](../../optimization/memory#vae-tiling) for high
resolutions or long videos:

```python
pipe.vae.enable_tiling()
```

## Notes

- `height` and `width` must be divisible by the video VAE's spatial compression ratio times the transformer's patch
size — `16` with the default [`Kandinsky6TI2VAPipeline`] configuration (`AutoencoderKLHunyuanVideo` at a
compression ratio of `8`, `patch_size=(1, 2, 2)`). `480x864`, used in the example above, satisfies this.
- [`Kandinsky6SRPipeline`]'s input `video` must have `1 + k * vae_scale_factor_temporal` frames for some integer `k`
(a temporal compression ratio of `4` with the default K-VAE configuration, so `121` frames works but `120` doesn't)
— trim or pad a video that doesn't already satisfy this before upscaling it.
- `expand_prompts=True` reuses the already-loaded Qwen2.5-VL text encoder for an extra generation pass before
denoising, so it adds latency but no extra model weights.
- Compile the repeated transformer blocks for faster repeated inference:
```python
pipe.transformer.compile_repeated_blocks(fullgraph=True)
```

## Kandinsky6TI2VAPipeline

[[autodoc]] Kandinsky6TI2VAPipeline
- all
- __call__

## Kandinsky6SRPipeline

[[autodoc]] Kandinsky6SRPipeline
- all
- __call__

## Kandinsky6SRLatentUpscalerBank

[[autodoc]] Kandinsky6SRLatentUpscalerBank
- forward

## Kandinsky6TI2VAPipelineOutput

[[autodoc]] pipelines.kandinsky6.pipeline_output.Kandinsky6TI2VAPipelineOutput

## Kandinsky6SRPipelineOutput

[[autodoc]] pipelines.kandinsky6.pipeline_output.Kandinsky6SRPipelineOutput
30 changes: 30 additions & 0 deletions docs/source/en/api/schedulers/piflow.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
<!--Copyright 2026 The Kandinsky Team and The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.
-->

# PiflowScheduler

`PiflowScheduler` is the few-step scheduler of the distilled Kandinsky 6 checkpoints, both the
text/image-to-video-and-audio model and the video super-resolution model. It implements
[π-Flow](https://huggingface.co/papers/2510.14974): the transformer predicts `n_grid` denoised estimates per latent
channel at a small number of grid points, and the scheduler integrates a network-free policy between them.

The reference implementation can be found at [Lakonik/LakonLab](https://github.com/Lakonik/LakonLab).

## PiflowScheduler

[[autodoc]] PiflowScheduler
- set_timesteps
- step

## PiflowSchedulerOutput

[[autodoc]] schedulers.scheduling_piflow.PiflowSchedulerOutput
Loading
Loading