Skip to content

Onboard Kandinsky 6.0 (T/I2V+audio, Lite/Pro, SR) - blocked on diffusers release #663

Closed
Closed
Onboard Kandinsky 6.0 (T/I2V+audio, Lite/Pro, SR) - blocked on diffusers release#663
Labels
featureWork bigger than a fix: designed with Don, built in stages (harnest R11)owner:donParked for human decision

Description

@dkackman
dkackman
opened on Oct 6, 2026

What

Onboard Kandinsky 6.0 (kandinskylab, MIT) as a new model family: a joint text/image-to-video+audio model (5 s clips, 44.1 kHz audio incl. lip-sync) with a separate tiled super-resolution pipeline.

Collection: https://huggingface.co/collections/kandinskylab/kandinsky-60-diffusers

Diffusers support

  • Merged to diffusers main on 2026-10-06 (Kandinsky6 huggingface/diffusers#14949, bc5e3bd9): Kandinsky6TI2VAPipeline, Kandinsky6SRPipeline, Kandinsky6Transformer3DModel (+ SR transformer).
  • Not in a release. Latest is v0.41.0 and the repo venv's 0.41.0.dev0 has no Kandinsky6* classes. Onboarding is blocked on a diffusers bump (git main or the next release). Check get_diffusers_state / list_classes first.
  • Components: Qwen2.5-VL + CLIP text encoders, AutoencoderKLHunyuanVideo, MMAudioVAE + vocoder (audio parts optional, sample_audio).

Checkpoints (all *-5s-Diffusers)

Variant Size Steps / CFG Notes
Lite-distill 3B 10 / 1.0 PiflowScheduler, must use guidance 1.0
Lite 3B default 50
Lite-pretrain 3B base, not for end use
Pro-distill 29-30B ~16 / 1.0
Pro 29B 50 / ~5.0 FlowMatchEuler
Pro-pretrain 29-30B base

Native 480x864, 121 frames. SR pipeline upscales to 1920x1080 (x2 / x2.25 / x4, 4 steps per tile, 2 in distilled).

24 GB (RTX 3090) feasibility

  • Lite / Lite-distill: yes. 3B transformer is ~6 GB in bf16; with enable_model_cpu_offload() the Qwen2.5-VL encoder is the largest piece but is offloaded. Start here, Lite-distill (10 steps, no CFG pass) is the right first template.
  • Pro: only quantized. 29B bf16 is ~58 GB. Needs the transformer in nf4/int8 (see docs/QUANTIZATION.md, docs/RECIPES_24GB.md) plus offload. Plausible, unverified; measure before promising. Pro with CFG (50 steps x 2 passes) will be slow.
  • SR on top of Lite output: tiled, so memory should be bounded, but verify.
  • All of the above is an estimate from the model cards. No VRAM numbers are published.

Plan (per the model-family-onboarding skill)

  1. Bump diffusers on lem (new-HF-dep deploy pattern), confirm classes load.
  2. Audit against vendor text (cards, GitHub, tech report, pipeline source) -> docs/proposals/audits/.
  3. Catalog: models/ entries (Lite-distill first, then Lite, Pro quantized), templates for t2v+audio, i2v+audio, and SR; cost drivers measured on lem.
  4. Check the <S>/<AUDCAP> prompt tags and the built-in prompt-beautifier system prompt in the pipeline. Point at it, do not copy it.
  5. Composition skill in plugins/dw/ (12 KiB cap; pinned by test).
  6. lem field run of Lite-distill, then Pro quantized if the 24 GB question is worth answering.

Not filed

HunyuanImage-3.0 (tencent): no diffusers support. pipelines/hunyuan_image holds only the 2.1 pipelines (HunyuanImagePipeline, refiner). 3.0 is an 80B MoE autoregressive model (13B active) run via transformers AutoModelForCausalLM with trust_remote_code=True, which the catalog forbids. Revisit if a diffusers PR lands.

Activity

  1. dkackman
    added
    owner:donParked for human decision
    featureWork bigger than a fix: designed with Don, built in stages (harnest R11)
    on Oct 6, 2026
  2. dkackman
    added 7 commits that reference this issue on Oct 6, 2026

    fix(audio): #663 - read a video+audio pipeline's rate off its audio V…

    e20fff5

    fix(updater): #663 - a git diffusers install replaces one of the same…

    8df5915

    feat(kandinsky6): #663 - audit, Lite-distill text- and image-to-video…

    c5875f2

    test(kandinsky6): #663 - record the family's prompt spelling and pipe…

    23d1847

    fix(downloads): #663 - a finished download is no longer reported fail…

    2295646

    fix(kandinsky6): #663 - decode the HunyuanVideo VAE in tiles

    95023a1

    docs(kandinsky6): #663 - park Apple Silicon: the HunyuanVideo VAE dec…

    68325bf

  3. dkackman commented on Oct 6, 2026

    @dkackman
    dkackman
    on Oct 6, 2026
    OwnerAuthor

    Status 2026-10-06: first onboarding pass on feat/663-kandinsky6; MPS parked

    Work is on branch feat/663-kandinsky6 (7 commits on develop). It has no field run yet, and the next step is lem.

    Done

    • Diffusers bump. The branch needs diffusers main; c6df88a5 includes #14949 and is installed on mini-ai. The version string is still 0.41.0.dev0, so check for the presence of Kandinsky6TI2VAPipeline, not the version.
    • Audit: docs/proposals/audits/2026-10-06-kandinsky-6-audit.md. These claims in this issue are corrected:
      • Pro-distill is 10 steps / CFG 1.0, not ~16. The 16 is a stale diffusers docstring; the cards, the vendor configs and the vendor's PR comment all say 10.
      • The pretrain checkpoints are presented as generation checkpoints (50 / 5.0), not "not for end use".
      • On 24 GB, the vendor runs Pro-distill unquantized with block offload (given enough host RAM), so "Pro only quantized" is wrong.
      • SR at x2.25 on 480x864 gives 1088x1952 in diffusers, not 1920x1080.
      • The beautifier is two user-turn instructions (_T2VA/_I2VA_EXPANSION_INSTRUCTION, off by default). The vendor's full prompting guide is comfyui/kandinsky6/beautifier_prompts/t2av_system.txt / i2av_system.txt.
      • The pipeline defaults (512x768, guidance 5.0) are not the model's, so the templates set 480x864 and 1.0 explicitly.
      • The vendor re-keys the weights as diffusers' module names change, so the templates pin a revision.
      • SR needs compiled flex attention (NABLA), so it is CUDA-only for now.
    • Catalog: workflows/templates/kandinsky6/ has text-to-video and image-to-video. Both use Lite-distill + audio: 10 PiFlow steps, guidance 1.0, model offload, VAE tiling, width/height multiples of 16, frames 4k+1 up to 509. There are two stored prompts in prompts/kandinsky6/ in the vendor beautifier's format, plus a README.
      • Compact catalog budget raised 10_000 -> 10_250 for the two templates. Needs Don's OK.
    • dw fixes found along the way, each with tests:
      • e20fff5d: the audio sample rate is now read off audio_vae.config.sample_rate when the vocoder has none. Without it, K6's 44.1 kHz audio would be saved at dw's default rate. This is unit-tested only, because no run has reached the audio decode yet.
      • 8df59150: the diffusers updater (POST /api/system/diffusers/update, update_diffusers) did nothing whenever the dev version string was unchanged. That covered both HEAD and pinned commits, and it still reported success. It now does --force-reinstall --no-deps and then a plain pip install diffusers for any new dependencies. install.sh and install.ps1 do the same. Verified live on mini-ai: pinned to 578c9b2, then HEAD back to c6df88a5.
      • 22956467: on huggingface_hub 1.33, a completed download was reported failed ('Tracker' object has no attribute 'set_description_str'). Verified that both K6 repos were complete despite the "failed" status.

    MPS: parked

    On mini-ai (64 GB, torch 2.14.1), Lite-distill t2v denoises fine: 10 steps in about 9:10, 54 s/step, with about 82% of memory free. The worker is then SIGKILLed in the HunyuanVideo VAE decode, whether tiled or not. Standalone probe results:

    • Live tensors stay under 1 GiB, but MPS driver memory grows by about 2 GiB per decode tile and is not released, by either torch.mps.empty_cache() or PYTORCH_MPS_LOW_WATERMARK_RATIO.
    • 3 latent frames (9 video frames) hold about 30 GiB in bf16 or fp16, and about 58 GiB in fp32. 9 latent frames get killed.
    • The audio VAE + vocoder decode uses about 1.1 GiB in 9 s.
    • A CPU decode of 3 latent frames had not finished after 10 minutes, so it is not a workaround.

    This looks like an MPS conv3d retention problem, not dw or diffusers. Revisit on a newer torch. Workflows have no MPS-only component override, so moving the VAE to CPU would slow CUDA as well.

    Next (on lem)

    1. Deploy the branch and install diffusers c6df88a5 (the updater fix makes update_diffusers work for this now).
    2. Run templates/kandinsky6/text-to-video and image-to-video; look at and listen to the output; write the cost for an RTX 3090. Check that the 24 GB layout (model offload + VAE tiling) fits.
    3. Composition skill plugins/dw/skills/kandinsky-6/, pinned by tests/test_plugin_skills.py.
    4. Follow-ups, in the audit's order: expand_prompts, speech/lip-sync, Lite 50/5.0, SR (CUDA), Pro-distill/Pro, and sizing image-to-video from the still's aspect ratio.

    Also seen, not from this branch: tests/test_api_contract.py::test_the_committed_document_is_current fails on a clean origin/develop (OpenAPI drift).

  4. dkackman
    added 2 commits that reference this issue on Oct 7, 2026

    feat(kandinsky6): #663 - measured cost on an RTX 3090; the dragon pro…

    e667b34

    feat(plugin): #663 - kandinsky-6 composition skill

    a8db431

  5. dkackman commented on Oct 7, 2026

    @dkackman
    dkackman
    on Oct 7, 2026
    OwnerAuthor

    Status 2026-10-06 (later): verified on lem, skill written

    feat/663-kandinsky6 has develop merged in (90cc1713) and is deployed on lem. Head is a8db4316.

    • Both templates succeeded on an RTX 3090. Output is 864x480, 121 frames at 24 fps, with 44.1 kHz audio, which confirms the e20fff5d rate fix live.
      • text-to-video: 3.41 min, including the first load.
      • image-to-video: 3.54 min warm, with about 13 s per denoising step.
      • Both peak near 10 GB. Model offload moves the 16.6 GB Qwen encoder on and off the card every run, so a warm run is not much faster than a cold one. cost is set at 3.5 and 3.6 min.
      • Results checked: the train clip follows its prompt (static camera, approaching locomotive, lantern, dusk). It gives one long whistle rather than two, and then a steam hiss. The i2v frame 0 matches the supplied still. The dragon's stored prompt is rewritten from the still itself.
    • Both live fixes are verified on lem too. update_diffusers pinned to c6df88a5 really changed the install from c60830ee, and the Lite-distill download finished completed.
    • New gap, not fixed: after update_diffusers, the server process keeps its old diffusers import, so admission refused Kandinsky6TI2VAPipeline as nonexistent until the server was restarted. The updater only restarts the worker. Options are to re-exec the server on success, or to report restart_required in the updater status. This is Don's call.
    • Skill: plugins/dw/skills/kandinsky-6/SKILL.md (6.1 KB). Every number in it is pinned in tests/test_plugin_skills.py::TestKandinsky6Skill to the diffusers module that enforces it (or, for the schedule, to the templates). Prompts point at Kandinsky Lab's t2av_system.txt / i2av_system.txt. It stops on mps.
    • The compact catalog budget is now 10_300 (measured 10_243 after develop's own raise). Needs Don's OK.

    Still owed, per the onboarding lifecycle: a cold drill of the skill (a fresh session with the plugin, plus a control run without it), then a PR to master.

  6. dkackman
    added 3 commits that reference this issue on Oct 7, 2026

    feat(kandinsky6): #663 - generate-and-upscale: tiled super-resolution…

    cce45b9

    docs(release): #663 - name Kandinsky 6, the diffusers updater fix and…

    1b2ef82

    Merge branch 'feat/663-kandinsky6' into develop (#663)

    a2f1820

  7. dkackman commented on Oct 7, 2026

    @dkackman
    dkackman
    on Oct 7, 2026
    OwnerAuthor

    Merged to develop: a2f18203, 2026-10-06

    lem and mini-ai are both deployed on develop. Both still run diffusers main (c6df88a5), which the family needs.

    Since the last update:

    • templates/kandinsky6/generate-and-upscale runs Lite-distill, then Kandinsky6SRPipeline (distilled VSR, transformer compiled), then pair_audio. Output is 1728x960. It took 9.2 min on an RTX 3090 and peaked at 19.3 GiB.
      • It fits 24 GB only with the SR transformer's tile_sizes held to the trained [[512, 512]] and the scale held to 2.
      • At the default 512x768 tile, the SR VAE's per-tile decode needs about 26 GB.
      • Compared with a bicubic 2x, the upscale adds real texture. The skill pins both limits.
    • Longest single segment on a 3090: 509 frames (21.2 s; the transformer's 128-latent-frame RoPE). It used about 22.2 GB at 172 s/step, 32 min.
      • At 4x the trained 121 frames, the look holds but the motion barely progresses: a subject walking toward the camera never gets anywhere. Chaining 121-frame clips will move better.
      • A one-off windowed SR (window_video -> 5 x 121-frame windows, overlap 16 -> join_windows) upscaled that 509-frame clip to 1728x960 in 29 min, with no measurable seams. It is not yet in the catalog.
    • The compact catalog budget is now 10_450, OK'd by Don.
    • docs/RELEASING.md Next names the family, the updater fix (including restarting the server after an update) and the download-status fix.

    Still open:

    • The cold drill of the kandinsky-6 skill (a fresh session with the plugin, plus a control run without it), then a PR to master.
    • Gap: after update_diffusers, the server process keeps its old diffusers import, so admission refuses new classes until the server restarts. Fix by re-execing the server on success, or report restart_required. Don's call.
    • MPS is parked (HunyuanVideo VAE decode memory retention on torch 2.14.1). Revisit on a newer torch.
    • Follow-ups:
      • expand_prompts, a speech/lip-sync example, Lite 50/5.0 and Pro.
      • SR at x2.25/x4 or with the non-distilled VSR.
      • Image-to-video sized from the still's aspect ratio.
      • A kandinsky6/upscale-clip template from the windowed SR.
    • Pre-existing, not from this branch: tests/test_api_contract.py OpenAPI drift on develop.
  8. dkackman
    closed this as completedon Oct 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureWork bigger than a fix: designed with Don, built in stages (harnest R11)owner:donParked for human decision

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions