Problems loading LTX-2 transformer GGUFs

[AmericanPresidentJimmyCarter]

Describe the bug

Can't seem to load GGUFs for ltx-2. More specifically, because time_embed.emb.timestep_embedder.linear_1.weight expected shape torch.Size([4096, 256]), but got torch.Size([4096, 512])

Reproduction

Running on commit 7f43cb1, using the weights from this repo: https://huggingface.co/unsloth/LTX-2-GGUF

$ python3 -c "
> from diffusers.models.transformers import LTXVideoTransformer3DModel
> import torch
> try:
>     transformer = LTXVideoTransformer3DModel.from_single_file(
>         '/home/user/.cache/huggingface/hub/models--unsloth--LTX-2-GGUF/snapshots/7d0ce36769829be32526a4ac9b8442e89dcaecbd/ltx-2-19b-dev-Q6_K.gguf',
>         torch_dtype=torch.bfloat16
>     )
>     print(f'Success: {type(transformer)}')
> except Exception as e:
>     print(f'Error: {e}')
> "

Logs

config.json: 100%|███████████████████████████████████████████████████████████████████████████| 501/501 [00:00<00:00, 4.29MB/s]
Error: Cannot load  because time_embed.emb.timestep_embedder.linear_1.weight expected shape torch.Size([4096, 256]), but got torch.Size([4096, 512]). If you want to instead overwrite randomly initialized weights, please make sure to pass both `low_cpu_mem_usage=False` and `ignore_mismatched_sizes=True`. For more information, see also: https://github.com/huggingface/diffusers/issues/1619#issuecomment-1345604389 as an example.

System Info

Linux x64, 3090

Who can help?

@DN6 @sayakpaul

[bonzai-depin]

Hello @AmericanPresidentJimmyCarter, it appears you're using the LTX (1) class instead of LTX-2's LTX2VideoTransformer3DModel.

The problem is that from_single_file doesn't seem to be supported in LTX-2 just yet @sayakpaul @rootonchair (?)

We had to build our own (dirty) GGUF loader to use quants, and enable LTX-2 to run on more consumer hardware.

On a side note, it would also be great to be able to use Gemma's GGUF as well, since it's actually the biggest bottleneck in running LTX-2. Especially on MPS devices.

[asomoza]

Sorry for deviating from the main issue, I understand that is for enabling from_single_file for LTX2 but in the meantime, as an alternative solution @bonzai-depin if you're open to alternatives you can use bitasandbytes quants and sdnq quantization right now with both the transformer and the text encoder:

Here are the examples on how to use them:

bitsandbytes

sdnq

[AmericanPresidentJimmyCarter]

I see -- this issue is just for generic GGUF support then.

Hello @AmericanPresidentJimmyCarter, it appears you're using the LTX (1) class instead of LTX-2's LTX2VideoTransformer3DModel.

The problem is that from_single_file doesn't seem to be supported in LTX-2 just yet @sayakpaul @rootonchair (?)

We had to build our own (dirty) GGUF loader to use quants, and enable LTX-2 to run on more consumer hardware.

On a side note, it would also be great to be able to use Gemma's GGUF as well, since it's actually the biggest bottleneck in running LTX-2. Especially on MPS devices.

I just did inference completely without a diffusers pipeline because the pipelines are consistently difficult to work with. I have limited RAM on some servers and there seems to be no way to hot load/unload the text encoder onto disk when you're done (as opposed to what diffusers forces you to do in pipelines, which is put it back into RAM).

Here's my script using quanto int8 and configurable groupwise layer offload using normal torch hooks.

https://gist.github.com/AmericanPresidentJimmyCarter/b77df7fa37d36c425959845c9e72aaa1

Once the models are cached in int8, it takes about 1min per second of LTX video+audio on the dev model at 1 megapixel. There are some weird things using this model in diffusers though, like hyperintense colour regions sometimes showing up after encoding the latents. I don't know, I give up on the diffusers impl for now.

[dg845]

There are some weird things using this model in diffusers though, like hyperintense colour regions sometimes showing up after encoding the latents.

Hi @AmericanPresidentJimmyCarter, would you be willing to share an example of this, so that we can debug? Thanks!

[AmericanPresidentJimmyCarter]

Sure. I'm using the script I gave above for quanto int8 and this image file, rescaled to <1MP (1024x512) so that I don't OOM.

Using the VAE in fp32 had no effect on creating the hyperintense regions as well. And for some reason I get inverted latents on output and have to use video = 1.0 - video?

ltx2_i2v_int8_quanto_lora06_12steps.mp4

As you can see, it makes a 5s video just fine, but there are some hyperintense regions in the output.

Another line that I copied from the pipeline was:

normalize_latents(init, vae.latents_mean, vae.latents_std) # Why is vae.config.scaling_factor not at the end?

That was straight out of the pipeline for the current code for i2v. The scaling factor is used when you call denormalize_latents later.

Changing normalize_latents to use the scaling factor doesn't seem to make a difference either. So, not sure how this is supposed to work or if I did something wrong.

edit: And sorry, the command I am using is CUDA_VISIBLE_DEVICES=1 python sample_ltx2_i2v_2.py --image "qwenimage_samples/city.png" --prompt "A busy night in futuristic tokyo, cinematic, high detail." --negative "worst quality, inconsistent motion, blurry, jittery, distorted" --out "ltx2_i2v_int8_quanto.mp4" --steps 20 --seed 42 --group_size=8 --keep_non_block_on_gpu

2nd edit: The LLM cleaned up version of the script had a bug so I just uploaded the original version. It should work now with the above command, sorry.

[bonzai-depin]

Sorry for deviating from the main issue, I understand that is for enabling from_single_file for LTX2 but in the meantime, as an alternative solution @bonzai-depin if you're open to alternatives you can use bitasandbytes quants and sdnq quantization right now with both the transformer and the text encoder:

Here are the examples on how to use them:

bitsandbytes

sdnq

Great, thanks for sharing. Between bitsandbytes and sdnq/triton, I'm afraid this won't run on MPS devices, though.

I see -- this issue is just for generic GGUF support then.

Hello @AmericanPresidentJimmyCarter, it appears you're using the LTX (1) class instead of LTX-2's LTX2VideoTransformer3DModel.
The problem is that from_single_file doesn't seem to be supported in LTX-2 just yet @sayakpaul @rootonchair (?)
We had to build our own (dirty) GGUF loader to use quants, and enable LTX-2 to run on more consumer hardware.
On a side note, it would also be great to be able to use Gemma's GGUF as well, since it's actually the biggest bottleneck in running LTX-2. Especially on MPS devices.

I just did inference completely without a diffusers pipeline because the pipelines are consistently difficult to work with. I have limited RAM on some servers and there seems to be no way to hot load/unload the text encoder onto disk when you're done (as opposed to what diffusers forces you to do in pipelines, which is put it back into RAM).

Here's my script using quanto int8 and configurable groupwise layer offload using normal torch hooks.

https://gist.github.com/AmericanPresidentJimmyCarter/b77df7fa37d36c425959845c9e72aaa1

Once the models are cached in int8, it takes about 1min per second of LTX video+audio on the dev model at 1 megapixel. There are some weird things using this model in diffusers though, like hyperintense colour regions sometimes showing up after encoding the latents. I don't know, I give up on the diffusers impl for now.

This is pretty much how we implemented the previous versions of LTX, for the same reasons (memory constraints). We're often surprised by how wasteful / rigid diffusers can be at times. But it also feels like (western) open source models are increasingly designed to upsell APIs, so I guess the huggingface team is doing their best take on a one size fits them all approach, in a fast-evolving technocratic environment.

[dg845]

I have opened a PR to add single file / GGUF support to LTX 2 at #12983.