Repository navigation
Is it not possible to load the SD3 Medium model in the diffusers format? #699
Description
Activity
Yep, it's not possible right now, and it looks very annoying to implement. For sd1.x/sdxl models, the diffusers format only changes the names of each tensor, so it's pretty easy to just map the names to the expected ones. But for DiT models like Flux and sd3/sd3.5, the diffusers format actually slightly modifies the architecture of the models, by splitting some matrices/tensors into smaller chunks (for example, the qkv matrices for the attention blocks are split into q, k and v matrices). That means to support it we would need to either concatenate the parts together at loading time, which would require a big refactor of the loading logic, or support both the split and unsplit variants of the models, which is also annoying, and would make the code even more of a spagetthi mess.
I added support for diffusers LoRAs, and it was already kind of a pain, but doing the same for the full models will be harder.
Reacted by vmobilisThis issue has had no activity for one year. The latest version of the code may already have fixed the problem.
If the issue still exists in the latest version, you can reopen this issue at any time with updated reproduction details.
When I tried to load the SD3 Medium model in diffusers format, I encountered the following error:
[INFO] stable-diffusion.cpp:232 - loading VAE from '/base_model/diffusers/vae/diffusion_pytorch_model.safetensors'
[INFO] model.cpp:908 - loading /base_model/diffusers/vae/diffusion_pytorch_model.safetensors using safetensors format
[ERROR] stable-diffusion.cpp:240 - failed to determine SD version from file: ''
new_sd_ctx_t initialization failed
The same error occurred with both CLIP and T5-DiT components