No longer able to apply LoRAs

[SkutteOleg]

When I try to apply LoRA to a quantized model, I get the following error:

sd.exe -m sd_xl_turbo_1.0.q8_0.gguf -p "a lovely cat<lora:res-adapter-v2-sdxl:1>" --taesd taesdxl.q8_0.gguf --steps 1 --cfg-scale 1
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce GTX 1660 SUPER, compute capability 7.5, VMM: yes
[INFO ] stable-diffusion.cpp:184  - loading model from 'sd_xl_turbo_1.0.q8_0.gguf'
[INFO ] model.cpp:786  - load sd_xl_turbo_1.0.q8_0.gguf using gguf format
WARNING: Behavior may be unexpected when allocating 0 bytes for ggml_calloc!
[INFO ] stable-diffusion.cpp:224  - Version: SDXL
[INFO ] stable-diffusion.cpp:255  - Weight type:                 q8_0
[INFO ] stable-diffusion.cpp:256  - Conditioner weight type:     q8_0
[INFO ] stable-diffusion.cpp:257  - Diffsuion model weight type: q8_0
[INFO ] stable-diffusion.cpp:258  - VAE weight type:             f32
[INFO ] tae.hpp:204  - loading taesd from 'taesdxl.q8_0.gguf', decode_only = true
[INFO ] model.cpp:786  - load taesdxl.q8_0.gguf using gguf format
WARNING: Behavior may be unexpected when allocating 0 bytes for ggml_calloc!
[INFO ] tae.hpp:226  - taesd model loaded
[INFO ] stable-diffusion.cpp:486  - total params memory size = 3763.23MB (VRAM 3763.23MB, RAM 0.00MB): clip 835.53MB(VRAM), unet 2925.36MB(VRAM), vae 2.34MB(VRAM), controlnet 0.00MB(VRAM), pmid 0.00MB(VRAM)
[INFO ] stable-diffusion.cpp:490  - loading model from 'sd_xl_turbo_1.0.q8_0.gguf' completed, taking 3.27s
[INFO ] stable-diffusion.cpp:517  - running in eps-prediction mode
[WARN ] stable-diffusion.cpp:617  - In quantized models when applying LoRA, the images have poor quality.
[INFO ] stable-diffusion.cpp:635  - Attempting to apply 1 LoRAs
[INFO ] model.cpp:789  - load res-adapter-v2-sdxl.safetensors using safetensors format
[INFO ] lora.hpp:33   - loading LoRA from 'res-adapter-v2-sdxl.safetensors'
D:\a\stable-diffusion.cpp\stable-diffusion.cpp\ggml\src\ggml-backend.c:224: GGML_ASSERT(buf != NULL && "tensor buffer not set") failed

This worked before. Same issue was encountered here: #291 (comment)
This commit 29ec316 appears to have broken something.

If it helps, here are the LoRAs and models I tried this with:

• res-adapter LoRA
• SDXL-Turbo (Q8_0)
• SDXL-Lightning (Q8_0)

[grauho]

I haven't had time to review all the new flux code but I solved this issue by adding "ggml_build_forward_expand(gf, zero_index);" to the end of build_lora_graph. My logic was that all the other places that use set_backend_tensor_data seem to resolve with that but for whatever reason the lora loading did not. I don't know if this is the intended solution but it did get LoRAs loading again for me.

[leejet]

It should be fixed now.

[SkutteOleg]

Thanks!