[Bug] Lens doesn't work on Vulkan without Flash Attention

[wbruna]

Git commit

29ab511 (master-655-29ab511)

Operating System & Version

Debian 13, Mesa 25.2.6

GGML backends

Vulkan

Command-line arguments used

sd-cli --backend Vulkan1 --diffusion-model lens_turbo-Q8_0.gguf --llm gpt-oss-20b-mxfp4.gguf --tae taef2.safetensors -p flower --cfg-scale 1 --steps 4

Steps to reproduce

On Vulkan, the bf16 safetensors file just produce NaNs:

With a direct q8_0 quant:

The following change helps with the q8_0:

diff --git a/src/lens.hpp b/src/lens.hpp
index b5ff068..1cadd29 100644
--- a/src/lens.hpp
+++ b/src/lens.hpp
@@ -37,6 +37,10 @@ namespace Lens {
             auto w2 = std::dynamic_pointer_cast<Linear>(blocks["w2"]);
             auto w3 = std::dynamic_pointer_cast<Linear>(blocks["w3"]);
 
+            if (sd_backend_is(ctx->backend, "Vulkan")) {
+                w2->set_scale(1.f / 128.f);
+            }
+
             auto gate = ggml_silu(ctx->ggml_ctx, w1->forward(ctx, x));
             auto up   = w3->forward(ctx, x);
             x         = ggml_mul(ctx->ggml_ctx, gate, up);

I've tested all scales from 1/16 to 1/512, but quality doesn't seem to improve. It also doesn't help with the bf16.

What you expected to happen

The same gen with --backend ROCm,clip=Vulkan:

What actually happened

Bad images; low quality even with the scaling.

Logs / error messages / stack trace

No response

Additional context / environment details

Device is:

ggml_vulkan: 1 = AMD Radeon RX 7600 XT (RADV NAVI33) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat

[ZahyGabi]

I have the same problem with CPU-generation, and didn't help my naive modification of the above patch (s / Vulkan / CPU /

[leejet]

I can’t reproduce this on my device using the Vulkan backend. You can try setting all ggml_ext_linear scales to 1/128.f to check whether the issue is related to the scale of the Linear layers. If that confirms the problem, then use bisection to identify which Linear layer is causing it.

diff --git a/src/ggml_extend.hpp b/src/ggml_extend.hpp
index 9178a31e..d1641363 100644
--- a/src/ggml_extend.hpp
+++ b/src/ggml_extend.hpp
@@ -1008,6 +1008,7 @@ __STATIC_INLINE__ ggml_tensor* ggml_ext_linear(ggml_context* ctx,
                                                ggml_tensor* b,
                                                bool force_prec_f32 = false,
                                                float scale         = 1.f) {
+    scale = 1 / 128.f;
     if (scale != 1.f) {
         x = ggml_ext_scale(ctx, x, scale);
     }

[wbruna]

It turns out --fa helps a lot: I get perfect images with the bf16 checkpoint on Vulkan too. For the q8_0 quant, I get garbage gens, but the w2 scaling gives a low quality result similar to the test above without --fa; so a more careful quant would probably be enough.

But without --fa, the suggested scale = 1 / 128.f change still gives bad results:

	bf16	q8
ROCm
Vulkan
Vulkan, --fa, w2 scaling

Strangely, scaling everything to 1 / 128 is worse than scaling w2 only.

[wbruna]

Found tensor rules that make Vulkan work for me with or without flash attention:

\.transformer_blocks\..*\.((txt_mod\.1)|(img_mod\.1)|(txt_qkv)|(img_qkv)|(attn\.img_qkv)|(attn\.txt_qkv)|(w[13]))\.weight$=q8_0

no --fa	--fa

[CMay]

Can confirm that something is wrong with Vulkan when running Lens or Lens Turbo even with --diffusion-fa with BF16 weights. At least on my machine.

512x512 works with cfg 1.0 and 4 steps. If you slowly walk up the resolution, it looks coherent up until about 832x832 where it blows out. 512x1024 works, 1024x512 works. At 1024x1024 the image is mostly artifact, but occasionally you see portions of structure in there. None of my tweaking helped any.

Switched to the ROCm 7.1.1 build without changing anything else. 1024x1024, 1440x1440 and 2560x1440 all look perfect.