[Bug] LLM text encoders ignore --fa: materialized attention hits Vulkan's 4 GiB buffer limit at 4096 tokens and falls back to CPU

[linxuhao]

Git commit

cc515a0 (master); also reproduced on 59c23bc

Operating System & Version

Ubuntu 24.04, Mesa 25.2.8 (RADV)

GGML backends

Vulkan

Command-line arguments used

Text encoding with a Qwen3-VL-32B LLM conditioner (MiniMax-H3), flash attention enabled (--fa, i.e. set_flash_attention_enabled(true) on the conditioner), prompt/vision input longer than 4096 tokens.

Steps to reproduce

Encode a conditioning input of ~4065 tokens, then ~4214 tokens, on a Vulkan device whose maxStorageBufferRange is 4 GiB and which does not expose VK_EXT_shader_64bit_indexing (RX 7900 XTX / RADV here).

What you expected to happen

With --fa, the LLM encoder uses ggml_flash_attn_ext, and encode time grows roughly linearly past 4096 tokens.

What actually happened

src/model/te/llm.hpp calls

x = ggml_ext_attention_ext(ctx, q, k, v, num_heads, attention_mask, true, false);

so the flash-attention flag is never used for LLM encoders (it has been false since the Qwen-Image text encoder was added in #851; other models such as ernie_image.hpp, wan.hpp and the VAEs pass ctx->flash_attn_enabled). The attention matrix [N*num_heads, n, n] F32 is materialized: for 64 heads that is 4034 MiB at 4065 tokens and 4335 MiB at 4214 tokens. Above maxStorageBufferRange (4 GiB) ggml-vulkan's supports_op rejects the tensor and the scheduler runs those ops on the CPU.

Measured on RX 7900 XTX, Qwen3-VL-32B Q4_K_M, same process, only prompt length changed:

tokens	encode	peak VRAM
4065	11.5 s	22.7 GB
4214	239 s	19.2 GB (lower: ops moved to CPU)
5081	357 s	19.9 GB

With the attention call switched to flash attention:

tokens	encode
4214	9.7 s
5081	12.0 s
9145	24.2 s
15644	49.7 s

On backends without the 4 GiB storage-buffer limit (CUDA, ROCm) there is no cliff, but the materialized matrix still costs O(n²) memory (≈16 GiB at 8k tokens for 64 heads).

Numerical note: at 2033 tokens, relative to a CPU-backend encode with the materialized path, Vulkan materialized differs by relative RMS 0.76 and Vulkan flash attention by 0.71 (CPU flash attention: 0.16). This model's unnormalized last hidden layer has very large activations, so backend differences dominate; flash attention did not make the result further from the CPU reference.

Additional context / screenshots

Found while encoding long multimodal (video) references; happy to provide more measurements.