ADFA-5188 | Enable KV cache quantization and flash attention

[jatezzz]

Description

This PR implements KV cache quantization (q8_0) and enables flash attention in the llama.cpp context parameters to drastically improve memory efficiency and generation speed.

• What: Configures the context parameters type_k and type_v to use GGML_TYPE_Q8_0 and enables flash attention (LLAMA_FLASH_ATTN_TYPE_AUTO).
• How: These settings are exposed through the configure-before-load pattern. If the native context creation fails (e.g., the model's head width is incompatible with the quantized block size), it gracefully falls back to the previous defaults: f16 KV cache and flash attention disabled.
• Why: Storing the KV cache as q8_0 halves its byte size, allowing for much longer conversations before context is dropped and drastically reducing mid-generation crashes on 4–6 GB RAM devices. Flash attention mitigates generation latency on longer contexts.

Details

Logs confirming n_ctx initialization, cache type used (q8_0 vs f16), and fallback activations.

Flash Attention Enabled

• Model: Qwen3 0.6B Q8_0
• Device RAM: ≥8 GB

Q8_0 KV cache

• Model: Llama 3.2 1B Instruct
• Device RAM: ≤3 GB
• Prompt: ~10K Tokens

Ticket

ADFA-5188

Observation

This implementation works in tandem with dynamic n_ctx sizing and should be validated alongside ADFA-5187, as the memory measurement relies on both features working concurrently.

#75 Needs to be merged first

[claude]

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

_{Tip: disable this comment in your organization's Code Review settings.}