Gemma 4 generates correct text again, and FunctionGemma 270M function calling works

[michalharakal]

Makes Gemma 4 work, and makes official FunctionGemma 270M function calling work. Fixes the failure recorded in ISSUE-gemma4-broken-generation.md (degenerate <turn|> output at ~0.04 tok/s while Llama 3.2 ran fine through the identical runtime). Refs #325, #330, #342, #346.

The CLI now answers the prompt from that report:

$ kgemma gemma-4-E2B-it-Q4_K_M.gguf "Translate 'hello world' to German" 48 0.0 --chat
The translation of "hello world" to German is **Hallo Welt**.

word for word what llama.cpp produces, stopping naturally on a stop token.

<turn|> was never mangled

It is Gemma 4's real turn-close token, id 106 — Gemma 4 replaced Gemma 2/3's <start_of_turn>/<end_of_turn>, and Gemma4TokenizerParityTest already pinned that against HF ids. The model was genuinely predicting it, for six reasons that stacked.

Why it was slow. The kgemma CLI never installed the kernel packs (kllama got this in #354; kgemma was skipped), so every matmul ran on the decoding reference kernel. token_embd was force-dequantized to dense FP32, taking the tied lm_head off the packed chain — llama keeps it packed and rewraps via PackedRowDequantTensorData, and that pattern transplants directly.

Why the output was garbage.

• BOS defaulted to 1 — which is Gemma's <eos> — so every generation was prefilled with an end-of-sequence token. The GGUF path never parsed token ids at all.
• The stop set held one id where Gemma 4 ships three ([1, 106, 50]); the plain CLI loop had no stop check whatsoever and ran to the step limit.
• No chat template in plain CLI mode on an instruction-tuned checkpoint.
• RMS-norm epsilon was hardcoded rather than read from metadata.

And two architecture parameters were wrong, each right for one family and applied to both:

• Gemma 4's attention scale is 1.0, not 1/sqrt(head_dim) — its QK-norm already controls the dot-product magnitude, so the extra division flattened the softmax ~16x. At the second token we put 41% of attention weight on the current token where llama.cpp puts ~1%. It had been 1.0 and was changed to the generic default on the theory that 1.0 "over-sharpened"; that conclusion was drawn while the V-norm bug below was corrupting the same attention block, so the evidence pointed the wrong way.
• V-norm is gemma4-only. gemmaNetwork() forced it for every checkpoint, but gemma3 has none — llama.cpp's graph is Vcur → MUL_MAT → RESHAPE → VIEW with nothing between, and the comment beside the flag already said so. Q and K stayed bit-exact while V was rescaled, so the error was invisible at position 0 and compounded: FunctionGemma matched llama.cpp at prefill n=3 and diverged from n=8.

Both are now conditional on the checkpoint's declared architecture rather than assumed constant across the family.

FunctionGemma 270M — official function calling

The existing gemma-iree scaffolding implements an Octopus-v2 <tool_N> fine-tune, not the released model. Added alongside it, untouched: FunctionGemmaOfficialChatTemplate and FunctionGemmaOfficialToolCallParserStrategy, ported from the Jinja chat_template embedded in the release GGUF — including its <escape>-token string convention, which quotes would have broken. FunctionGemma.Style.OFFICIAL + callWithTools, and top-k/top-p in SamplingUtils.

It emits and parses call:get_weather{location:<escape>Paris<escape>}, and matches llama.cpp's greedy argmax at all ten sampled prefill lengths (3, 8, 16, 24, 32, 48, 64, 72, 80, 85).

Speed

	before	after
E2B decode	443 ms/token	189 ms/token
E2B prefill	432 ms/token	186 ms/token
FunctionGemma load	2254 ms	1672 ms

From a fused eager RMS norm (that phase was 64% of decode — the declared FusedRmsNormOps escape hatch is implemented by no backend, so the fast path never fired; now 0.8%) plus the engine-side kernel work in SKaiNET#1221.

Gemma4WeightLoader also gains the dtypePolicy / keepBf16Native / keepF16Native capability llama's loader already had — a gemma3 checkpoint no longer has to be widened to FP32. Realising its decode benefit needs narrow-float routing in matmulWeightTransposed, noted in the commit.

Gates

Gemma4ChatGoldenTokenTest is the new one worth knowing about: it pins tokenization against llama.cpp byte-for-byte, then asserts the greedy first token at a position where the reference is ~99% confident — a far better probe than the old "Hi" prompt, where llama.cpp's own distribution is nearly flat and the argmax flips on ordinary quantization noise.

Green: that gate, Gemma4ChatModelGgufSmokeTest, Gemma4TokenizerParityTest, FunctionGemmaOfficialGgufTest (tokenizer parity + template pin + e2e tool call), FunctionGemmaOfficialFormatTest, KernelBootstrapOrderingTest, plus llm-core / llm-inference-gemma / gemma-iree suites.

Model-gated harnesses are kept, since the numbers above should be reproducible rather than folklore: prefill-vs-decode profiles, per-module self-timing via forward hooks (this is what found that 73% of decode sat in two dense FP32 projections), and llama.cpp layer-parity probes driven by llama-eval-callback. Their doc comments record the row-indexing gotcha that briefly made a correct implementation look broken.

Requires SKaiNET#1221 for the engine-side fixes; built against it via -PuseLocalSkainet=true.