fix(tokenizer): implement metaspace prefix handling for issue #473

[Sehul07]

Closes #473

[michalharakal]

Thanks @Sehul07. Two things, one about the branch and one about the fix itself.

The branch carries both lanes. This PR and #482 point at the same three commits, so the two diffs are identical right now. The last commit (af0872f7, the BPEStrategy change) belongs to #474. Could you reset this branch to 04ede43e and force-push, so that #481 only contains the #473 work and #482 only the #474 work? One PR per lane keeps each review small and lets them merge independently.

The #473 fix is not in the diff yet. After the split, what remains here is a comment change and re-indentation in detectFromVocab; addSpacePrefix is still hard-coded to false at the GGUFTokenizer(...) call at the end of fromTokenizerJson. The actual change is small and the value is already plumbed:

1. In fromTokenizerJson, locate the "pre_tokenizer" block and read "prepend_scheme" (for older files also a normalizer of type Prepend with "prepend": "▁", possibly inside a Sequence).
2. Map it: always → true, first → true, never → false, no block at all → false (today's behaviour).
3. Pass that boolean as addSpacePrefix instead of the constant.

The engine does the same sniffing in SentencePieceTokenizer.kt (lines 382–391 at the commit linked in the issue) — copying that shape is fine.

Tests (step 4 of #473): a synthetic ▁-vocab file with prepend_scheme: always encodes "Hi" to the ▁Hi id; the same file with never encodes to the Hi id. If you have Gemma 4 locally, Gemma4TokenizerParityTest must stay green ("Hi" → 10979); if not, just say so and I will run it.

Please also revert the indentation-only change in detectFromVocab so the diff is only the fix, add a CHANGELOG.md line and Closes #473 in the description.

CI runs are approved on your branches now, so you get compiler and test feedback on every push.

[michalharakal]

CI finished on this branch and fails in :llm-core:compileCommonMainKotlinMetadata; the build treats Kotlin deprecation warnings as errors:

e: llm-core/src/commonMain/kotlin/sk/ainet/apps/llm/tokenizer/BPEStrategy.kt:43:21 'expect fun String(chars: CharArray): String' is deprecated. Use CharArray.concatToString() instead.

That line goes away entirely with the table-based preprocess suggested in the #482 review, so no need to patch it separately.

[michalharakal]

Thanks @Sehul07. I took the branch split off your plate and finished the lane here:

• reset the branch to 04ede43e, so this PR carries only the [Lane 2 · kotlin-core] tokenizer.json parity — GGUFTokenizer: honour Metaspace prepend_scheme instead of a hard-coded addSpacePrefix=false #473 work (Fix/issue 474 byte encoding #482 now carries only the BPEStrategy commit);
• detectPrependScheme reads pre_tokenizer.prepend_scheme (always/first → prefix, never → none) and, failing that, a Prepend normalizer with "prepend": "▁", also inside a Sequence; a file that declares neither keeps the old behaviour, so Gemma 4 is unchanged (Gemma4TokenizerParityTest green);
• reverted the indentation-only change in detectFromVocab, kept your comment;
• GGUFTokenizerPrependSchemeTest: always, first, never, no block, legacy Prepend normalizer;
• CHANGELOG.md line.

With #480 merged alongside, the mmBERT file now encodes the #473 acceptance sentence exactly as the reference: "Which department handles this request?" → [12236, 9888, 26446, 736, 3853, 235336].

Squash-merge with you as author once CI is green. Same note as on #480 about the commit email, it applies here too.