Fix/issue 472 array merges

[Sehul07]

Closes #472

[michalharakal]

Thanks for picking this up so quickly, @Sehul07 — and for the walk-the-array approach: branching on " vs [ inside the merges block is exactly the right shape, and it keeps the string form working.

A few things before this can go in:

1. It does not compile yet. findStringEnd and parseStringArray are called at lines 178 and 195, but neither is defined in GGUFTokenizer.kt. They exist only as private members of TokenizerJsonParser in HuggingFaceBPETokenizer.kt, so they are not visible here. Two easy options:

• copy both small functions into the GGUFTokenizer companion next to findMatchingBracket / unescapeJsonString, or
• make TokenizerJsonParser and those two functions internal and call them from here.

CI is approved on your branch now, so the next push will show the compiler result directly. Locally, ./gradlew :llm-core:jvmTest catches it in under a minute.

2. Please revert gradle/wrapper/gradle-wrapper.properties. The timeout/retry change is unrelated to the issue and also drops the file's trailing newline. If the wrapper download was flaky for you, that is worth its own small issue.

3. Test. Step 3 of #472 has it ready: a 4-token vocab {"a":0,"b":1,"c":2,"ab":3} once with merges [["a","b"]] and once with ["a b"]; both must encode "ab" to [3]. One test class, two cases — that is the proof that the two forms are now identical.

4. Housekeeping. A line under Unreleased in CHANGELOG.md, and a short PR description with Closes #472 so the lane closes automatically on merge.

Nice to have, not required: the removed debug println for the vocab size could stay, just to keep the diff focused on the merges.

Ping me when it is pushed and I will re-review the same day.

[michalharakal]

CI finished on this branch (run 38085857435) and shows the compile errors from the review, nothing else:

e: llm-core/src/commonMain/kotlin/sk/ainet/apps/llm/tokenizer/GGUFTokenizer.kt:178:46 Unresolved reference 'findStringEnd'.
e: llm-core/src/commonMain/kotlin/sk/ainet/apps/llm/tokenizer/GGUFTokenizer.kt:195:46 Unresolved reference 'parseStringArray'.
e: llm-core/src/commonMain/kotlin/sk/ainet/apps/llm/tokenizer/GGUFTokenizer.kt:197:50 Cannot infer type for type parameter 'V'. Specify it explicitly.

The third one disappears once parseStringArray returns List<String>. Task: :llm-core:compileCommonMainKotlinMetadata.

[michalharakal]

Thanks @Sehul07, the walk-the-array shape was the right call, so I finished the branch rather than send it around once more:

• added findStringEnd / parseStringArray to the GGUFTokenizer companion (the two helpers the loop calls);
• restored the gradle-wrapper.properties from develop;
• GGUFTokenizerMergesFormatTest: the string form and the array form rank the same merge, plus one case with a b escape;
• a line under Unreleased in CHANGELOG.md.

Your commits stay as they are, mine sits on top; I will squash-merge with you as author once CI is green.

One request for next time: your commits are authored as your-github-sehulvagadiya@gmail.com, which GitHub cannot link to your account, so they do not show on your profile or in the contributors list. Either add that address under Settings → Emails or set git config user.email to the address you registered. It only takes a minute and all your commits become yours retroactively.

Heads-up on the bigger picture: the tokenizer.json path you fixed here is being moved into the engine as a set of input front ends (SKaiNET #1331). Your fix ships in the next transformers patch for everyone on 0.57, and the new code already carries the same case as a golden. If you want a follow-up in the same area, #476 (golden fixtures for every family that loads a tokenizer.json) and #478 (docs) are open and will survive the move unchanged.