Repository navigation
SKEEP-006: input front ends and model identity (text lane) - #1332
Merged
michalharakal merged 9 commits intoOct 12, 2026
Merged
michalharakal merged 9 commits into
michalharakal merged 9 commits into
Conversation
One resolvable input front-end contract (tokenizers first; audio and image features on the same contract later) keyed on a string-based model identity with an immutable family registry. Lifts the normative clauses, the module layout, the rollout and the acceptance criteria into the SKEEP track so the decision outlives the implementation PRs. Trigger: public Kotlin API, compatibility/migration policy, runtime integration. Related: SKaiNET-developers#1321, SKaiNET-transformers SKaiNET-developers#471 and SKaiNET-developers#330.
…try (SKEEP-006) New module skainet-model-identity (sk.ainet.model), same target set as skainet-io-core, binary-compatibility validation from the first release: - ModelIdentity(format, architecture, family, variant) — family is a String, not an enum, so downstream repositories add families without engine edits - CheckpointView: the per-format read-only window (metadata + side files) the resolvers see; InMemoryCheckpointView for tests and in-process callers - ModelFamilyDescriptor + immutable ModelFamilyRegistry (compose with plus; no global register(), no service loading); builtIn = llama, qwen, gemma, bert with the name maps the 0.57 string switch knew - ModelFormat, NameMap, RoleTableNameMap and TransformerNameMaps move here from io-core; TransformerNameMaps.Gguf.forArchitecture is deprecated in favour of ModelFamilyRegistry.nameMapFor settings.gradle.kts registers this module and the two front-end modules that follow.
…resolver (SKEEP-006) New module skainet-frontend-core (sk.ainet.frontend): - FrontEnd<I, O>, InvertibleFrontEnd, StreamingFrontEnd/FrontEndStream (specified now, first implementor is the audio lane) - FrontEndSpec: the serialisable twin of a resolved front end (text, audio, image, composite); what frontend.json carries and describe() returns - the text contract in sk.ainet.frontend.text: Vocabulary, SpecialTokens (nullable bos/eos, addBos/addEos, stopIds, declared specials), Tokenizer with the 0.57 one-argument encode/decode semantics plus the addSpecialTokens / skipSpecialTokens overloads, Encoding, BatchTokenizer, TextFrontEnd, PromptRenderer marker - audio and image contracts (PcmAudio, AudioFeatures, AudioFrontEnd, RgbImage, ImageGeometry, ImageFeatures, ImageFrontEnd) so FrontEndSet is typed; kernels follow in their own modules - multimodal composition: InputSegment, MultimodalInput, ModelInputs, PlaceholderSpan, PlaceholderPolicy, CompositeFrontEnd - FrontEndProvider (one per modality module), FamilyFrontEnds (family decoration hook), FrontEndResolver with the fixed resolution order and a loud UnsupportedFrontEndException
…nigram, WordPiece, goldens (SKEEP-006) New module skainet-frontend-text (sk.ainet.frontend.tokenizer) holds every tokenizer algorithm and the TextFrontEnds provider; skainet-io-core keeps deprecated aliases (WARNING; ERROR in 0.59, removed in 0.60) and a forwarding TokenizerFactory for the Java callers that pin its statics. Moved and reworked: QwenByteLevelBpeTokenizer, SentencePieceTokenizer, SpecialTokenSplitter (now the generic added-token splitter), TekkenTokenizer (reachable from the resolver), ByteToUnicode (public). New: MetaspaceBpeTokenizer (tokenizer.json BPE with Metaspace / Prepend and byte fallback — Llama 2, TinyLlama, mmBERT; the encode side of SKaiNET-developers#912), UnigramTokenizer (Viterbi lattice — T5, ALBERT, Gemma exports), WordPieceTokenizer (BERT; encodeWithMetadata, vocab.txt), Normalizer and PreTokenizer built from the tokenizer.json blocks, TokenizerJson and TokenizerConfig readers, TextFrontEnds with the dispatch table (FE-R-02). Fixes from SKaiNET-developers#1321: array-form merges (SKaiNET-developers#1323), every added token matched atomically and counted in vocabSize (SKaiNET-developers#1324), the normalizer block applied with unknown types failing loudly (SKaiNET-developers#1325), Metaspace BPE (SKaiNET-developers#1326). Readers: GgufCheckpointView (io-gguf), HfDirectoryCheckpointView (io-safetensors); StreamingGGUFReader.nameMap() resolves through the family registry; ModelArchitecture.fromGguf and ModelLoaderRegistry deprecated. Tests: the four pure core tests move with the code; dispatch tests ported to the provider; a golden corpus recorded with the Hugging Face tokenizers library 0.23.3 over GPT-2, Qwen2.5, TinyLlama, T5-small, bert-base-uncased and the two public Laya tokenizers (exact id parity incl. specials, emoji, CJK, decomposed characters, leading/trailing whitespace). Binary-compat dumps for the new modules and the first dump of skainet-io-core.
Reference page for the resolver, the dispatch table and the compatibility notes; the kllama tutorial points at it. CHANGELOG lists the added modules, the tokenizer fixes and every deprecation. VERSION_NAME moves to 0.58.0-SNAPSHOT for local integration builds.
…back) The public google/gemma-4-E2B-it tokenizer.json is BPE with a Replace normalizer, a MergedWithPrevious space split and byte fallback; the chat prompt, the atomic chat markers and the bare 'Hi' encode to the Hugging Face reference ids through the resolver — the migration gate of SKEEP-006 for the Gemma runtimes downstream.
…spatchTest
Four backtick test names carried "(#NNNN)" suffixes; Kotlin/Native rejects
'(', ')' and '#' in identifiers, so compileTestKotlinMacosArm64 (and every
other Native test leg) failed before a single test ran. Spelled the issue
references as "- issue NNNN" instead - JVM, JS, wasm and Native all accept
that form.
…izer.json path TextFrontEnds.fromTokenizerJson handed SpecialTokenSplitter a plain surface-form-to-id map, so the lstrip / rstrip / single_word flags parsed from tokenizer.json never reached the matcher. The mmBERT and ModernBERT <mask> token is declared lstrip, and the reference library absorbs the whitespace before it into the special token; the engine instead encoded that whitespace as a standalone metaspace piece (one extra id per mask). SpecialTokenSplitter now takes the AddedToken list and matches through the shared AddedTokenMatcher; the 0.57 map constructor stays as a secondary constructor. Covered by a splitter unit test and by the 40-text mmBERT corpus of the Laya reference fixtures (SKEEP-006).
michalharakal
added
the
skeep
3 tasks done
This was referenced Oct 12, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements SKEEP-006 (tracking issue #1331): one input front-end abstraction for the engine, with the text lane and the model-identity unification in this PR. Audio and image lanes are follow-up SKEEPs.
What changes
Three new modules
skainet-models/skainet-model-identity(sk.ainet.model):ModelIdentity,CheckpointView,ModelFamilyDescriptor, immutableModelFamilyRegistry. Families are strings plus descriptors, not an enum.skainet-frontend/skainet-frontend-core(sk.ainet.frontend):FrontEnd/FrontEndSpeccontract (discriminatormodality), the text, audio, image and multimodal contracts,FrontEndResolverwith theFrontEndProviderSPI.skainet-frontend/skainet-frontend-text(sk.ainet.frontend.tokenizer): the tokenizer algorithms move here out of io-core. New: Metaspace BPE with byte fallback, Viterbi Unigram, WordPiece as aBatchTokenizer, NFC normalizer, added-token matcher withlstrip/rstrip/single_word, array-form merges.TextFrontEndsis the text provider andtokenizer.jsonentry point.io modules:
GgufCheckpointView(io-gguf),HfDirectoryCheckpointView(io-safetensors). io-core keeps deprecated typealiases and a forwardingTokenizerFactory, so existing Kotlin and Java callers compile unchanged (the Java pin tests inskainet-test-javaare untouched).ModelArchitecture.fromGguf,TransformerNameMaps.Gguf.forArchitectureandModelLoaderRegistryare deprecated with replacements.Contract note: one-argument
encode/decodekeep the 0.57 semantics; the special-token behaviour is in the overloadsencode(text, addSpecialTokens)/decode(ids, skipSpecialTokens)(SKEEP clause FE-C-05a).Docs: SKEEP-006 page (status Draft, for the maintainers to flip), reference page
input-frontends.adoc, kllama tutorial update, CHANGELOG,VERSION_NAME→0.58.0-SNAPSHOT.Verification
tokenizers0.23.3: gpt2, qwen25, tinyllama, t5-small, bert-base-uncased, gemma4-e2b, laya-modernbert, laya-mmbert (generator script in the test resources).jvmTestfor the three new modules, io-core, io-gguf, io-safetensors;skainet-test-java;jvmApiCheckper module with refreshed dumps.linuxX64Test,jsTest,wasmJsTest,wasmWasiTestgreen locally for the touched modules.testAndroidHostTest,assembleunder the configuration cache, Apple targets, klib ABI dumps (need macOS). CI on this PR is the first full pass.Downstream
SKaiNET-transformers consumes this as
0.58.0-SNAPSHOTfrom a local Maven build onmichalharakal/SKaiNET-transformersbranchfeature/skeep-006-frontend-consumer: every runtime, CLI and facade resolves its tokenizer through the engine, with the legacy classes deprecated at WARNING. That PR follows the 0.58.0 release.Issues
Closes #1323, closes #1324, closes #1327 (the array-form merges, non-special added tokens and the ModernBERT / mmBERT goldens are part of this branch). Refs #1321, #1322, #1328, #1329, #1331.
Review notes
FrontEndSetis typed; resolver takes providers because core cannot depend on text; family decoration throughFamilyFrontEnds).(,)and#in backtick test names, so issue references in test names are writtenissue NNNN.