Repository navigation
feat(io): SKEEP-007 Tier 2 -- quantize at load for SafeTensors and GGUF - #1353
Merged
michalharakal merged 4 commits intoOct 16, 2026
Conversation
…ce kernel Adds the SKEEP-007 proposal (the browser lane: SIMD128 sidecar kernels, quantize-at-load) and lands its common-code tier: - matmulWeightTransposed on a canonical packed weight prefers an exact kernel for the row-major view, then the one-time cached relayout into kernel feed order served by the scalar packed kernels, and only then the decoding reference kernel. Every non-JVM target took the reference path on every call before. - a dense heap FP32 [out, in] weight is multiplied straight from its rows (4x4 tiles, one sequential accumulator per output, bit-identical to transpose-then-matmul) on tiers without a vectorised [k, n] kernel; preferDirectWeightTransposed() keeps the JVM and Apple tiers on their cached transpose. - NarrowFloatDenseTensorData implements RowDequantSource so 16-bit token tables are read row by row instead of widened at load. - ByteArrayRandomAccessSource in skainet-io-core (single array or chunks); JsBlobRandomAccessSource stops logging every read. - benchmarks/wasm-bench: a Kotlin/Wasm executable timing the matmul shapes of the shipped models; the Node baseline is recorded in the SKEEP page.
EncodingRequest.RequantizeTo has existed since SKaiNET-developers#1109 and the planner prices it, but only the GGUF loader executed it and only for BITNET_PLANES. Every Hugging Face checkpoint was dense FP32 on every target. - Q8_0Quantizer (ggml-faithful) beside Q4_0Quantizer, and the Requantizers registry (Q8_0, Q4_0, BITNET_PLANES) that loaders consult. - SafeTensorsParametersLoader / ShardedSafeTensorsParametersLoader take weightForm / weightFormFor; RequantizeTo packs every rank-2 float matrix whose rows block, leaves vectors and non-blocking rows dense, honours KERNEL_FEED, and refuses unknown encodings at construction. - The GGUF loader's planes-only requantize path serves every registered encoding, with feed-order relayout for Q8_0/Q4_0. - PlannerProfile.quantizeDenseTo + resolver rule (OUT_IN pinned) so a browser or phone profile can ask for it when a kernel can feed the result. - skainet-quantize in skainet-tensor-tools: SafeTensors -> GGUF with the matrices quantized offline, names kept, --keep prefixes written as F16.
Kotlin/JS keeps a Float local in double precision while the transpose-then- matmul path rounds through a Float32Array on every step, so the direct row-dot differs from it in the last bits there, as the golden tests already acknowledge for that target. The tiled-vs-definition tests keep pinning the bits where bits are stable.
…p-007-quantize-at-load
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SKEEP-007 Tier 2, stacked on #1352 (Tier 0): quantize at load, for every loader, not only GGUF.
Why
EncodingRequest.RequantizeTo(encoding)has been part ofWeightFormsince #1109 and the memory planner prices it, but only the GGUF loader executed it, and only forBITNET_PLANES(#1150). Every Hugging Face checkpoint — SafeTensors, F32/F16/BF16 — was dense FP32 on every target. In a browser that is the difference between a 322M-parameter model fitting (about 340 MB as Q8_0) and not fitting (1.3 GB FP32 plus the loader's buffers); on the JVM and Android it is the difference between the packed kernel packs and the dense path.What changes
Q8_0Quantizer(skainet-lang-core, beside the existingQ4_0Quantizer): ggml-faithfulquantize_row_q8_0—d = amax / 127as an FP16 scale,round(x / d)codes clamped to ±127, row-major 34-byte blocks. Q8_0 first because eight-bit codes keep a classifier's decisions where four bits may not, and every tier already has its kernel.Requantizers: the registry fromTensorEncodingto quantizer (Q8_0, Q4_0, BITNET_PLANES). Loaders consult it instead of hard-coding one format; an encoding without an entry is refused at construction as before.SafeTensorsParametersLoader,ShardedSafeTensorsParametersLoader): newweightForm/weightFormForparameters, validated up front (SafeTensorsWeightForms).RequantizeToapplies to rank-2 float tensors whose rows block by the encoding's block size; vectors, norms and non-blocking rows load as before, so a uniformRequantizeTo(Q8_0)is safe over a whole checkpoint.KERNEL_FEEDrelays the fresh blocks into input-block-major order.WeightShapeOrientationis moot for SafeTensors (Linearweights are[out, in]);MAPPEDmeans heap, as the GGUF loader documents.requantizeRequested,requantizedTensor) to every registered encoding, with feed-order relayout for Q8_0/Q4_0. The parity test's refusal case moves from Q8_0 to Q4_K, which still has no quantizer.PlannerProfile.quantizeDenseToand the resolver rule: a profile can ask for dense sources to be requantized when a kernel on the target can feed the encoding; the resolver pinsOUT_INfor that case (per-row scales) and falls back to keeping dense bytes when no kernel exists.skainet-quantize(skainet-tensor-tools, Gradle taskquantize): SafeTensors → GGUF with the weight matrices quantized offline, tensor names kept,--keepprefixes written dense as F16 (or F32). This is for downloads that should not carry F16: the browser then uses the GGUF loader, which already honoursWeightForm.Tests
Q8_0QuantizerTest: ggml block layout, round trip within one code step, zero block, blocking rules, registry routing and refusal.QuantizeDenseResolverTest: requantize when asked and feedable, keep dense otherwise, packed sources untouched.SafeTensorsRequantizeTest: uniform form packs matrices and leaves vectors/non-blocking rows dense (F32 and F16 sources),KERNEL_FEEDdecodes to the same values, per-tensor overrides, up-front refusals.GgufRequantizeTest: dense F32 matrix → Q8_0 within one step; feed order equal in values.WeightFormLoaderParityTestupdated.QuantizeCliTest: SafeTensors → Q8_0 GGUF → streaming loader round trip with a kept F16 table and a dense bias.Stacking and verification
Stacked on #1352: the first commit here is that PR's Tier 0 commit, and this one should merge after it (the diff collapses to the Tier 2 commit once #1352 lands). Draft until then.
scripts/pr-gate.sh --quick(every module'sjvmTest+apiCheck): passed.skainet-lang-core,skainet-io-safetensors,skainet-io-ggufJVM suites andskainet-tensor-toolstests: green.skainet-lang-coreAPI dump regenerated (additive:Q8_0Quantizer,Requantizers,PlannerProfile.quantizeDenseTo).