Packed-quant byte-order (block layout) is an unwritten, contradictory contract across the engine and downstream converters #973

Summary

The 0.40.1 hotfix (#969, closing #968) is locally correct and globally wrong: it fixed ops.transpose for one population of packed-quant tensors (canonical/row-major bytes) while silently corrupting the other population that legitimately exists today (already kernel-native bytes). This just caused a real downstream regression, independently reproduced and fixed at the call-site level in SKaiNET-transformers#311 — but that PR is a point fix, not a fix for the underlying gap. Filing this to track the actual root cause: nothing in the type system, API contract, or test suite says which physical byte order a Q*BlockTensorData.packedData holds, and the engine's own code contradicts itself about it in at least two places.

Full analysis, evidence index, and a proposed structural design live in packed-quant-layout-postmortem.md, written this session while investigating the transformers-side regression (happy to attach/paste the full doc into this issue or a linked gist if useful — trimmed here to the engine-actionable parts).

The regression that surfaced this (already fixed downstream)

Two block orderings exist for a 2-D [out, in] packed weight's quant blocks (grid = outDim × blocksPerRow):

• canonical / row-major: flat block index o * blocksPerRow + b. What GGUF stores, what Q4_0Quantizer emits, what toFloatArray() assumes.
• kernel-native / input-block-major: flat block index b * outDim + o. What every heap matmul kernel (scalar, Panama, native C, JNI) actually reads.

#969's fix makes ops.transpose perform a real canonical→kernel-native block-grid permutation — correct for canonical input. But SKaiNET-transformers' classic packed path (BlockQuantPacking.pack(), pre-#311) eagerly relayouts GGUF bytes to kernel-native at load time, relying on the pre-0.40.1 shape-swap-only transpose to pass them through unchanged at forward time. On 0.40.1, that transpose now applies the canonical→kernel-native permutation to bytes that are already kernel-native → double permutation → garbage (not a crash — transpose(transpose(W)) ≠ W for non-square block grids, so nothing detects it).

Both sides were "right" by their own local documentation. The engine's own type kdoc (Q5_1TensorData.kt:24-27, still true in 0.40.1) states blocks are input-block-major and the CPU-ops lazy transpose is a pure shape swap. The 0.40.1 transpose's own comment says the opposite. The engine now contradicts its own type contract, and no test crosses the repo boundary to catch it — engine CI builds canonical-only fixtures (NativeLazyTransposeGroundTruthReproTest from #968), transformers CI (pre-#311) built kernel-native fixtures. Each suite proved its own convention; neither was wrong about its own inputs.

Census: 7 contradicting conventions already live in this repo alone

Also: the engine's only documented canonical→kernel-native MemSeg relayout lives in the downstream repo (JvmQuantizedVectorKernels.kt:288 literally names GemmaMemSegConverter as responsible for honoring the kernel's layout) — layout knowledge owned by the wrong repo.

Two more latent hazards:

• transpose(transpose(W)) ≠ W on 0.40.1 for non-square block grids — nothing detects double-transposition.
• Performance regression for engine-native users: Linear.onForward (Linear.kt:76-77) does weight.t() per forward; on 0.40.1 that's now an O(bytes) copy of the whole packed weight on every forward call, uncached.

(SKaiNET-transformers has its own parallel census — three different conventions applied to the same GGUF K-quant tensor depending on which converter loads it, an inlined Apertus relayout that duplicated and diverged from the shared packer, a mixed-convention legacy weights object disambiguated by an unsafe shape heuristic. Tracked/fixed on that side via #311; not re-listed here since it's downstream-repo scope.)

The deeper semantic problem

ops.transpose on a block-quantized tensor is not a representable operation. Blocks quantize 32/256-element runs along the input dimension; a true transpose needs runs along the other axis, i.e. requantization. What the engine calls "packed transpose" is really a layout conversion into kernel feed order wearing transpose's name and a swapped shape label that lies about the data:

• the result's shape=[in,out] + bytes have no self-consistent packed interpretation → toFloatArray() on it is garbage (contradiction Flatten #7 above);
• the operation can't validate its input, because the input's byte order isn't knowable from the type alone (root cause of both Packed-quant lazy transpose silently corrupts matmul for any weight with >1 block/row (all formats, not just Q5_0/Q5_1) #968 and this issue);
• it isn't an involution (see the double-transpose hazard above);
• reference implementations don't have this operation: ggml/llama.cpp's mul_mat computes x·Wᵀ natively against [out,in] weights — quantized data is never "transposed."

Proposed direction (not a full spec — for discussion)

Principle: one convention, one owner, explicit in the type, loud on violation.

1. Make block order type-visible. Follow the engine's own narrow-float precedent (NarrowFloatInputMajorTensorData): a packed weight converted to kernel feed order becomes a distinct type (e.g. KernelPackedWeightData, logical shape stays [out,in]), not the same class with secretly different bytes. Plain Q*BlockTensorData then has exactly one meaning: canonical row-major. Kernels/dispatch accept only the kernel-packed type; toFloatArray()/get() on it either de-permute correctly or refuse.
2. Remove ops.transpose from the packed hot path. Give the engine a weight-transposing matmul as the primitive (matmulWT(x, W[out,in]), the ggml/BLAS-op(B) shape). ops.transpose on packed data becomes a loud error pointing at the primitive (or an explicit dequant). This deletes both the semantic lie and the per-forward O(bytes) copy fix: physically reorder packed-quant blocks in ops.transpose (all-zero matmul, not just Q5_0/Q5_1) #969 introduced.
3. Engine owns prepacking. Move the row-major→block-major relayout into the engine as prepackForMatmul(weight): KernelPackedWeightData, so every downstream converter (transformers, and any future consumer) calls one engine-owned function instead of maintaining private copies that can silently diverge (as Apertus' did).
4. Normalize orientation at the load boundary. One documented rule: every 2-D matmul weight enters the engine as logical [out,in] (HF convention), canonical bytes; the GGUF loader reverses ne dims (fixes contradiction MaxPooling2D #6).
5. Unify or annotate the carrier split. Heap vs MemSeg currently imply opposite byte orders (contradiction MNIST data set loader #3) — make that explicit under (1) instead of implicit-by-marker-interface. Delete the dead canonical Q4_K MemSeg kernel (contradiction convolution layer (Conv2D) #4). Correct the stale kdocs (Q5_0/Q5_1TensorData, kernel SPI "Matches packedData", CHANGELOG:1038).
6. Guardrails: a cross-repo contract-fixtures artifact (canonical + kernel-native builders per format, engine-published, downstream-consumed) so a byte-layout change can never again ship as a green-CI hotfix; a written semver policy that packedData byte semantics are public API (breaking change = minor/major, never a patch); one normative doc (docs/packed-weight-layout.md) that every kdoc links to instead of restating.

Why this needs to be a real issue, not just closed alongside #311

#311 unblocks the immediate downstream pin bump by making the transformers-side classic path stop relayouting (i.e. it adapts to the engine's 0.40.1 contract). That's a valid tactical fix, but it does nothing about contradictions #2–#7 above, which live entirely in this repo and can bite the next caller (engine-native user, a different downstream converter, a future format) regardless of what transformers does. The guessing-game nature of "which byte order does this instance actually hold" is the actual bug; #968/#969 and this transformers regression are two symptoms of it six weeks apart.

Not recommended: another byte-order guess/patch in transpose(). Any guess re-creates the same trap for whichever producer population it doesn't anticipate — see this exact pattern already happening twice.