Repository navigation
fix(vlm): load non-affine VLM weights with the right quant mode and group_size - #334
Merged
Merged
Conversation
…roup_size
Vision encoders (and any caller that does not thread an explicit
quantization mode) loaded weights through `from_weights`, which hardcoded
the affine mode. On block-float VLMs that aborted in quantized_matmul
("Biases must be provided for affine") because mxfp4/nvfp4/mxfp8 weights
carry no zero-point biases.
Two reconciliations make these load correctly:
- Auto-detect the mode in both QuantizedEmbedding::from_weights and
UnifiedLinear::from_weights: the presence of a `.biases` tensor means
affine; its absence means a block-float scheme, distinguished by bits
(8 -> mxfp8) and group_size (16 -> nvfp4, else mxfp4).
- Reconcile group_size from the tensor shapes for block-float modes in
UnifiedLinear::from_weights_with_mode. For affine, bits are inferred
with the trusted group_size (unchanged). For block-float modes bits are
fixed by the mode, so derive group_size instead:
in_features = packed_in * (32/bits) = num_groups * group_size. A mixed
export such as minicpm-v-4.6-mxfp4 stores some weights at group_size 32
while config.json reports 64; the old loader passed 64 straight through
and mismatched in the matmul.
minicpm-v-4.6-mxfp4 now produces a coherent image description and
benchmarks at 139.85 tok/s decode on GB10. Affine models and correctly
exported block-float models are byte-identical (inferred params equal the
passed ones): qwen2.5-7b-4bit 53 tok/s and gpt-oss-20b-mxfp4 78 tok/s
unchanged.
inureyes
added a commit
that referenced
this pull request
Jun 17, 2026
…rop qwen2.5-vl-3b (#335) Follows the loader fix (#334). Updates the GB10 0.3.1 benchmark CSVs and docs: - minicpm-v-4.6-mxfp4: now passes after #334. Text 347.59 prefill / 142.14 decode tok/s, image input 625.47 / 138.70 (48 tok). Added to the image-input table and flipped to a pass in the text table. - qwen2.5-vl-3b: removed. It was a broken duplicate of qwen2.5-vl-3b-4bit (the model dir was deleted); the working 4bit variant stays. Both CSVs go 148 -> 147 rows. - Reclassified six by-design non-failures from fail (red x) to N.A. (white circle): docling-layout-heron and granite-speech (not standalone text generators) and the four MTP/DFlash drafters (gemma-4-12b/31b assistant, qwen3.5-27b/4b-dflash) that need a target model. The raw CSV FAIL:bench markers stay (they genuinely produce no text-gen output); the N.A. tag is a doc-layer classification. GB10 0.3.1 summary is now 147 total: 136 pass, 0 code failures, 9 N.A., 2 too-large (qwen3-next-480b, deepseek-v3, shown red x, OOM), 54 VLM image rows. Legend and the by-cause section were rewritten to match; index counts and the benchmark-CSV table updated in step.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Vision encoders (and any caller that does not thread an explicit quantization mode) load weights through
from_weights, which hardcoded the affine mode. On block-float VLMs that aborts inquantized_matmul(Biases must be provided for affine) because mxfp4/nvfp4/mxfp8 weights carry no zero-point biases. minicpm-v-4.6-mxfp4 was the visible casualty.Two reconciliations fix it:
QuantizedEmbedding::from_weightsandUnifiedLinear::from_weights: a.biasestensor means affine; its absence means a block-float scheme, distinguished bybits(8 → mxfp8) andgroup_size(16 → nvfp4, else mxfp4).UnifiedLinear::from_weights_with_mode. For affine, bits are inferred against the trustedgroup_size(unchanged). For block-float modes the bits are fixed by the mode, sogroup_sizeis derived from the shapes instead:in_features = packed_in * (32/bits) = num_groups * group_size. A mixed export such as minicpm-v-4.6-mxfp4 stores some weights atgroup_size32 whileconfig.jsonreports 64; the old loader passed 64 straight through and mismatched in the matmul.Verification (GB10, CUDA)
Biases must be provided for affine, then group_size mismatch)For affine models and correctly exported block-float models the inferred params equal the passed ones, so behavior is byte-identical.