Skip to content

fix(vlm): load non-affine VLM weights with the right quant mode and group_size - #334

Merged
inureyes merged 1 commit into
mainfrom
fix/vlm-mxfp4-quant-mode
Jun 17, 2026
Merged

inureyes merged 1 commit into
mainfrom
fix/vlm-mxfp4-quant-mode

Conversation

@inureyes

Copy link
Copy Markdown
Member

Summary

Vision encoders (and any caller that does not thread an explicit quantization mode) load weights through from_weights, which hardcoded the affine mode. On block-float VLMs that aborts in quantized_matmul (Biases must be provided for affine) because mxfp4/nvfp4/mxfp8 weights carry no zero-point biases. minicpm-v-4.6-mxfp4 was the visible casualty.

Two reconciliations fix it:

  1. Mode auto-detect in QuantizedEmbedding::from_weights and UnifiedLinear::from_weights: a .biases tensor means affine; its absence means a block-float scheme, distinguished by bits (8 → mxfp8) and group_size (16 → nvfp4, else mxfp4).
  2. group_size reconciliation for block-float modes in UnifiedLinear::from_weights_with_mode. For affine, bits are inferred against the trusted group_size (unchanged). For block-float modes the bits are fixed by the mode, so group_size is derived from the shapes instead: in_features = packed_in * (32/bits) = num_groups * group_size. A mixed export such as minicpm-v-4.6-mxfp4 stores some weights at group_size 32 while config.json reports 64; the old loader passed 64 straight through and mismatched in the matmul.

Verification (GB10, CUDA)

Model Before After
minicpm-v-4.6-mxfp4 (with image) abort (Biases must be provided for affine, then group_size mismatch) coherent description, 139.85 tok/s decode
qwen2.5-7b-4bit (affine sanity) 53 tok/s 53 tok/s (path unchanged)
gpt-oss-20b-mxfp4 (block-float sanity) 78 tok/s 78 tok/s (inferred params equal passed)

For affine models and correctly exported block-float models the inferred params equal the passed ones, so behavior is byte-identical.

…roup_size

Vision encoders (and any caller that does not thread an explicit
quantization mode) loaded weights through `from_weights`, which hardcoded
the affine mode. On block-float VLMs that aborted in quantized_matmul
("Biases must be provided for affine") because mxfp4/nvfp4/mxfp8 weights
carry no zero-point biases.

Two reconciliations make these load correctly:

- Auto-detect the mode in both QuantizedEmbedding::from_weights and
  UnifiedLinear::from_weights: the presence of a `.biases` tensor means
  affine; its absence means a block-float scheme, distinguished by bits
  (8 -> mxfp8) and group_size (16 -> nvfp4, else mxfp4).

- Reconcile group_size from the tensor shapes for block-float modes in
  UnifiedLinear::from_weights_with_mode. For affine, bits are inferred
  with the trusted group_size (unchanged). For block-float modes bits are
  fixed by the mode, so derive group_size instead:
  in_features = packed_in * (32/bits) = num_groups * group_size. A mixed
  export such as minicpm-v-4.6-mxfp4 stores some weights at group_size 32
  while config.json reports 64; the old loader passed 64 straight through
  and mismatched in the matmul.

minicpm-v-4.6-mxfp4 now produces a coherent image description and
benchmarks at 139.85 tok/s decode on GB10. Affine models and correctly
exported block-float models are byte-identical (inferred params equal the
passed ones): qwen2.5-7b-4bit 53 tok/s and gpt-oss-20b-mxfp4 78 tok/s
unchanged.
@inureyes inureyes added bug status:review Under review type:bug Bug fixes, error corrections, or issue resolutions and removed bug labels Jun 17, 2026
@inureyes
inureyes merged commit ec31534 into main Jun 17, 2026
5 checks passed
@inureyes
inureyes deleted the fix/vlm-mxfp4-quant-mode branch June 17, 2026 05:26
inureyes added a commit that referenced this pull request Jun 17, 2026
…rop qwen2.5-vl-3b (#335)

Follows the loader fix (#334). Updates the GB10 0.3.1 benchmark CSVs and docs:

- minicpm-v-4.6-mxfp4: now passes after #334. Text 347.59 prefill /
  142.14 decode tok/s, image input 625.47 / 138.70 (48 tok). Added to the
  image-input table and flipped to a pass in the text table.
- qwen2.5-vl-3b: removed. It was a broken duplicate of qwen2.5-vl-3b-4bit
  (the model dir was deleted); the working 4bit variant stays. Both CSVs
  go 148 -> 147 rows.
- Reclassified six by-design non-failures from fail (red x) to N.A.
  (white circle): docling-layout-heron and granite-speech (not standalone
  text generators) and the four MTP/DFlash drafters (gemma-4-12b/31b
  assistant, qwen3.5-27b/4b-dflash) that need a target model. The raw CSV
  FAIL:bench markers stay (they genuinely produce no text-gen output); the
  N.A. tag is a doc-layer classification.

GB10 0.3.1 summary is now 147 total: 136 pass, 0 code failures, 9 N.A.,
2 too-large (qwen3-next-480b, deepseek-v3, shown red x, OOM), 54 VLM image
rows. Legend and the by-cause section were rewritten to match; index
counts and the benchmark-CSV table updated in step.
@inureyes inureyes added this to the 0.3 milestone Jun 21, 2026
@inureyes inureyes self-assigned this Aug 31, 2026
@inureyes inureyes added status:done Completed and removed status:review Under review labels Sep 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

status:done Completed type:bug Bug fixes, error corrections, or issue resolutions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant