Skip to content

perf(moe): 6-bit/mixed-bit fused decode-MoE; wire dots.llm1 (#268 step 2c) - #278

Merged
inureyes merged 1 commit into
mainfrom
perf/issue-268-fused-moe-6bit
Jun 14, 2026
Merged

inureyes merged 1 commit into
mainfrom
perf/issue-268-fused-moe-6bit

Conversation

@inureyes

Copy link
Copy Markdown
Member

Summary

Step 2c of #268: extend the two-kernel fused decode-MoE (#276) to 6-bit and mixed bit widths, and route dots.llm1's MoE decode through it (it fell back before, being 4/6-bit mixed).

What changed

The two-kernel split makes mixed widths natural — kernel A (gate/up + swiglu) and kernel B (down) each get their own bits template arg, so dots.llm1 (gate/up 4-bit, down 6-bit) needs only a 6-bit path in kernel B.

6-bit is not a power of 2: MLX packs 4 weights into 3 bytes (pack_factor 4, bytes_per_pack 3). The lane reads the down row as bytes and reconstructs the four 6-bit values with the bit layout taken from MLX quantized.h qdot:

v0 = b0 & 0x3f
v1 = (b0 >> 6) | ((b1 & 0x0f) << 2)
v2 = (b1 >> 4) | ((b2 & 0x03) << 4)
v3 = b2 >> 2
  • FFI fused_moe_expert_kernel now takes gu_bits (gate/up, 4/8) + d_bits (down, 4/6/8) instead of one bits; group_size stays shared.
  • Rust guards relax: gate/up must match each other and be 4/8-bit; down may be 4/6/8-bit; 6-bit down requires Dff % 16 == 0 (whole uint32 columns).
  • dots1.rs MoE forward dispatches to the fused kernel on single-token decode behind MLXCEL_FUSED_MOE, mirroring qwen3_moe.

Validation (M1 Ultra, greedy temp-0)

  • dots.llm1 (the 6-bit path): output byte-identical to the SwitchGLU fallback. Decode 13.1 → 13.7 tok/s (+4.7%).
  • qwen3-30b-a3b: byte-identical (no regression from the signature change).
  • cargo fmt --check + cargo clippy clean.

Still off by default (MLXCEL_FUSED_MOE), single-token decode only.

Scope

nemotron-h's MoE runs through the separate C++ full-forward path (fused_moe_forward), not the Rust SwitchGLU, so it is not wired here — a possible follow-up.

Refs #268.

…p 2c)

Extend the two-kernel fused decode-MoE (#268 step 2b) to 6-bit and mixed bit
widths, and route dots.llm1's MoE decode through it.

The two-kernel split makes mixed widths natural: kernel A (gate/up + swiglu) and
kernel B (down) each get their own `bits` template arg, so dots.llm1's experts
(gate/up 4-bit, down 6-bit) need only a 6-bit path in kernel B. 6-bit is not a
power of 2: MLX packs 4 weights into 3 bytes (pack_factor 4, bytes_per_pack 3),
so the lane reads the down row as bytes and reconstructs the four 6-bit values
with the bit layout from MLX quantized.h qdot:
  v0=b0&0x3f, v1=(b0>>6)|((b1&0x0f)<<2), v2=(b1>>4)|((b2&0x03)<<4), v3=b2>>2.

FFI `fused_moe_expert_kernel` takes `gu_bits` (gate/up, 4/8) and `d_bits`
(down, 4/6/8) instead of one `bits`; group_size stays shared. The Rust guards
relax accordingly (gate/up must match each other and be 4/8-bit; down may be
4/6/8-bit; down 6-bit needs Dff % 16 == 0 for whole uint32 columns). dots1's
MoE forward now dispatches to the fused kernel on single-token decode behind
MLXCEL_FUSED_MOE, mirroring qwen3_moe.

Validated greedy temp-0 byte-identical with the fallback on dots.llm1 (the
6-bit path) and qwen3-30b-a3b (no regression). dots.llm1 decode 13.1 -> 13.7
tok/s (+4.7%) on M1 Ultra; still off by default. nemotron-h's MoE goes through
the separate C++ full-forward path and is not wired here.
@inureyes inureyes added type:performance Performance improvements area:models Model architectures, weights, loading, metadata area:inference Generation, sampling, decoding (incl. speculative, DRY) platform:macos macOS (Apple Silicon) specific priority:medium Medium priority labels Jun 14, 2026
@inureyes
inureyes merged commit e48ea22 into main Jun 14, 2026
5 checks passed
@inureyes
inureyes deleted the perf/issue-268-fused-moe-6bit branch June 14, 2026 04:21
inureyes added a commit that referenced this pull request Jun 14, 2026
…#283)

* docs(moe): document MLXCEL_FUSED_MOE flags, per-model gains, and M5 follow-up (#268)

Add a usage/flags section to the fused decode-MoE design doc: the
MLXCEL_FUSED_MOE / MLXCEL_FUSED_MOE_SGY / MLXCEL_FUSED_MOE_RELU2 env vars, the
measured per-model decode gains on M1 Ultra (gemma4 +13%, qwen3.5 +8.7%, dots
+4.7%, qwen3-30b +3.5%, nemotron ~0%), the f16-jitter-class parity caveat, and
the list of covered models. Mark roadmap steps 3-4 done (#278/#279/#280/#281)
and add step 5: validate on M5, then decide on flipping the flag default-on.

* docs(moe): reference the M5 validation issue (#282)
@inureyes inureyes added this to the 0.3 milestone Jun 21, 2026
@inureyes inureyes self-assigned this Aug 31, 2026
@inureyes inureyes added the status:done Completed label Sep 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:inference Generation, sampling, decoding (incl. speculative, DRY) area:models Model architectures, weights, loading, metadata platform:macos macOS (Apple Silicon) specific priority:medium Medium priority status:done Completed type:performance Performance improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant