Skip to content

feat: packed int4 convrot model loading and dispatch - #2010

Draft
noctrex wants to merge 3 commits into
leejet:masterfrom
noctrex:convrot-w4-family
Draft

noctrex wants to merge 3 commits into
leejet:masterfrom
noctrex:convrot-w4-family

Conversation

@noctrex

@noctrex noctrex commented Sep 20, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds direct loading and execution of ComfyUI packed int4 convrot checkpoints (convrot_w4a4, asym_w4a8_int8). The packed nibble weights stream to the backend unconverted: no load-time reblock, no sidecar cache, roughly half a byte per weight parameter. Formats are documented in docs/int4_convrot.md (added here).

Loader/graph side:

  • packed-w4 tensor recognition in safetensors IO; packed payloads and companion tensors (row scales, codebook, s_channel, s_rel) pass through raw
  • block-builder support mirroring the int8 convrot chain: the activation rotation/quantize is shared with int8 tensorwise and cached per (input, group size)
  • activation quantize gated on unit scale for int4 inputs
  • docs: docs/int4_convrot.md

GGML dependency

The kernels live in the GGML submodule. The ggml-side changes are submitted as leejet/ggml#10 ("ggml : add packed int4 convrot mul_mat for CPU, CUDA, HIP and Vulkan"), which includes the HIP quantize-gate fix this PR's pin depends on.

This PR therefore bumps the ggml pin to a branch on my fork. Per the contribution guidelines, ggml updates are validated by the maintainer - keeping this PR in draft until that side is settled; the pin can be retargeted the moment the ggml content lands in leejet/ggml.

Backend notes (disclosure)

  • int4 (both kinds) runs on CPU, CUDA, HIP and Vulkan; convrot group sizes 64 and 256 verified.
  • On HIP, plain int8_tensorwise models and int8 convrot text encoders currently fall back to CPU; that enablement is the separate INT8 HIP/BLAS ggml PR's scope and does not affect int4 models.

Verification

All runs on RX 7900 XTX (gfx1100), Windows, ROCm 10 / Vulkan, 1024x1024, 8 steps, seed 42:

  • FLUX.2-klein-4b int4: Vulkan 11.9 s sampling (1.24 s/it, faster than the int8 baseline's 12.3 s), ROCm 9.56 s post-fix
  • FLUX.2-klein-4b w4a8: Vulkan 13.9 s, ROCm 12.40 s
  • FLUX.2-klein-4b int8 baseline: Vulkan 12.3 s (visual reference; images equivalent)
  • z_image-turbo int4 (gs=256 regression check): ROCm 14.38 s
  • Loader fixture against the merged loader: W4A4 bitwise vs CPU oracle, W4A8 within the int8-requant gate, raw packed passthrough intact
  • zero unsupported-node/CPU-fallback warnings for the int4 paths; images on-prompt and visually equivalent to int8

Linked context

Checklist

LLM Disclosure: Yes, used GLM-5.3-Flash

- loader: packed-w4 convrot tensor recognition; packed payloads stream
  to the backend unconverted (no load-time reblock, no sidecar cache)
- safetensors IO, tensor storage, block builder support for the
  packed-w4 payloads
- activation quantize gated on unit scale for w4 inputs
- ggml: pointer to the squashed w4 convrot kernel commit (6a7f9c31,
  includes group-size-64 quantize support as h64-PR overlap)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant