Conversation
- loader: packed-w4 convrot tensor recognition; packed payloads stream to the backend unconverted (no load-time reblock, no sidecar cache) - safetensors IO, tensor storage, block builder support for the packed-w4 payloads - activation quantize gated on unit scale for w4 inputs - ggml: pointer to the squashed w4 convrot kernel commit (6a7f9c31, includes group-size-64 quantize support as h64-PR overlap)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds direct loading and execution of ComfyUI packed int4 convrot checkpoints (
convrot_w4a4,asym_w4a8_int8). The packed nibble weights stream to the backend unconverted: no load-time reblock, no sidecar cache, roughly half a byte per weight parameter. Formats are documented indocs/int4_convrot.md(added here).Loader/graph side:
docs/int4_convrot.mdGGML dependency
The kernels live in the GGML submodule. The ggml-side changes are submitted as leejet/ggml#10 ("ggml : add packed int4 convrot mul_mat for CPU, CUDA, HIP and Vulkan"), which includes the HIP quantize-gate fix this PR's pin depends on.
This PR therefore bumps the
ggmlpin to a branch on my fork. Per the contribution guidelines, ggml updates are validated by the maintainer - keeping this PR in draft until that side is settled; the pin can be retargeted the moment the ggml content lands inleejet/ggml.Backend notes (disclosure)
int8_tensorwisemodels and int8 convrot text encoders currently fall back to CPU; that enablement is the separate INT8 HIP/BLAS ggml PR's scope and does not affect int4 models.Verification
All runs on RX 7900 XTX (gfx1100), Windows, ROCm 10 / Vulkan, 1024x1024, 8 steps, seed 42:
Linked context
Checklist
LLM Disclosure: Yes, used GLM-5.3-Flash