feat(rocm): serve on AMD GPUs through the HIP toolchain - #137
Open
paralin wants to merge 1 commit into
Open
Conversation
paralin
force-pushed
the
amd
branch
3 times, most recently
from
August 24, 2026 08:52
22b6aee to
7e1d742
Compare
FreeToken built and served only against CUDA: setup.py linked its host-side extensions against cudart, the kernel JIT compiled with nvcc-only flags, and several triton call sites passed NVIDIA-only launch options. On an AMD GPU the engine could not even finish booting. Build _pinned_tensor and _cpu_moe against HIP through a shim header that maps every CUDA runtime symbol they use onto its HIP equivalent. Teach the kernel JIT the same trick: drop --expt-relaxed-constexpr (hipcc rejects it; relaxed constexpr is already its default) and force-include a shim, packaged with the kernel csrc tree, that maps the CUDA launch-config surface onto hipLaunchKernelEx. PDL launch attributes have no equivalent on this runtime; every kernel served on AMD builds with use_pdl=false, so the shim drops attributes instead of setting them. Three smaller fixes complete the port: - norm.py and activation.py omit the triton launch_pdl keyword under HIP; the AMD launcher rejects it outright. NVIDIA keeps the upstream launch_pdl=pdl call for both true and false. - attention.py floors block_h at 16 under HIP because RDNA3 WMMA cannot select an instruction for tl.dot below M=16. Padded head lanes were already masked. - fast_index_copy.cuh guards its PTX streaming-load/store inline asm: CUDA keeps it verbatim; HIP has no equivalent asm, so those builds fall back to plain device loads and stores. The cache-policy hints are dropped on HIP only; correctness is unchanged. setup.py detects the backend from torch: ROCm builds of torch take the HIP branch, CUDA builds keep the original cudart link and nvcc toolchain check unchanged, and anything else fails with a clear error. Every other edit is gated on HIP detection at run time. ROCm installs should resolve torch from the ROCm wheel index first; the pinned PyPI triton conflicts with the ROCm-bundled one.
7 tasks
skywalk1411
added a commit
to skywalk1411/FreeToken
that referenced
this pull request
Aug 27, 2026
Both were made mid-investigation, before the real cause of a since-fixed crash (the raw-PTX bug in activation.py, and separately the e4m3_native() tuple- comparison bug) was actually found. Re-tested each in isolation -- eager, batched, and inside real CUDA graph capture+replay -- now that those are fixed, and both work fine at the original, CUDA-tuned settings: - decode_paged_attention: the block_h>=16 floor (kept -- RDNA WMMA genuinely has no instruction below M=16, confirmed independently and matches upstream FlashML-org#137) was sufficient on its own. The USE_TL_DOT broadcast-sum fallback this PR had added on top was solving a problem that was actually in a different kernel; removed, restoring real matrix-core-accelerated decode attention. - _select_extend_tile: the 128x64 -> 64x32 shrink on HIP was diagnosed as a VGPR-exhaustion issue via a py-spy trace mid-investigation, before the session had isolated the actual crash to activation.py. Re-verified end-to-end against Qwen3.6-35B-A3B-FP8's GDN/split-extend path (the kernel this shrink targeted) at the original tile size: no crash, correct output. Reverted to the CUDA-tuned tile. Both re-verified against real chat completions (Qwen3-8B for the decode path, Qwen3.6-35B-A3B-FP8 for the extend/split path) after reverting, not just the isolated kernel tests.
dbourdea
pushed a commit
to dbourdea/FreeToken
that referenced
this pull request
Aug 29, 2026
Both were made mid-investigation, before the real cause of a since-fixed crash (the raw-PTX bug in activation.py, and separately the e4m3_native() tuple- comparison bug) was actually found. Re-tested each in isolation -- eager, batched, and inside real CUDA graph capture+replay -- now that those are fixed, and both work fine at the original, CUDA-tuned settings: - decode_paged_attention: the block_h>=16 floor (kept -- RDNA WMMA genuinely has no instruction below M=16, confirmed independently and matches upstream FlashML-org#137) was sufficient on its own. The USE_TL_DOT broadcast-sum fallback this PR had added on top was solving a problem that was actually in a different kernel; removed, restoring real matrix-core-accelerated decode attention. - _select_extend_tile: the 128x64 -> 64x32 shrink on HIP was diagnosed as a VGPR-exhaustion issue via a py-spy trace mid-investigation, before the session had isolated the actual crash to activation.py. Re-verified end-to-end against Qwen3.6-35B-A3B-FP8's GDN/split-extend path (the kernel this shrink targeted) at the original tile size: no crash, correct output. Reverted to the CUDA-tuned tile. Both re-verified against real chat completions (Qwen3-8B for the decode path, Qwen3.6-35B-A3B-FP8 for the extend/split path) after reverting, not just the isolated kernel tests.
samuelishida
pushed a commit
to samuelishida/FreeToken
that referenced
this pull request
Sep 5, 2026
What: - Remove .agents/learnings and .plans/rocm-consolidation files from the branch. - Remove internal increment and plan-path references from source comments and public installation docs. - Keep implementation comments that explain correctness, ownership, profiler intent, source attribution, or ROCm safety behavior. - Clarify public ROCm documentation: gfx1100 has recorded serving smoke on ROCm 7.2.1; the ROCm 7.14.x container is a reference environment, and other target cells remain compile-only until physical serving evidence exists. Why: - Keep merge surface focused on code, tests, reproducibility tooling, and user-facing documentation. - Prevent private planning history, review workflow language, stale plan paths, and local process notes from entering the upstream repository. - Avoid presenting compile success or a reference container as cross-target serving or performance proof. Related upstream work informing this branch: - PR FlashML-org#132: portable ROCm/HIP foundation. - PR FlashML-org#133: TVM-FFI index/store portability. - PR FlashML-org#135: RCCL tensor-parallel communication. - PR FlashML-org#136: native GGUF ROCm build and Q4_0 kernels. - PR FlashML-org#137: earlier AMD serving bring-up. - PR FlashML-org#217: source-fork ROCm, Qwen3.5 GGUF, and performance experiments. - PR FlashML-org#241: gfx1150 build, JIT, Triton, and attention hardening. - PR FlashML-org#260: gfx1151 validation and fallback/build evidence. - PR FlashML-org#316: HIP graph-capture-safe expert copies. - PR FlashML-org#378: CPU/Hybrid MoE graph replay safety. - Local branch milestones: 436263f, 926c1e8, e1d1856, 8a70c7e, and e5fd30f. Evidence: - 170 focused tests passed after cleanup. - gfx1100 is the only target with end-to-end Qwen3.5 GGUF serving smoke recorded here. - Remaining matrix targets are compile-only; no new throughput claim is published without a matching A/B manifest.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
FreeToken built and served only against CUDA: setup.py linked its host-side extensions against cudart, the kernel JIT compiled with nvcc-only flags, and several triton call sites passed NVIDIA-only launch options. On an AMD GPU the engine could not even finish booting.
Build _pinned_tensor and _cpu_moe against HIP through a shim header that maps every CUDA runtime symbol they use onto its HIP equivalent. Teach the JIT the same trick: drop --expt-relaxed-constexpr (hipcc rejects it; relaxed constexpr is already its default) and force- include a shim that maps the CUDA launch-config surface onto hipLaunchKernelEx. PDL launch attributes have no equivalent on this runtime; every kernel served on AMD builds with use_pdl=false, so the shim drops attributes instead of setting them.
Three smaller fixes complete the port:
Drop the explicit triton==3.6.0 pin so the ROCm-bundled AMD-backend triton survives installation; torch's own wheels already constrain their matching triton build. Every edit is gated on HIP detection at build or run time, so NVIDIA paths are unchanged.