Skip to content

fix(rocm): make TVM-FFI index and store JIT kernels portable to HIP - #133

Draft
zihaomu wants to merge 11 commits into
FlashML-org:mainfrom
zihaomu:feat/rocm-rdna4-jit
Draft

fix(rocm): make TVM-FFI index and store JIT kernels portable to HIP#133
zihaomu wants to merge 11 commits into
FlashML-org:mainfrom
zihaomu:feat/rocm-rdna4-jit

Conversation

@zihaomu

@zihaomu zihaomu commented Aug 24, 2026

Copy link
Copy Markdown

Draft follow-up to #132. This branch is based on the #132 head. Until #132 merges, GitHub Files changed also includes the foundation diff; the incremental review scope is the four files listed below.

Summary

  • disable NVIDIA programmatic-dependent-launch instructions and launch attributes on ROCm while preserving the CUDA path;
  • accept kDLROCM tensors in the generic index and store TVM-FFI kernels;
  • keep the CUDA PTX implementation unchanged and use the HIP-compatible common launch path;
  • use kWarpThreads as the single source for wave-size assertions;
  • cover split=1/2/4 row copies, masked vocabulary misses, both index widths, and both store index widths.

Incremental review scope

Relative to zihaomu:feat/rocm-rdna3-rdna4-foundation:

  • python/freetoken/kernel/csrc/include/freetoken/utils.cuh
  • python/freetoken/kernel/csrc/jit/index.cu
  • python/freetoken/kernel/csrc/jit/store.cu
  • tests/kernels/test_jit_index_store.py

Diff size: 88 additions, 14 deletions.

Dependency

Validation

  • GPU: AMD Radeon AI PRO R9700 (gfx1201)
  • PyTorch 2.11.0+rocm7.14.0, HIP 7.14.60850
  • cold JIT plus pinned-memory focused run: 14 passed
  • warm tests/kernels/test_jit_index_store.py: 6 passed
  • covered fp32 D=256 (split=2), D=512 (split=4), lower/upper masked reset, int64 indexing, and int32/int64 store indices.

Compatibility

CUDA branches remain behind the non-ROCm preprocessor path. Physical NVIDIA regression testing is not claimed; this PR remains Draft until #132 and CUDA CI establish the final base.

bouclem and others added 8 commits August 24, 2026 13:53
- Add hip_compat.h shim mapping CUDA runtime API to HIP equivalents
- Update pinned_tensor.cpp to compile under both nvcc and hipcc
- Add ROCm detection in arch.py (is_rocm, get_rocm_gfx_arch, is_gfx11xx_family)
- Guard NVIDIA arch checks to return None on ROCm
- Skip nvcc version check in _toolchain.py when on ROCm
- Add ROCm build path in setup.py (ROCM_HOME, amdhip64, --offload-arch)
- Add _hip_cflags() in kernel/utils.py for JIT compilation on ROCm
- Add is_rocm() and driver_hip_version() in backend.py
- Add rocm-smi fallback in __main__.py for clangd generation
- Add TODO(ROCm) for NCCL->RCCL, flashinfer/sgl_kernel ROCm builds,
  Triton autotune RDNA3 tuning, PDL equivalent, hiprtc JIT cache
- Add AMD ROCm classifier in pyproject.toml
@zihaomu
zihaomu force-pushed the feat/rocm-rdna4-jit branch from 89ad7ab to 07a352a Compare August 31, 2026 07:54
samuelishida pushed a commit to samuelishida/FreeToken that referenced this pull request Sep 5, 2026
What:
- Remove .agents/learnings and .plans/rocm-consolidation files from the branch.
- Remove internal increment and plan-path references from source comments and public installation docs.
- Keep implementation comments that explain correctness, ownership, profiler intent, source attribution, or ROCm safety behavior.
- Clarify public ROCm documentation: gfx1100 has recorded serving smoke on ROCm 7.2.1; the ROCm 7.14.x container is a reference environment, and other target cells remain compile-only until physical serving evidence exists.

Why:
- Keep merge surface focused on code, tests, reproducibility tooling, and user-facing documentation.
- Prevent private planning history, review workflow language, stale plan paths, and local process notes from entering the upstream repository.
- Avoid presenting compile success or a reference container as cross-target serving or performance proof.

Related upstream work informing this branch:
- PR FlashML-org#132: portable ROCm/HIP foundation.
- PR FlashML-org#133: TVM-FFI index/store portability.
- PR FlashML-org#135: RCCL tensor-parallel communication.
- PR FlashML-org#136: native GGUF ROCm build and Q4_0 kernels.
- PR FlashML-org#137: earlier AMD serving bring-up.
- PR FlashML-org#217: source-fork ROCm, Qwen3.5 GGUF, and performance experiments.
- PR FlashML-org#241: gfx1150 build, JIT, Triton, and attention hardening.
- PR FlashML-org#260: gfx1151 validation and fallback/build evidence.
- PR FlashML-org#316: HIP graph-capture-safe expert copies.
- PR FlashML-org#378: CPU/Hybrid MoE graph replay safety.
- Local branch milestones: 436263f, 926c1e8, e1d1856, 8a70c7e, and e5fd30f.

Evidence:
- 170 focused tests passed after cleanup.
- gfx1100 is the only target with end-to-end Qwen3.5 GGUF serving smoke recorded here.
- Remaining matrix targets are compile-only; no new throughput claim is published without a matching A/B manifest.
Hekapo pushed a commit to Hekapo/FreeTokenROCm that referenced this pull request Sep 6, 2026
Adapt the portable JIT and wave32 GGUF safeguards from upstream PRs FlashML-org#132, FlashML-org#133, and FlashML-org#136 while preserving the Windows ROCm build path and row-aware Triton fallback.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants