ROCm: validate native gfx1151 serving on Strix Halo

[dbourdea]

Summary

Adds and validates a native ROCm/HIP execution path for AMD Radeon 8060S gfx1151, while retaining NVIDIA CUDA as a separate execution path.

The branch includes the upstream ROCm work plus focused fixes from full-model validation on the GMKtec EVO-X2:

• Explicit HIP detection so gfx1151 cannot be treated as an NVIDIA compute capability.
• HIP-safe gating for CUDA-only optional backends and launch options.
• Native JIT and fast indexed-copy support for HIP tensors.
• Safe native Triton serial NVFP4 prefill on HIP while the grouped route remains under quality investigation.
• GGUF JIT fallback discovery for ROCm Thrust headers and HIP runtime libraries.
• Native Qwen GGUF support with packed Q4_K, Q5_K, Q6_K, and Q8_0 weights.

Validation

Validated on a GMKtec EVO-X2 with AMD Radeon 8060S gfx1151, PyTorch 2.13.0+rocm10.0.0, and HIP 7.15.26333:

• OpenAI-compatible Qwen text and streaming requests.
• Native Gemma 4 GGUF multimodal image requests.
• Deterministic text, JSON, multi-turn state, long-context, and visual-quality controls.
• Native host-extension and GGUF JIT compilation.
• Focused ROCm regression coverage and reproducible benchmark controls.

Detailed source, build, validation, quality, and performance boundaries are recorded in the repository documentation and retained raw artifacts.

Scope

This PR does not add llama-swap integration. CUDA graph capture remains disabled for the HIP MVP. Performance claims remain limited to the documented same-model, same-format controls and do not claim parity where the evidence does not prove it.

[Copilot]

🟡 Changes recommended

There are build-detection/compilation-guard issues in the new ROCm linking path that can cause ROCm builds to incorrectly include CUDA headers or link against the wrong runtime.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a native ROCm/HIP execution path targeting AMD gfx1151 (Strix Halo / Radeon 8060S) while keeping NVIDIA CUDA behavior as a separately gated path, including build/link fixes, runtime feature gating, and validation docs.

Changes:

• Add explicit HIP/ROCm runtime detection to prevent gfx11xx capability tuples from being treated as NVIDIA SM versions and to disable CUDA-only optional backends on HIP.
• Extend native/JIT paths (TVM JIT matchers, fast index copy, Triton kernels, GGUF JIT) to accept ROCm devices and avoid CUDA-only flags/launch kwargs/PTX.
• Add regression tests plus detailed LAN-223 validation and ROCm deployment documentation.

File summaries

File	Description
tests/utils/test_rocm_runtime.py	Adds regression tests for HIP gating and CUDA-only optional backend suppression.
tests/kernels/test_pinned_tensor.py	Adjusts UVA identity test behavior for HIP runtime semantics.
setup.py	Links native extensions against HIP runtime when ROCm is detected; skips nvcc toolchain checks on ROCm.
python/freetoken/utils/arch.py	Adds is_rocm_runtime() and gates CUDA capability queries on non-HIP builds.
python/freetoken/utils/init.py	Exports is_rocm_runtime from utils.
python/freetoken/moe/fused_nvfp4.py	Routes HIP prefill to safer serial Triton NVFP4 path.
python/freetoken/kernel/utils.py	Drops nvcc-only flags when compiling under HIP.
python/freetoken/kernel/triton/norm.py	Avoids passing CUDA-only launch_pdl kwarg to AMD Triton backend.
python/freetoken/kernel/triton/e4m3_compat.py	Forces e4m3 “native” detection off on HIP to avoid tuple false-positives.
python/freetoken/kernel/triton/attention.py	Adjusts HIP decode tiling to satisfy RDNA WMMA constraints.
python/freetoken/kernel/triton/activation.py	Avoids inlining PTX on HIP and avoids CUDA-only launch kwargs.
python/freetoken/kernel/gguf.py	Adds ROCm fallback discovery for Thrust headers and libamdhip64.so linker path.
python/freetoken/kernel/csrc/pinned_tensor.cpp	Switches pinned tensor extension to HIP/CUDA compat shim header.
python/freetoken/kernel/csrc/jit/store.cu	Allows ROCm tensors in TVM JIT store path device checks.
python/freetoken/kernel/csrc/jit/index.cu	Allows ROCm tensors in TVM JIT index path device checks.
python/freetoken/kernel/csrc/jit/fast_index_copy.cuh	Adds HIP-safe load/store fallbacks and broadens accepted device types.
python/freetoken/kernel/csrc/include/freetoken/utils.cuh	Adds HIP runtime includes/aliases and HIP LaunchKernel implementation.
python/freetoken/kernel/csrc/hip_compat.h	Introduces CUDA-runtime API name shims for HIP linking in host C++ extensions.
python/freetoken/kernel/csrc/gguf/dispatch.h	Fixes HIP shuffle mask width expectations by widening masks for HIP.
python/freetoken/kernel/csrc/cpu_moe/cpu_moe_ext.cpp	Switches CPU MoE extension to HIP/CUDA compat shim header.
python/freetoken/kernel/backend.py	Adds HIP detection and disables CUDA-only optional backend probes on ROCm.
python/freetoken/engine/engine.py	Skips optional Triton prefill warmup by default on ROCm unless explicitly enabled.
pyproject.toml	Loosens torch/triton constraints and adds ROCm classifier.
docs/lan223-rocm-validation-2026-08-28.md	Records LAN-223 reproducible ROCm validation evidence and known limitations.
docs/amd-rocm-gfx1151.md	Adds ROCm/GFX1151 port documentation and validation procedure.
.gitignore	Ignores ROCm hipify-generated artifacts (*.hip, *_hip.*).

Review details

• Files reviewed: 25/26 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[dbourdea]

Fixed in 54d6ab2. setup.py now passes FREETOKEN_USE_ROCM=1 to both host extensions whenever the active PyTorch build reports torch.version.hip. hip_compat.h selects HIP when that explicit macro is present, in addition to the HIP compiler macros. Verified on LAN-223: setup.py build_ext --inplace compiled and linked both _pinned_tensor and _cpu_moe against libamdhip64.

[dbourdea]

Fixed in 54d6ab2. IS_ROCM now derives solely from torch.version.hip rather than CUDA_HOME/ROCM_HOME precedence, so a CUDA toolkit installed beside an active HIP PyTorch build cannot select cudart or trigger nvcc checks. Added regression coverage and verified 12 focused ROCm runtime and pinned-extension tests on LAN-223, plus a successful native host-extension build.