Skip to content

fix(rocm): make TVM-FFI index and store JIT kernels portable to HIP - #133

Draft
zihaomu wants to merge 11 commits into
FlashML-org:mainfrom
zihaomu:feat/rocm-rdna4-jit
Draft

zihaomu wants to merge 11 commits into
FlashML-org:mainfrom
zihaomu:feat/rocm-rdna4-jit

Conversation

@zihaomu

@zihaomu zihaomu commented Aug 24, 2026 •

Copy link
Copy Markdown
Contributor

Draft follow-up to #132. This branch is based on the #132 head. Until #132 merges, GitHub Files changed also includes the foundation diff; the incremental review scope is the four files listed below.

Summary

  • disable NVIDIA programmatic-dependent-launch instructions and launch attributes on ROCm while preserving the CUDA path;
  • accept kDLROCM tensors in the generic index and store TVM-FFI kernels;
  • keep the CUDA PTX implementation unchanged and use the HIP-compatible common launch path;
  • use kWarpThreads as the single source for wave-size assertions;
  • cover split=1/2/4 row copies, masked vocabulary misses, both index widths, and both store index widths.

Incremental review scope

Relative to zihaomu:feat/rocm-rdna3-rdna4-foundation:

  • python/freetoken/kernel/csrc/include/freetoken/utils.cuh
  • python/freetoken/kernel/csrc/jit/index.cu
  • python/freetoken/kernel/csrc/jit/store.cu
  • tests/kernels/test_jit_index_store.py

Diff size: 88 additions, 14 deletions.

Dependency

Validation

  • GPU: AMD Radeon AI PRO R9700 (gfx1201)
  • PyTorch 2.11.0+rocm7.14.0, HIP 7.14.60850
  • cold JIT plus pinned-memory focused run: 14 passed
  • warm tests/kernels/test_jit_index_store.py: 6 passed
  • covered fp32 D=256 (split=2), D=512 (split=4), lower/upper masked reset, int64 indexing, and int32/int64 store indices.

Compatibility

CUDA branches remain behind the non-ROCm preprocessor path. Physical NVIDIA regression testing is not claimed; this PR remains Draft until #132 and CUDA CI establish the final base.

nekomario28 added a commit to nekomario28/FreeToken that referenced this pull request Aug 25, 2026
@zihaomu
zihaomu force-pushed the feat/rocm-rdna4-jit branch from 89ad7ab to 07a352a Compare August 31, 2026 07:54
samuelishida pushed a commit to samuelishida/FreeToken that referenced this pull request Sep 5, 2026
What:
- Remove .agents/learnings and .plans/rocm-consolidation files from the branch.
- Remove internal increment and plan-path references from source comments and public installation docs.
- Keep implementation comments that explain correctness, ownership, profiler intent, source attribution, or ROCm safety behavior.
- Clarify public ROCm documentation: gfx1100 has recorded serving smoke on ROCm 7.2.1; the ROCm 7.14.x container is a reference environment, and other target cells remain compile-only until physical serving evidence exists.

Why:
- Keep merge surface focused on code, tests, reproducibility tooling, and user-facing documentation.
- Prevent private planning history, review workflow language, stale plan paths, and local process notes from entering the upstream repository.
- Avoid presenting compile success or a reference container as cross-target serving or performance proof.

Related upstream work informing this branch:
- PR FlashML-org#132: portable ROCm/HIP foundation.
- PR FlashML-org#133: TVM-FFI index/store portability.
- PR FlashML-org#135: RCCL tensor-parallel communication.
- PR FlashML-org#136: native GGUF ROCm build and Q4_0 kernels.
- PR FlashML-org#137: earlier AMD serving bring-up.
- PR FlashML-org#217: source-fork ROCm, Qwen3.5 GGUF, and performance experiments.
- PR FlashML-org#241: gfx1150 build, JIT, Triton, and attention hardening.
- PR FlashML-org#260: gfx1151 validation and fallback/build evidence.
- PR FlashML-org#316: HIP graph-capture-safe expert copies.
- PR FlashML-org#378: CPU/Hybrid MoE graph replay safety.
- Local branch milestones: 436263f, 926c1e8, e1d1856, 8a70c7e, and e5fd30f.

Evidence:
- 170 focused tests passed after cleanup.
- gfx1100 is the only target with end-to-end Qwen3.5 GGUF serving smoke recorded here.
- Remaining matrix targets are compile-only; no new throughput claim is published without a matching A/B manifest.
Hekapo pushed a commit to Hekapo/FreeTokenROCm that referenced this pull request Sep 6, 2026
Adapt the portable JIT and wave32 GGUF safeguards from upstream PRs FlashML-org#132, FlashML-org#133, and FlashML-org#136 while preserving the Windows ROCm build path and row-aware Triton fallback.
@zihaomu
zihaomu force-pushed the feat/rocm-rdna4-jit branch from 07a352a to ccea1ff Compare September 16, 2026 07:42
@zihaomu

zihaomu commented Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

Dependency refresh (2026-09-16): rebased the two PR-specific JIT commits onto the conflict-resolved #132 head c2af4c6. git range-diff reports both commits as patch-equivalent, so no JIT behavior was changed during the refresh. Static AST and git diff --check validation pass. The new head is ccea1ff; GitHub reports this PR as mergeable/clean. GPU-required JIT execution tests were not rerun during this dependency-only rebase.

LZ-QWQ added a commit to LZ-QWQ/FreeToken that referenced this pull request Sep 19, 2026
bouclem and others added 10 commits September 21, 2026 10:54
- Add hip_compat.h shim mapping CUDA runtime API to HIP equivalents
- Update pinned_tensor.cpp to compile under both nvcc and hipcc
- Add ROCm detection in arch.py (is_rocm, get_rocm_gfx_arch, is_gfx11xx_family)
- Guard NVIDIA arch checks to return None on ROCm
- Skip nvcc version check in _toolchain.py when on ROCm
- Add ROCm build path in setup.py (ROCM_HOME, amdhip64, --offload-arch)
- Add _hip_cflags() in kernel/utils.py for JIT compilation on ROCm
- Add is_rocm() and driver_hip_version() in backend.py
- Add rocm-smi fallback in __main__.py for clangd generation
- Add TODO(ROCm) for NCCL->RCCL, flashinfer/sgl_kernel ROCm builds,
  Triton autotune RDNA3 tuning, PDL equivalent, hiprtc JIT cache
- Add AMD ROCm classifier in pyproject.toml
@zihaomu
zihaomu force-pushed the feat/rocm-rdna4-jit branch from ccea1ff to 9ec8271 Compare September 21, 2026 03:01
LZ-QWQ pushed a commit to LZ-QWQ/FreeToken that referenced this pull request Sep 22, 2026
LZ-QWQ pushed a commit to LZ-QWQ/FreeToken that referenced this pull request Sep 22, 2026
LZ-QWQ pushed a commit to LZ-QWQ/FreeToken that referenced this pull request Sep 22, 2026
@zihaomu zihaomu mentioned this pull request Sep 23, 2026
29 tasks

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants