ROCm: validate native gfx1151 serving on Strix Halo

[dbourdea]

Summary

Adds and validates a native ROCm/HIP execution path for AMD Radeon 8060S gfx1151, while retaining NVIDIA CUDA as a separate execution path.

The branch includes the upstream ROCm work plus focused fixes from full-model validation on the GMKtec EVO-X2:

• Explicit HIP detection so gfx1151 cannot be treated as an NVIDIA compute capability.
• HIP-safe gating for CUDA-only optional backends and launch options.
• Native JIT and fast indexed-copy support for HIP tensors.
• Safe native Triton serial NVFP4 prefill on HIP while the grouped route remains under quality investigation.
• GGUF JIT fallback discovery for ROCm Thrust headers and HIP runtime libraries.
• Native Qwen GGUF support with packed Q4_K, Q5_K, Q6_K, and Q8_0 weights.

Validation

Validated on a GMKtec EVO-X2 with AMD Radeon 8060S gfx1151, PyTorch 2.13.0+rocm10.0.0, and HIP 7.15.26333:

• OpenAI-compatible Qwen text and streaming requests.
• Native Gemma 4 GGUF multimodal image requests.
• Deterministic text, JSON, multi-turn state, long-context, and visual-quality controls.
• Native host-extension and GGUF JIT compilation.
• Focused ROCm regression coverage and reproducible benchmark controls.

Detailed source, build, validation, quality, and performance boundaries are recorded in the repository documentation and retained raw artifacts.

Scope

This PR does not add llama-swap integration. CUDA graph capture remains disabled for the HIP MVP. Performance claims remain limited to the documented same-model, same-format controls and do not claim parity where the evidence does not prove it.

[Copilot]

🟡 Changes recommended

There are build-detection/compilation-guard issues in the new ROCm linking path that can cause ROCm builds to incorrectly include CUDA headers or link against the wrong runtime.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a native ROCm/HIP execution path targeting AMD gfx1151 (Strix Halo / Radeon 8060S) while keeping NVIDIA CUDA behavior as a separately gated path, including build/link fixes, runtime feature gating, and validation docs.

Changes:

• Add explicit HIP/ROCm runtime detection to prevent gfx11xx capability tuples from being treated as NVIDIA SM versions and to disable CUDA-only optional backends on HIP.
• Extend native/JIT paths (TVM JIT matchers, fast index copy, Triton kernels, GGUF JIT) to accept ROCm devices and avoid CUDA-only flags/launch kwargs/PTX.
• Add regression tests plus detailed LAN-223 validation and ROCm deployment documentation.

File summaries

File	Description
tests/utils/test_rocm_runtime.py	Adds regression tests for HIP gating and CUDA-only optional backend suppression.
tests/kernels/test_pinned_tensor.py	Adjusts UVA identity test behavior for HIP runtime semantics.
setup.py	Links native extensions against HIP runtime when ROCm is detected; skips nvcc toolchain checks on ROCm.
python/freetoken/utils/arch.py	Adds is_rocm_runtime() and gates CUDA capability queries on non-HIP builds.
python/freetoken/utils/init.py	Exports is_rocm_runtime from utils.
python/freetoken/moe/fused_nvfp4.py	Routes HIP prefill to safer serial Triton NVFP4 path.
python/freetoken/kernel/utils.py	Drops nvcc-only flags when compiling under HIP.
python/freetoken/kernel/triton/norm.py	Avoids passing CUDA-only launch_pdl kwarg to AMD Triton backend.
python/freetoken/kernel/triton/e4m3_compat.py	Forces e4m3 “native” detection off on HIP to avoid tuple false-positives.
python/freetoken/kernel/triton/attention.py	Adjusts HIP decode tiling to satisfy RDNA WMMA constraints.
python/freetoken/kernel/triton/activation.py	Avoids inlining PTX on HIP and avoids CUDA-only launch kwargs.
python/freetoken/kernel/gguf.py	Adds ROCm fallback discovery for Thrust headers and libamdhip64.so linker path.
python/freetoken/kernel/csrc/pinned_tensor.cpp	Switches pinned tensor extension to HIP/CUDA compat shim header.
python/freetoken/kernel/csrc/jit/store.cu	Allows ROCm tensors in TVM JIT store path device checks.
python/freetoken/kernel/csrc/jit/index.cu	Allows ROCm tensors in TVM JIT index path device checks.
python/freetoken/kernel/csrc/jit/fast_index_copy.cuh	Adds HIP-safe load/store fallbacks and broadens accepted device types.
python/freetoken/kernel/csrc/include/freetoken/utils.cuh	Adds HIP runtime includes/aliases and HIP LaunchKernel implementation.
python/freetoken/kernel/csrc/hip_compat.h	Introduces CUDA-runtime API name shims for HIP linking in host C++ extensions.
python/freetoken/kernel/csrc/gguf/dispatch.h	Fixes HIP shuffle mask width expectations by widening masks for HIP.
python/freetoken/kernel/csrc/cpu_moe/cpu_moe_ext.cpp	Switches CPU MoE extension to HIP/CUDA compat shim header.
python/freetoken/kernel/backend.py	Adds HIP detection and disables CUDA-only optional backend probes on ROCm.
python/freetoken/engine/engine.py	Skips optional Triton prefill warmup by default on ROCm unless explicitly enabled.
pyproject.toml	Loosens torch/triton constraints and adds ROCm classifier.
docs/lan223-rocm-validation-2026-08-28.md	Records LAN-223 reproducible ROCm validation evidence and known limitations.
docs/amd-rocm-gfx1151.md	Adds ROCm/GFX1151 port documentation and validation procedure.
.gitignore	Ignores ROCm hipify-generated artifacts (*.hip, *_hip.*).

Review details

• Files reviewed: 25/26 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[dbourdea]

Fixed in 54d6ab2. setup.py now passes FREETOKEN_USE_ROCM=1 to both host extensions whenever the active PyTorch build reports torch.version.hip. hip_compat.h selects HIP when that explicit macro is present, in addition to the HIP compiler macros. Verified on LAN-223: setup.py build_ext --inplace compiled and linked both _pinned_tensor and _cpu_moe against libamdhip64.

[dbourdea]

Fixed in 54d6ab2. IS_ROCM now derives solely from torch.version.hip rather than CUDA_HOME/ROCM_HOME precedence, so a CUDA toolkit installed beside an active HIP PyTorch build cannot select cudart or trigger nvcc checks. Added regression coverage and verified 12 focused ROCm runtime and pinned-extension tests on LAN-223, plus a successful native host-extension build.

[dbourdea]

Follow-up GMKtec EVO-X2 validation evidence for the current PR head:

• Native ROCm/HIP Gemma 4 GGUF vision is verified through OpenAI-compatible image_url data URLs with the sibling mmproj projector.
• Deterministic visual controls passed for solid red, solid green, and a red-left/blue-right spatial fixture.
• A quality-gated long visible response passed in FreeToken: 51 words, 63 completion tokens, 1,093.83 ms TTFT, and 53.87 visible-output tokens per second.
• The matched ROCm 10 llama.cpp Gemma path recognized the image in reasoning_content but did not emit visible content even at a 1,024-token cap. Its 55.91 generated tokens per second is reasoning-channel throughput and is deliberately not presented as a visible-output comparison.
• Every isolated candidate run restored the protected Qwen service. The final recovery-contract check reached /health status: ok after the cold load.
• Focused GMKtec EVO-X2 unit coverage passed: 140 tests for Gemma image preprocessing, mmproj mapping, OpenAI image input, shaped message transport, and streaming-model behavior.

Detailed reproducible artifacts and boundaries are recorded in docs/gmktec-evo-x2-gemma4-q4-vision-control-20260830.md.

[dbourdea]

Validation update for 42e5be2:

• Repaired the Gemma 4 multimodal path by preserving image tensors through the tokenizer-to-scheduler wire and by emitting channel-planar RGB patches for the projector convolution layout.
• FreeToken passed 21 of 21 repeated deterministic Gemma visual controls, covering red, green, blue, yellow, horizontal placement, and vertical placement. The matched ROCm llama.cpp control passed the same seven-fixture set.
• Qwen OpenAI-compatible API controls passed: deterministic text and JSON checks, 10 of 10 multi-turn state-retention sessions, 9 of 9 long-context marker retrieval samples through 7,736 reported prompt tokens, and a clean-memory 30 of 30 multi-turn endurance run.
• Focused regression tests passed: 21 of 21.
• Same-host fixed-workload throughput evidence is documented: FreeToken NVFP4 measured 28.15 median decode TPS and llama.cpp ROCm Q4_K_M measured 48.87. This is not quantization-equivalent, so it is reported as an observed configuration result rather than a general runner ranking.

The commit adds GMKtec EVO-X2 validation documentation, reproducible local runner controls, and regression coverage. It does not include private model files or host artifacts.

[dbourdea]

Validation follow-up: full-context MoE cache telemetry is now recorded in commit 5c7f0fd.

A temporary loopback-only Qwen instance enabled --moe-collect-stats with the validated native ROCm/HIP configuration, 0.35 memory ratio, and 8,192-token KV reservation. It resolved 8,903 MoE cache slots and 8,224 KV pages. The fixed warmup plus three-sample scheduler workload completed 3/3 at 28.035 mean visible decode TPS with 0.0066 TPS standard deviation. Across 40,800 decode-layer calls, the device counters reported eight active experts per layer, 0.586 missing experts per layer, and a 7.33% cache-miss rate.

This supports the prior measured rejection of a larger static cache: reducing misses by raising cache capacity did not improve sustained end-to-end TPS. The normal no-counter Qwen service was immediately restored and passed the deterministic AIME quality gate with the required SHA-1 0acef4eab6f4 at 28.60 visible decode TPS, 399.08 ms TTFT, and 38.49 ms p99 stream-event gap.

The report documents the exact methodology, limitation, and GMKtec EVO-X2 artifact root. This is evidence for the native AMD port, not a claim of quantization-equivalent parity with the separate Q4_K_M llama.cpp control.

[dbourdea]

Exact-format ROCm comparison update in commit d8a2dd6.

The branch now serves the exact same Qwen3.6-35B-A3B-UD-Q4_K_M.gguf used by the llama.cpp control through native FreeToken ROCm/HIP. The Qwen hybrid GGUF path keeps Q8_0, Q6_K, Q4_K, and Q5_K weights packed, uses the native Gated-DeltaNet path, and was launched with a source-revision-specific gfx1151 cache. A strict no-JIT verifier successfully resolved all 82 helper modules before model startup.

Matched protocol: same model file, same host and GPU, auto DPM policy, 8,192-token context, one request, same fixed prompt, greedy decoding, 256 requested output tokens, warmup, and three scored samples.

Runtime	Mean visible decode TPS	Median TPS	Quality suite
FreeToken ROCm/HIP	48.444	48.450	3/3 pass
llama.cpp ROCm 10	49.125	49.131	3/3 pass

The fresh same-format difference is 0.680 TPS, or 1.39% in favor of this llama.cpp control. This removes the previous NVFP4-versus-Q4_K_M representation mismatch. FreeToken is near parity but does not yet meet or exceed llama.cpp under this strict workload. A temporary high DPM policy was also screened and rejected because it reduced FreeToken Q4 throughput to 47.287 TPS while quality remained correct.

The normal NVFP4 FreeToken endpoint was restored after the time-share test and passed its deterministic AIME hash gate. The report includes artifact locations, startup geometry, the 512-token raw-prompt check, and the full method.

[dbourdea]

Final GMKtec EVO-X2 native ROCm/HIP Q4 recovery evidence is now on commit a937862.

• The one-hour endurance battery completed 60 of 60 deterministic three-turn sessions with every visible answer correct.
• The FreeToken runner process group stayed at 0 KiB swap for the entire run. Whole-host swap was retained separately as diagnostic telemetry, ranging from 33.07 MiB to 38.17 MiB.
• Maximum-turn TTFT: mean 0.424 s, p95 0.414 s, p99 and maximum 1.184 s.
• Maximum visible-token gap: mean 24.95 ms, p95 25.98 ms, p99 and maximum 27.17 ms.
• The normal NVFP4 service was restored after the battery. It completed an approximately eight-minute cold expert initialization, then passed a live OpenAI-compatible API request with the correct visible answer.

The matched exact-Q4 comparison remains transparent: FreeToken recovery configuration was 47.960 TPS versus fresh llama.cpp ROCm 10 at 48.831 TPS, 1.78 percent lower. This PR does not claim throughput parity where the measured result is below it.

Raw artifacts and the machine-checkable endurance summary are documented in docs/gmktec-evo-x2-rocm-validation-2026-08-30.md.

[andyelka-creator]

Tried this branch on a different ROCm target than the one validated here — a discrete gfx1011 card (AMD Radeon Pro V520 / BC-160, Navi12) instead of the gfx1151 APU (GMKtec EVO-X2) this PR's validation evidence above is based on. Wanted to share what I found in case it's useful, since the code paths that broke are architecture-agnostic (not gfx1151-specific).

Setup: 1× BC-160 (8GB), AMD TheRock nightly ROCm/PyTorch (torch==2.12.0+rocm10.2.0a20260921, device-gfx1011), this branch (amd-rocm-gfx1151) + a few files cherry-picked from PialGhosh2233/FreeToken-rocm-gfx1200 for a more complete qwen3_5_moe GGUF adapter. Model: Qwen3.6-35B-A3B-Q4_K_M.gguf (standard llama.cpp quant, not Unsloth-UD).

Bugs found getting it to load (all reproduce regardless of GPU arch, I think):

1. layers/moe.py make_moe_layer() doesn't accept weight_format or extra_attrs kwargs that qwen3_5_moe/moe.py passes to it — TypeError. extra_attrs carries gguf_expert_types/similar and needs a setattr(layer, name, value) loop after construction, not just accepting-and-dropping it (the value is read back later).
2. models/register.py has two name mismatches for the qwen35moe classes (Qwen3_5MoeGGUFForCausalLM vs Qwen35MoeGGUFForCausalLM, and a capitalization mismatch on Qwen3_5MoEForCausalLM) that raise a lookup KeyError/AttributeError at model-class resolution time.
3. nextn/MTP draft layer mishandled: GGUF metadata's nextn_predict_layers (1 in this checkpoint) isn't subtracted from block_count anywhere — the code treats the trailing MTP layer (blk.40.* here, real tensors: attn_q/k/v + a nextn.* head) as a normal layer and classifies it via full_attention_interval modulo math, which gets it wrong (predicted linear-attention, actually full-attention) → KeyError on linear_attn.in_proj.weight. Fix: exclude the trailing nextn_predict_layers blocks from num_hidden_layers, weight iteration, and expert-bank loading entirely (no speculative decoding support needed for this to just work).
4. ModelConfig.gguf_expert_bank_types / gguf_expert_layer_types are set via object.__setattr__ on ad-hoc (non-dataclass-field) attributes in qwen3_5_moe/gguf.py's parse_gguf_config. EngineConfig.model_config (engine/config.py) does return replace(model_config, quant=quant) right after — dataclasses.replace() only carries over declared fields, so both attributes are silently dropped, and the scheduler subprocess crashes with AssertionError: config was not built by qwen35moe parse_gguf_config. Fix: declare both as real ModelConfig fields (default None), matching the existing pattern of gguf_q6_down_layer_ids etc.

gfx1011-specific / hardware finding: --dtype bfloat16 (server default) crashes at CUDA-graph-capture time with LLVM ERROR: Cannot select: intrinsic %llvm.amdgcn.fdot2.bf16.bf16 — RDNA1 (gfx1010/gfx1011) has no bf16 dot-product intrinsic at the ISA level (added in RDNA2/CDNA). --dtype float16 avoids it. Worth a note in the README for anyone targeting RDNA1 discrete cards specifically, since gfx1151 (RDNA3.5) won't hit this.

Where I got stuck: after all of the above plus --moe-cache-size 256 --kv-reserve-tokens 256 --max-running-requests 1 --disable-moe-prefill-overlap (to fit 8GB) and --dtype float16, the server loads the model, compiles the native HIP GGUF kernels (had to manually vendor rocThrust/rocPRIM headers + hand-generate their version headers — TheRock's _rocm_sdk_core/_rocm_sdk_libraries don't ship them at all, that's a TheRock packaging gap, will file separately there), captures the CUDA graph, and serves HTTP — but every generation degenerates into a repeated ! token (both /v1/chat/completions and /v1/completions, temperature=0, reproducible across different prompts). Ruled out fp16 instability in the GDN/Mamba recurrent state (already fp32 by default via ssm_state_dtype()). Haven't isolated it further between the Triton fp16 attention path, the freshly-compiled native HIP GGUF-dequant/MoE kernels (never exercised on gfx1011 before, as far as I can tell — every validation report on this PR is gfx1151/gfx1200), or the Q5_1 bank requantization. Flagging in case anyone else hits the same wall on a discrete RDNA1 card, or has a pointer to where to look next.

Happy to share the exact patch diffs if useful — didn't want to open a PR against a WIP branch without checking first.

[dbourdea]

Rebase update:

• Rebased amd-rocm-gfx1151 onto current upstream main at cc1f5c2, then force-pushed the rebased head 9bdb73b.
• GitHub now reports no conflicts with main.

Focused local validation:

• tests/benchmarks/test_lan223_qwen_benchmark.py: 23 passed.
• tests/utils/test_accel_install_contract.py and tests/utils/test_rocm_setup_preflight.py: 4 passed.
• git diff --check passed.

HIP safety and qualification boundary:

• The ROCm Triton router remains opt-in (FREETOKEN_ROCM_TRITON_ROUTER=1); HIP defaults to the pure-Torch reference path.
• Live AMD full-model requalification was not rerun because the target host environment had drifted. No host services were changed, and this update makes no new performance or parity claim.

[andyelka-creator]

Follow-up after deeper isolation (rounds 2-6, same BC-160/gfx1011 host as above).

Ruled out further:

• GDN/Gated-DeltaNet recurrence is not the source: swapping gdn_decode_fla/gdn_prefill_chunk_fla for a bit-for-bit pure-PyTorch reference implementation (transcribed from HF's torch_recurrent_gated_delta_rule) produces the identical corruption.
• The native HIP GGUF-dequant/MoE kernels are not the source either: forcing --moe-strategy cpu (experts computed on host, avx2) still corrupts, just with a different failure signature (100% NaN logits vs. the GPU-offload path's exact-zero logits) — same visible !!!!!!! symptom via argmax of a degenerate tensor either way.

Narrowed to the GPU-offload routed-expert path on decode: layer-by-layer instrumentation (NaN/absmax after each sublayer, and separately for router/shared-expert/routed-expert inside Qwen3_5MoE.forward) shows prefill is consistently clean; corruption appears only on single-token decode steps, at a different layer each run, with router/shared_expert clean at the exact moment routed (the offload-cached expert GEMM) goes NaN. HIP_LAUNCH_BLOCKING=1 AMD_SERIALIZE_KERNEL=3 (fully serialized kernel execution) does not fix it, so it isn't a launch-ordering race.

The decode admission path (ensure_experts → flashlib.kernels.slot_cache.lru_ensure) is vendored from flashlib (FlashML-org/flashlib), whose own package metadata describes it as Triton+CuteDSL kernels for NVIDIA GPUs (requires nvidia-cutlass-dsl) — AMD/ROCm correctness was never claimed there. I pulled the compiled AMDGPU ISA for _lru_ensure_kernel from the local Triton cache and checked the tl.debug_barrier() the kernel's own comment flags as required (source line 184, guarding the LRU victim-selection reload against a stale pre-bump view): it compiles to a complete s_waitcnt lgkmcnt(0) + s_waitcnt_vscnt + s_barrier + buffer_gl0_inv sequence — correctly formed, not missing or malformed. So that specific hazard the kernel already guards against is not the cause here; something else in the same kernel (or in how FreeToken drives it) still produces per-token-dependent, non-reproducible-by-layer corruption specific to decode on this GPU.

Opened a separate issue against flashlib itself with the same evidence, since the kernel in question is theirs, not this PR's: FlashML-org/flashlib#24

Filing this as additional evidence rather than a fix — stopping active work here since it's now outside what source-reading/tracing can resolve without live kernel-level debugging (rocgdb) against flashlib's code specifically.

[dbourdea]

Validation follow-up for commit cbd5c278:

• The ROCm 7.14 resolver/install path and native host-extension build are now recorded for this exact candidate. The isolated Linux build passed pip check, built/imported _pinned_tensor, _cpu_moe, and _row_store, and the focused install/preflight tests passed 9/9 on both Windows and Linux. The ROCm compiler also accepted the edited fast-copy header with gfx1151 as the target.
• This was package/host-build validation only: LAN-215 is gfx1150, GPU execution was hidden, and no device kernel or model was run. It does not add new gfx1151 runtime or parity evidence. The build logs are retained privately and are not attached here.
• The September 22–23 report of Qwen3.6 decode corruption on gfx1011 used additional adapter changes and has since narrowed the failure to decode-time offload-routed experts; the related flashlib issue fix(cuda): support Turing GPUs #24 remains open. We have not reproduced or resolved that report here, and are not treating it as either evidence for or against this PR's gfx1151 qualification.

GitHub currently reports no check runs or status contexts for this head. No merge or new parity claim is implied.
CI configuration clarification (base f5b9700c): the committed .github/workflows tree contains only issue-labeling, scheduled/manual nightly wheels, manual beta promotion, and release workflows. It contains no pull_request trigger or unit-test workflow. Thus the zero checks for cbd5c278 reflect the current base workflow configuration; there is no AMD/ROCm CI result waiting to arrive. I have not added a CI workflow because no hosted AMD runner has been established for this repository.