Conversation
paralin
force-pushed
the
amd
branch
3 times, most recently
from
August 24, 2026 08:52
22b6aee to
7e1d742
Compare
FreeToken built and served only against CUDA: setup.py linked its host-side extensions against cudart, the kernel JIT compiled with nvcc-only flags, and several triton call sites passed NVIDIA-only launch options. On an AMD GPU the engine could not even finish booting. Build _pinned_tensor and _cpu_moe against HIP through a shim header that maps every CUDA runtime symbol they use onto its HIP equivalent. Teach the kernel JIT the same trick: drop --expt-relaxed-constexpr (hipcc rejects it; relaxed constexpr is already its default) and force-include a shim, packaged with the kernel csrc tree, that maps the CUDA launch-config surface onto hipLaunchKernelEx. PDL launch attributes have no equivalent on this runtime; every kernel served on AMD builds with use_pdl=false, so the shim drops attributes instead of setting them. Three smaller fixes complete the port: - norm.py and activation.py omit the triton launch_pdl keyword under HIP; the AMD launcher rejects it outright. NVIDIA keeps the upstream launch_pdl=pdl call for both true and false. - attention.py floors block_h at 16 under HIP because RDNA3 WMMA cannot select an instruction for tl.dot below M=16. Padded head lanes were already masked. - fast_index_copy.cuh guards its PTX streaming-load/store inline asm: CUDA keeps it verbatim; HIP has no equivalent asm, so those builds fall back to plain device loads and stores. The cache-policy hints are dropped on HIP only; correctness is unchanged. setup.py detects the backend from torch: ROCm builds of torch take the HIP branch, CUDA builds keep the original cudart link and nvcc toolchain check unchanged, and anything else fails with a clear error. Every other edit is gated on HIP detection at run time. ROCm installs should resolve torch from the ROCm wheel index first; the pinned PyPI triton conflicts with the ROCm-bundled one.
7 tasks
skywalk1411
added a commit
to skywalk1411/FreeToken
that referenced
this pull request
Aug 27, 2026
Both were made mid-investigation, before the real cause of a since-fixed crash (the raw-PTX bug in activation.py, and separately the e4m3_native() tuple- comparison bug) was actually found. Re-tested each in isolation -- eager, batched, and inside real CUDA graph capture+replay -- now that those are fixed, and both work fine at the original, CUDA-tuned settings: - decode_paged_attention: the block_h>=16 floor (kept -- RDNA WMMA genuinely has no instruction below M=16, confirmed independently and matches upstream FlashML-org#137) was sufficient on its own. The USE_TL_DOT broadcast-sum fallback this PR had added on top was solving a problem that was actually in a different kernel; removed, restoring real matrix-core-accelerated decode attention. - _select_extend_tile: the 128x64 -> 64x32 shrink on HIP was diagnosed as a VGPR-exhaustion issue via a py-spy trace mid-investigation, before the session had isolated the actual crash to activation.py. Re-verified end-to-end against Qwen3.6-35B-A3B-FP8's GDN/split-extend path (the kernel this shrink targeted) at the original tile size: no crash, correct output. Reverted to the CUDA-tuned tile. Both re-verified against real chat completions (Qwen3-8B for the decode path, Qwen3.6-35B-A3B-FP8 for the extend/split path) after reverting, not just the isolated kernel tests.
dbourdea
pushed a commit
to dbourdea/FreeToken
that referenced
this pull request
Aug 29, 2026
Both were made mid-investigation, before the real cause of a since-fixed crash (the raw-PTX bug in activation.py, and separately the e4m3_native() tuple- comparison bug) was actually found. Re-tested each in isolation -- eager, batched, and inside real CUDA graph capture+replay -- now that those are fixed, and both work fine at the original, CUDA-tuned settings: - decode_paged_attention: the block_h>=16 floor (kept -- RDNA WMMA genuinely has no instruction below M=16, confirmed independently and matches upstream FlashML-org#137) was sufficient on its own. The USE_TL_DOT broadcast-sum fallback this PR had added on top was solving a problem that was actually in a different kernel; removed, restoring real matrix-core-accelerated decode attention. - _select_extend_tile: the 128x64 -> 64x32 shrink on HIP was diagnosed as a VGPR-exhaustion issue via a py-spy trace mid-investigation, before the session had isolated the actual crash to activation.py. Re-verified end-to-end against Qwen3.6-35B-A3B-FP8's GDN/split-extend path (the kernel this shrink targeted) at the original tile size: no crash, correct output. Reverted to the CUDA-tuned tile. Both re-verified against real chat completions (Qwen3-8B for the decode path, Qwen3.6-35B-A3B-FP8 for the extend/split path) after reverting, not just the isolated kernel tests.
samuelishida
pushed a commit
to samuelishida/FreeToken
that referenced
this pull request
Sep 5, 2026
What: - Remove .agents/learnings and .plans/rocm-consolidation files from the branch. - Remove internal increment and plan-path references from source comments and public installation docs. - Keep implementation comments that explain correctness, ownership, profiler intent, source attribution, or ROCm safety behavior. - Clarify public ROCm documentation: gfx1100 has recorded serving smoke on ROCm 7.2.1; the ROCm 7.14.x container is a reference environment, and other target cells remain compile-only until physical serving evidence exists. Why: - Keep merge surface focused on code, tests, reproducibility tooling, and user-facing documentation. - Prevent private planning history, review workflow language, stale plan paths, and local process notes from entering the upstream repository. - Avoid presenting compile success or a reference container as cross-target serving or performance proof. Related upstream work informing this branch: - PR FlashML-org#132: portable ROCm/HIP foundation. - PR FlashML-org#133: TVM-FFI index/store portability. - PR FlashML-org#135: RCCL tensor-parallel communication. - PR FlashML-org#136: native GGUF ROCm build and Q4_0 kernels. - PR FlashML-org#137: earlier AMD serving bring-up. - PR FlashML-org#217: source-fork ROCm, Qwen3.5 GGUF, and performance experiments. - PR FlashML-org#241: gfx1150 build, JIT, Triton, and attention hardening. - PR FlashML-org#260: gfx1151 validation and fallback/build evidence. - PR FlashML-org#316: HIP graph-capture-safe expert copies. - PR FlashML-org#378: CPU/Hybrid MoE graph replay safety. - Local branch milestones: 436263f, 926c1e8, e1d1856, 8a70c7e, and e5fd30f. Evidence: - 170 focused tests passed after cleanup. - gfx1100 is the only target with end-to-end Qwen3.5 GGUF serving smoke recorded here. - Remaining matrix targets are compile-only; no new throughput claim is published without a matching A/B manifest.
ezutfen
pushed a commit
to Zutfen-LLC/FreeToken
that referenced
this pull request
Sep 12, 2026
…L-org#153 maintainer review) Corrected classification: Branch B BACKEND_REQUIRES_MULTI_CHUNK, terminal ISSUE117_ARM_C_REMEDIATION_BLOCKED. - The accepted FlashML-org#137 failing population (prompt_len 65-67, all >64) is multi-chunk by frozen-contract necessity: the R4 wire contract rejects token_count > max_token_count (64), the frozen boundary geometry (prefill_bytes = 64*3840*2, activation staging buffers) and the wire receive buffer are sized for exactly 64 rows. Runtime capacity 256 is session KV capacity, not boundary authority. A 65-67-row single call is therefore inadmissible without changing a frozen semantic/wire contract, which exceeds FlashML-org#153 authority — Branch A (UNNECESSARY_PARTITION_POLICY) was wrongly derived from the C2 53-row control's legality and is withdrawn. - The <=64 single-chunk policy cleanup is RETAINED as useful but does NOT remediate the accepted failing population: 65/66/67 still execute as the unchanged accepted 64+remainder path. No CPU-provable backend/state defect exists in the chunk-2 extend path (source inspection: metadata/causal semantics correct, fixed tiles, no autotune/atomics; instability is execution-level per FlashML-org#137) — branch-B option 1 unavailable without new physical evidence, so BLOCKED. - Timing-unit regression fixed: two_stage prefill accumulator restored to perf_counter_ns on both sides (prefill_ns field is nanoseconds); structural AST regression contract added (no bare perf_counter call survives; every *_ns field is ns-sourced). - Capacity ownership corrected: strategy (frozen constant owner) -> stage_chain/two_stage (chunk policy) and strategy -> last_stage_service (wire bound). The wire service no longer imports the chain runtime (removes the service->coordinator dependency of the reviewed head); no runtime module imports the wire service; sender and receiver derive from the same frozen constant. - 65-67 static/reduction controls added: exact failing population classified multi-chunk, partition equal to the accepted hand-literal loop, request-content equality, C2 53-row retained as causal control only (explicitly not a failing-population substitute). Focused suite: 79 passed. Full research suite: 532 passed, 1 deselected pre-existing base failure, 72 subtests. CPU-only; no GPU, no node access, no h109-*.
KE7
added a commit
to KE7/FreeToken
that referenced
this pull request
Sep 19, 2026
Floor the decode-attention GQA head tile (BLOCK_H) at 16 under HIP. RDNA WMMA has no matrix-core instruction below a 16x16 tile, so a decode GQA group smaller than 16 (e.g. group=4) leaves tl.dot's M dimension too small to lower on this backend. The kernel already masks lanes >= VALID_BLOCK_H for non-power-of-two groups, so padding BLOCK_H up to 16 is safe: it only adds masked-out, discarded head lanes. Gated on torch.version.hip, so the CUDA path is untouched. Surgically extracted from pr-241/pr-260 (skywalk1411's gfx1150 Strix Halo work), not a raw cherry-pick: those branches bundle additional out-of-scope content (extend-tile VGPR shrink experiments later reverted as unnecessary, vision/perf-tuning deltas). Only the one change that branch's own investigation confirmed was needed and kept (matches upstream FlashML-org#137's independent finding of the same M<16 WMMA limit) is hand-ported here. NOT YET RUN on real hardware -- requires independent verification on gfx1150/gfx1151 (Strix Halo / Ryzen AI Max, Radeon 8060S/890M) before this is trusted in production. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
KE7
added a commit
to KE7/FreeToken
that referenced
this pull request
Sep 19, 2026
Floor the decode-attention GQA head tile (BLOCK_H) at 16 under HIP. RDNA WMMA has no matrix-core instruction below a 16x16 tile, so a decode GQA group smaller than 16 (e.g. group=4) leaves tl.dot's M dimension too small to lower on this backend. The kernel already masks lanes >= VALID_BLOCK_H for non-power-of-two groups, so padding BLOCK_H up to 16 is safe: it only adds masked-out, discarded head lanes. Gated on torch.version.hip, so the CUDA path is untouched. Surgically extracted from pr-241/pr-260 (skywalk1411's gfx1150 Strix Halo work), not a raw cherry-pick: those branches bundle additional out-of-scope content (extend-tile VGPR shrink experiments later reverted as unnecessary, vision/perf-tuning deltas). Only the one change that branch's own investigation confirmed was needed and kept (matches upstream FlashML-org#137's independent finding of the same M<16 WMMA limit) is hand-ported here. NOT YET RUN on real hardware -- requires independent verification on gfx1150/gfx1151 (Strix Halo / Ryzen AI Max, Radeon 8060S/890M) before this is trusted in production. Assisted-by: Claude Sonnet 5
dbourdea
pushed a commit
to dbourdea/FreeToken
that referenced
this pull request
Sep 20, 2026
Both were made mid-investigation, before the real cause of a since-fixed crash (the raw-PTX bug in activation.py, and separately the e4m3_native() tuple- comparison bug) was actually found. Re-tested each in isolation -- eager, batched, and inside real CUDA graph capture+replay -- now that those are fixed, and both work fine at the original, CUDA-tuned settings: - decode_paged_attention: the block_h>=16 floor (kept -- RDNA WMMA genuinely has no instruction below M=16, confirmed independently and matches upstream FlashML-org#137) was sufficient on its own. The USE_TL_DOT broadcast-sum fallback this PR had added on top was solving a problem that was actually in a different kernel; removed, restoring real matrix-core-accelerated decode attention. - _select_extend_tile: the 128x64 -> 64x32 shrink on HIP was diagnosed as a VGPR-exhaustion issue via a py-spy trace mid-investigation, before the session had isolated the actual crash to activation.py. Re-verified end-to-end against Qwen3.6-35B-A3B-FP8's GDN/split-extend path (the kernel this shrink targeted) at the original tile size: no crash, correct output. Reverted to the CUDA-tuned tile. Both re-verified against real chat completions (Qwen3-8B for the decode path, Qwen3.6-35B-A3B-FP8 for the extend/split path) after reverting, not just the isolated kernel tests.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
FreeToken built and served only against CUDA: setup.py linked its host-side extensions against cudart, the kernel JIT compiled with nvcc-only flags, and several triton call sites passed NVIDIA-only launch options. On an AMD GPU the engine could not even finish booting.
Build _pinned_tensor and _cpu_moe against HIP through a shim header that maps every CUDA runtime symbol they use onto its HIP equivalent. Teach the JIT the same trick: drop --expt-relaxed-constexpr (hipcc rejects it; relaxed constexpr is already its default) and force- include a shim that maps the CUDA launch-config surface onto hipLaunchKernelEx. PDL launch attributes have no equivalent on this runtime; every kernel served on AMD builds with use_pdl=false, so the shim drops attributes instead of setting them.
Three smaller fixes complete the port:
Drop the explicit triton==3.6.0 pin so the ROCm-bundled AMD-backend triton survives installation; torch's own wheels already constrain their matching triton build. Every edit is gated on HIP detection at build or run time, so NVIDIA paths are unchanged.