Skip to content

feat(windows): gfx1102 CPU-MoE serve stack (hybrid, 8 GB) - #7

Merged
Maxritz merged 1 commit into
Maxritz:mainfrom
AronVanAmmers:windows-gfx1102-bundle
Sep 18, 2026
Merged

Maxritz merged 1 commit into
Maxritz:mainfrom
AronVanAmmers:windows-gfx1102-bundle

Conversation

@AronVanAmmers

Copy link
Copy Markdown

Summary

Bundled Windows/ROCm changes from the gfx1102 report on #82, ported onto this fork (not a dump of FlashML PR FlashML-org#132).

On a Framework Laptop 16 (RX 7700S, gfx1102, 8 GB, 96 GB RAM with 8.2 GB hardware reserved) openai/gpt-oss-20b served at ~10.5 tok/s with CPU experts and PCIe fetch off. Offload / default hybrid stay at ~0.02 tok/s; --moe-hybrid-max-fetch 0 is a bypass, not a HIP gather fix.

This fork already has TCP ZMQ, a global Selector policy, and GPU offload TDR work. This PR adds what was still missing for the 8 GB CPU-MoE path:

  • Build _cpu_moe on Windows with clang-cl + compiler-rt (MSVC would be scalar-only). Hoist <windows.h> out of the anonymous namespace so clang-cl atomics compile. Include hip_compat.cuh on ROCm.
  • physical_core_cpus() via GetLogicalProcessorInformationEx (sysfs fallback counted SMT as cores: 16 threads on 8, ~3x less CPU-MoE bandwidth).
  • Skip init_process_group when TheRock torch is USE_DISTRIBUTED=0 at TP size 1; guard all_reduce / shutdown.
  • Shared zmq_addr(): zmq.has("ipc"), crc32, ports below 49152, per-instance suffix (the previous SHA1 ports could land in the Windows ephemeral range and did not isolate two servers).
  • uvicorn loop="asyncio:SelectorEventLoop" (policy alone is not enough on uvicorn 0.52). Floor uvicorn>=0.36.

tvm-ffi hipcc flags are still a separate overlay: https://github.com/AronVanAmmers/tvm-ffi/tree/windows-hip-0.1.13-post3 (TVM_FFI_ROCM_ARCH_LIST=gfx1102).

Test plan

  • pytest tests/utils/test_zmq_addr.py
  • Framework 16 / gfx1102, clang-cl, TheRock HIP, overlay tvm-ffi, then:
$env:HIP_VISIBLE_DEVICES = "0"
$env:TVM_FFI_ROCM_ARCH_LIST = "gfx1102"
python -m freetoken.cli serve --model C:\models\gpt-oss-20b `
  --moe-backend hybrid --moe-cpu-threads 8 --moe-hybrid-max-fetch 0 `
  --moe-cache-auto --host 127.0.0.1 --port 1919
  • One-token completion (not /health) as the readiness gate

Build _cpu_moe with clang-cl, Windows physical-core topology, skip missing torch.distributed, and keep ZMQ TCP ports below the ephemeral range. Validated on Framework 16 / RX 7700S / gpt-oss-20b at ~10.5 tok/s with hybrid fetch disabled.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants