Amd glm52

[CjiW]

Motivation

Sync production AMD MI300X PD (Prefill-Decode) deployment code and binaries into the fork. The production cluster (Azure VMSS, SGLang v0.5.14 + AITER, GLM-5.2 FP8) runs from ACR images built on top of 5af1f949c; this PR makes the deployment state reproducible from git, and unifies the three role images into a single image.

🚀 Unified image (NEW — 2026-08-24)

The three role images previously shipped separately; they can be merged into one image since prefill/decode (Python server) and router (Rust binary) do not conflict:

Old (3 images)	New (1 image)
...-ropefix6 (prefill)	b200routeraca.azurecr.io/mindverse/sglang:v0.5.14-cp-layersplit-v17-unified
...-v17-patched (decode)	(digest sha256:2d7471420f60..., ~92GB)
...-cacheaware-fix (router)

Build method (unified v2 — REQUIRED full package replace, not just .so):

⚠️ v1 attempt (only overwriting .so) FAILED at runtime: Router.__new__() got an unexpected keyword argument 'prefill_group_ids' — the ropefix6 python router.py wrapper uses the NEW API while cacheaware Rust .so expects the OLD API. Must replace the whole sglang_router package (router.py + launch_router.py + mini_lb.py + router_args.py + .so) from the cacheaware image.

BASE=b200routeraca.azurecr.io/mindverse/sglang:v0.5.14-cp-layersplit-v17-patched-ropefix6
CA=b200routeraca.azurecr.io/mindverse/sglang:v0.5.14-cp-layersplit-v17-cacheaware-fix
NEW=b200routeraca.azurecr.io/mindverse/sglang:v0.5.14-cp-layersplit-v17-unified
# extract FULL sglang_router package from cacheaware image
CID_CA=$(docker create $CA)
docker cp $CID_CA:/opt/venv/lib/python3.10/site-packages/sglang_router/ /tmp/ca_sglang_router
docker rm $CID_CA
# build unified: base ropefix6 + overwrite full sglang_router package
CID=$(docker create $BASE)
docker cp /tmp/ca_sglang_router/. $CID:/opt/venv/lib/python3.10/site-packages/sglang_router/
docker commit -m "unified v2: ropefix6 + FULL cacheaware sglang_router pkg" $CID $NEW
docker push $NEW

Verify inside image:

docker run --rm $NEW grep -c prefill_group_ids /opt/venv/.../sglang_router/router.py  # must be 0 (cacheaware old API)
docker run --rm $NEW md5sum /opt/venv/.../sglang_router/sglang_router_rs.abi3.so        # 84946bbb...
docker run --rm $NEW md5sum /opt/venv/.../mooncake/engine.cpython-310-x86_64-linux-gnu.so # 0e1b8fdd...

End-to-end verified 2026-08-24 (unified v2, node-0 prefill + node-4 decode+router):

• prefill/decode /health_generate 200 (1s)
• router workers_count=2, chat completion HTTP 200 (0.46s)

Role launch (same image, different entrypoint):

Role	Command
prefill	python -m sglang.launch_server --disaggregation-mode prefill --port 30100 ...
decode	python -m sglang.launch_server --disaggregation-mode decode --port 30200 ...
router	python -m sglang_router.launch_router --pd-disaggregation -p <prefill> -d <decode> ...

All three ship: python fixes (ropefix4/5 + 25-file AMD code + cp_layersplit_pool.py) + mooncake engine_v3.so (md5 0e1b8fdd...) + router sglang_router_rs.abi3.so (cacheaware, md5 84946bbb...).

Modifications

1. Python source fixes (verified byte-identical against deployed image)

• utils.py (dsa_cp_round_robin_split_data): add .contiguous() to input_[indices] — fixes aiter rope stride assertion crash when tokens % cp_size != 0 (single-element tensor stride=[8] non-contiguous view).
• deepseek_v2.py (rebuild_cp_kv_cache): .clone() before slice write — fixes "input tensor and written-to tensor refer to a single memory location" (CP path view-alias between k_pe/k_nope and latent_cache).
• AMD-specific bring-up code (25 files, +513/-116): HiCache (hicache.cuh/py, decode_hicache_mixin, hicache_storage, memory_pool_host), CP layersplit (cp_utils +84), DSA (dsa_indexer, dsa_backend), PD disaggregation (conn/decode/prefill/pd_disaggregation_hook), unified_radix_cache (+109, tenant-aware eviction), mooncake_store, server_args (+65), etc.
• cp_layersplit_pool.py (new, 318 lines): CP layer-split KV pool, imported by 8 modules (dsa_backend, prefill, cache_controller, model_runner_kv_cache_mixin, kv_cache_builder, hybrid_pool_assembler, disaggregation/utils). Was present in deployed image (md5 485081e3) but never tracked in git.
• cache_aware.rs (sgl-model-gateway): low-cache-match policy uses smallest tree size (cache-aware routing), not min load.

2. Binary patches (via git-lfs, patches/)

• patches/mooncake/engine_v3.so (67 MB, LFS, md5 0e1b8fdd2d39e007b58c3e1711ec398b): the only binary patch needed for PD disaggregation. Official mooncake engine.so segfaults on "Tree cache initialized"; V3 image version works. libasio.so + store.so are identical to official (no patch needed).
• patches/mooncake/engine_dmabuf_patched.so (7 MB, LFS): dmabuf variant.
• patches/mooncake/sglang_router_rs_abi3_router.so (42 MB, LFS, md5 84946bbb...): router image's separately-compiled Rust router binary (cacheaware-fix build; prefill/decode images ship 36.6 MB 395c3a11...).
• patches/mooncake/patch_mooncake_hip_transport.sh + patch_mooncake_dmabuf.sh: source patch scripts (see below for what they change).
• patches/mooncake/hip_approach_b.h + nvlink_allocator_approach_a.cpp: HIP transport approach source (approach B / nvlink allocator).

3. What the mooncake patch changes (code-level)

Two issues fixed in mooncake-transfer-engine (source in /sgl-workspace/Mooncake/mooncake-transfer-engine/src):

Issue 1 — HIP IPC transport hijacks cross-node transfers → segfault

mooncake was compiled with -DUSE_HIP=ON -DENABLE_MULTI_PROTOCOL=ON, registering both HIP IPC transport (priority=4) and RDMA transport (priority=2). Since HIP priority > RDMA, cross-node transfers wrongly picked HIP IPC — which only supports same-node — producing hipIpcOpenMemHandle failed → segfault.

// transfer_engine_impl.cpp: skip HIP transport registration entirely
- #ifdef USE_HIP
+ #ifdef USE_HIP_DISABLED

// multi_transport.cpp: drop HIP priority below RDMA (4 → 0)
- return 4;   // HipTransport::priority()
+ return 0;

Issue 2 — GPUDirect RDMA (dmabuf) not propagated to rdma_transport target

CMakeCache had USE_HIP_DMABUF:BOOL=ON but the flag wasn't passed to the rdma_transport compile target, so dmabuf registration never happened.

# transport/rdma_transport/CMakeLists.txt (append)
target_compile_definitions(rdma_transport PRIVATE USE_HIP_DMABUF)

Apply + rebuild:

# inside the mooncake source tree (docker build stage):
bash patch_mooncake_hip_transport.sh        # fix issue 1 (apply)
bash patch_mooncake_dmabuf.sh               # fix issues 1+2, recompile engine.so
# or use prebuilt: patches/mooncake/engine_v3.so (md5 0e1b8fdd...)

4. Worker role image matrix (for reference — superseded by unified image)

Role	Old image	Python fixes	Rust router .so
prefill	...-ropefix6	full (ropefix + AMD + cp_layersplit_pool)	395c3a11... 36.6MB
decode	...-v17-patched	AMD w/o ropefix	395c3a11... 36.6MB
router	...-cacheaware-fix	—	84946bbb... 42.3MB

Note: 7 triton_3_1_0 MI300X moe configs exist in main but were absent in the deployed image (build-time cleanup); intentionally NOT deleted here (valid tuned params). patches/v3/*.diff (topk FP8 precision / layers LoRA aliasing / aiter LoRA residual) are B300-environment patches, NOT applied to this 1p3d deployment — kept for reference only.

Accuracy Tests

• Verified on production cluster (2p2d, 4× MI300X 192GB): streaming requests return usage, warmup 200, dynamic worker register/remove OK.
• ropefix: shape=[1] stride=[8] crash no longer occurs (was 8× before every crash).
• mooncake engine_v3.so: long-request KV transfer 0 errors (previously segfault on official engine.so).
• Unified image: engine.so + router.so md5 verified inside image after build.