feat(mol): SGLANG_OPT_MOL_LORA_STAGE_CFG — MoL LoRA stage N-tile 调参（几何生效；实测 −3.9%，非 CTA-bound，默认关） - #24
Merged
Conversation
profile 实测（1022 decode, bs=20/23, 60 步, cat=kernel）：NCCL 通信占 kernel busy 的 20.3-21.2%，其中 TP 维 bf16 AllReduce（ncclDevKernel_AllReduce_Sum_bf16_RING_LL） 稳定占 10.4-10.5%，是通信里最大单项。本 commit 按 Ferrite 的 ferrite_p2p_ar_v5 协议 实现 NVLink P2P 全归约，入口改为 bf16 进/出 + fp32 累加（对齐 torch/NCCL bf16 AR 语义），用 flag 启用、不满足条件自动回落。 * kernels：store（把自己 partial 写进每个 peer 的 staging 槽 → publish ready） + pubred（publish 自己的 ready → poll 各 peer → 按 rank 升序 fp32 累加 → bf16 写回） epoch 双缓冲（(e&1)*world + my_rank），__threadfence_system 保证 store 先于 flag； grid 在 store 上是 (blocks, world) 三维，把 peer 写并发化。 * host 侧：IPC peer 指针表（cudaIpcGetMemHandle/OpenMemHandle）、staging [2][world][stride]、ready 表、epoch 计数器、ctypes 启动；**全部初始化时预分配** → cuda-graph 可捕获（decode 现在 100% graph 命中）。 * 接线：GroupCoordinator 建 self.p2p_ar；_resolve_outplace_all_reduce_method 优先 返回 "p2p"；_all_reduce_out_place 分发（先 clone 再原地 reduce，遵守 out-of-place 契约）。 * 降级矩阵（silently fall back 到原路径）：非 bf16 / 非连续 / numel%4!=0 / 超 MAX_ELEMS / world∉[2,8] / 非单机 NVLink-P2P / IPC 失败。 * 开关：SGLANG_P2P_AR=1 + SGLANG_P2P_AR_LIB=/path/libp2p_ar.so (+ SGLANG_P2P_AR_MAX_ELEMS，默认 8M 元素)；build.sh 独立构建 .so（不重建 sglang/sgl_kernel），B300 用 long-form -gencode arch=compute_103a,code=sm_103a。 未验证（下一个 commit / 部署步骤）：.so 构建、P2P vs NCCL 逐元素正确性、graph 内 100 次 epoch 推进、A/B 性能与 cases_50 回归。上线需重启（SGLANG_P2P_AR 是启动 env）。
* tools/analyze_step_profile.py：把 torch-profiler trace 按 kernel 名分类成 moe/attn/comm(AR,AG,RS,p2p)/lora/eagle/norm/elementwise/other，并输出 top-kernel。 **只统计 cat=="kernel"** —— profiler 的 gpu_user_annotation 是嵌套 span， 算进来会虚高 ~4 倍且分类错位（第一版把 eagle 报成 59.6% 就是这个坑）。 * docs/agent/job-step-profile.md：采集方法（引擎自带 /start_profile + num_steps 自动停，**不重启**）、窗口纪律（warmup / 同 bs 同 seq_len / profiler 开销 ~10-16% 所以绝对延迟以日志法为准）、分类口径、三窗口实测表、结论。 实测（1022 decode，60 步，TP0，cat=kernel）： bs≈11: comm 20.1% (AR 10.7 / AG 5.5 / RS 3.9) MoE 43.4% wall/step 85.6ms bs=23 : comm 20.3% (AR 10.5 / AG 5.7 / RS 4.2) MoE 45.7% wall/step 99.3ms bs=20 : comm 21.2% (AR 10.4 / AG 6.7 / RS 4.1) MoE 44.6% wall/step 95.5ms 结论：AR 稳定 10.4-10.7%（占通信 ~52%，最大单项，RING）；通信合计 20-21% 不随 batch 恶化；单点最大仍是 MoE 44-46%。⇒ P2P AR 收益上限 ≈ step −5~6%。
The arch probe piped nvcc --help straight into `grep -q`. grep exits on the first match, nvcc then dies of SIGPIPE (141), and with `set -o pipefail` the whole pipeline evaluates as a failure -- so the script took the "no compute_103a support" branch and emitted a .so with only sm_90a/sm_80 cubins. The build still "succeeded" (exit 0, file produced), so on the B300 boxes this only surfaced at load time: the extension cannot run on SM103. Observed on 1022 while deploying: `cuobjdump -lelf libp2p_ar.so` listed only sm_90a/sm_80, even though /usr/local/cuda/bin/nvcc (13.0) does advertise compute_103a. Capturing the help text first fixes the probe; the rebuilt .so now contains sm_103a (verified with cuobjdump).
…error 1)
Both PD nodes came up with, on every rank:
p2p_ar: init failed (cuda error 1); disabled
cudaErrorInvalidValue came from cudaIpcOpenMemHandle: cudaIpcMemHandle_t is a
64-byte struct and the x86-64 ABI classifies it as MEMORY, so it is passed by
value. The code passed a `ctypes.c_char * 64` array, which ctypes marshals as a
*pointer* to those bytes -- the callee then reads the pointer value as the handle
payload and rejects it.
Pinned with a two-process probe on the B300 decode node (allocate + get the
handle in process A, open it in process B, same GPU):
c_char*64 array, flags=cudaIpcMemLazyEnablePeerAccess -> err=1
struct by value, flags=cudaIpcMemLazyEnablePeerAccess -> err=0 (ptr resolved)
struct by value, flags=0 -> err=1
Fixes:
* add the _IpcMemHandle (64-byte) ctypes.Structure and use it for both
cudaIpcGetMemHandle (byref) and cudaIpcOpenMemHandle (by value);
* set explicit argtypes/restype for the three IPC calls so ctypes can never
fall back to integer marshalling;
* keep the lazy-peer-access flag (flags=0 fails, as the probe shows).
Python-only change: no .so rebuild needed.
…and grid blowup Root cause (measured on the B300 decode node, bs=23, 60-step profiler window, kernel-only/TP0): the virtual-experts (MoL) LoRA shrink/expand stages reuse the MoE stage-config path, and for these shapes (num_experts = max_loras x n_routed, K = rank) the autotuner has no entry, so the fallback BLOCK_SIZE_M=64 / BLOCK_SIZE_N=64 is used. _get_routing() hands BLOCK_SIZE_M to moe_align_block_size, which pads every touched virtual-expert bucket to a full 64-row block: ~1.6k blocks x 64 rows ~= 1e5 padded rows for ~1.1k real (token, virtual-expert) pairs (~93x inflation). The launch grid is grid = blocks * cdiv(N, BLOCK_SIZE_N), so the down-delta expand alone launches 153,984 CTAs and takes 160.6us/layer (~14% of an 85.1ms decode step) at ~85GB/s effective -- ~1% of HBM roofline -- while its gate_up siblings (grid=6416) do the same kind of work in 7.5/6.8us. Three measured grids reconcile exactly with the formula: 153984 = 1604*96 (N=6144), 6416 = 1604*4 (N=256), 1604 = 1604*1 (rank 2r=64). Fix: SGLANG_OPT_MOL_LORA_STAGE_CFG (default off) retunes only BLOCK_SIZE_N of the MoL stages to min(256, stage GEMM N), i.e. 4x fewer N-tiles. BLOCK_SIZE_M is deliberately left alone: grid_m = cdiv(padded_rows, BLOCK_SIZE_M) == blocks is invariant to it (align pads to that unit) and shrinking it only splits buckets into more blocks. BLOCK_SIZE_K and the K-loop are untouched, so every output element still reduces over K inside a single tile -> bit-identical. Gain estimate: 153,984 -> 38,496 CTAs; if tile/CTA-bound as the measurements indicate, 160.6us -> ~40us per layer (~9-10ms/step ~= 11% of step time). Also updates AGENTS.md (section 8 pitfall, section 9 index) and adds docs/agent/mol-lora-stage-cfg.md with the evidence, the arithmetic and the A/B acceptance protocol.
Code review of the previous commit found one real (currently masked) bug plus two hardening items and one wrong statement in the comments/docs: 1. Bug -- shrink call site passed the wrong stage N. The shrink kernel documents its weight as B[E, N, K] and _invoke_moe_lora_shrink_splitk itself reads N = weight.shape[1], but we passed lora_a_virtual.shape[2] (the contraction dim). It caused no damage only because that invoke path clamps BLOCK_SIZE_N = min(config, max(16, N)) -- i.e. another function's guard silently swallowed it. Pass shape[1] so the semantics match the kernel. 2. Harden -- apply the override only when the autotuner fell through to the fallback config (used_fallback). Never retune a config the autotuner actually produced for these shapes. 3. Harden -- read the flag live instead of caching its value (the EnvBool object is what gets cached); matches how the other SGLANG_OPT_* flags are read and keeps envs.<X>.override() working. 4. Correct the grid arithmetic in the code comment, environ.py, AGENTS.md and docs/agent/mol-lora-stage-cfg.md. The launch grid is built host-side as cdiv(sorted_token_ids.shape[0], BLOCK_SIZE_M) * cdiv(B.shape[1], BLOCK_SIZE_N) and sorted_token_ids is the worst-case *untrimmed* routing buffer, so grid_m ~= virtual_E + numel/BLOCK_SIZE_M only weakly depends on BLOCK_SIZE_M. The previous "grid_m == blocks, invariant to BLOCK_SIZE_M" wording was wrong. The conclusion is unchanged: BLOCK_SIZE_N is the lever (4x fewer N-tiles). Logic test extended to pin all four points (21 checks, all pass).
The kernel-side effect of SGLANG_OPT_MOL_LORA_STAGE_CFG is not observable on this
deployment (the engine's /start_profile|/stop_profile path returns 500 and never
writes a trace), so the flag's effect has to be provable from the logs: log once,
per process, the stage configs actually handed to the kernels.
grep MOL-PROBE shows shrink/expand BLOCK_SIZE_{M,N,K}; with the flag on the
expand BLOCK_SIZE_N must be min(256, N) (64 before), which is exactly what turns
num_pid_n = cdiv(N, BLOCK_SIZE_N) into a 4x smaller launch grid.
…loyment SGLANG_P2P_AR=1 was never exercised on hardware before today; bringing it up on 1021/1022 surfaced three real bugs in the init chain, all now fixed in code and documented with the measurement that pinned each one: 1. cudaIpcMemHandle_t is passed BY VALUE (64-byte MEMORY-class struct); passing a `ctypes.c_char * 64` array marshals a pointer instead and the open fails with cudaErrorInvalidValue (1). Two-process probe: array -> err=1, struct by value -> err=0, struct with flags=0 -> err=1 (lazy peer access is required). 2. `handle.bytes` truncates at the first NUL (only ~19/64 bytes are non-zero), so the published handle ended up 4 (sometimes 0) elements long -> gloo all_gather "invalid tensor size at index 0 (expected (4), got (64))". Read it with ctypes.string_at(..., 64), and exchange it as CPU tensors (the group is gloo). 3. build.sh probed `nvcc --help | grep -q compute_103a` under `set -o pipefail`: grep exits early, nvcc dies of SIGPIPE, the pipeline reads as a failure, and the script silently emitted a .so without sm_103a (still exit 0). Capture the help text first; verify with cuobjdump -lelf. Also notes that this deployment's engine profiler is unusable (stop_profile -> "Profiling is not in progress", no trace ever written), so kernel-level evidence has to come from host-side one-shot log probes.
…lowed)
Fourth init bug, exposed only after the IPC handle issues were fixed:
p2p_ar: init failed (Attempted to set the storage of a tensor on device
"cpu" to a storage on different device "cuda:N". This is no longer allowed;
the devices must match.)
staging_local/ready_local were built as `torch.Tensor().set_(self._blob, ...)`:
torch.Tensor() is a CPU tensor, self._blob is CUDA, and modern torch rejects
re-pointing a CPU tensor at a CUDA storage. Slice the CUDA blob and .view() it
instead, which keeps the storage on the right device.
… progress
With SGLANG_P2P_AR=1 actually enabled (all 4 init bugs fixed), the decode engine's
graph capture deadlocked on 1022:
[15:21:16] Capture target verify CUDA graph begin ... Capturing batches
(bs=64 avail_mem=22.70 GB): 0%| | 0/27
(8+ minutes later) still 0/27, all 8 GPUs pinned at 100% utilization
Every earlier boot (p2p_ar failing -> NCCL fallback) finished capture in ~1-2
minutes, so the P2P kernel is the trigger: it spin-waits on peer ready flags, and
during capture ranks walk shapes/steps in their own order, so the waits pair up
wrongly and the capture never completes.
Guard: should_p2p_ar() returns False while torch.cuda.is_current_stream_capturing(),
so captured graphs use the fallback all-reduce (NCCL) and eager steps keep using
P2P. One-shot log line + kill-switch SGLANG_P2P_AR_CAPTURE_GUARD=0.
… measured result (-3.9%, not CTA-bound)
Chronostasys
force-pushed
the
feat/mol-lora-stage-cfg
branch
from
September 15, 2026 09:04
75fe39e to
b060576
Compare
san-tian
pushed a commit
that referenced
this pull request
Sep 17, 2026
背景：分支上有了新修复（PD 隐式重载守卫、base 不占池槽…）之后，上机有两条路 —— 挂 overlay 单文件（少挂一个挂载点会被 Docker 建成目录 → 引擎起不来），或重建整镜像。 这里给第三条、也是最短的一条：在 v3 上**叠加目标 commit 的代码**。 为什么叠加就够（2026-09-17 实测，详见 README §7）：v3 的 python/ 树 == commit 0524363 —— 3544 个文件里 3445 个 sha256 一致，8 个不同（3 个是我们自己的 overlay 文件、5 个是被 构建期改写的 pyproject*.toml），另有 91 个构建期产物不在 git 树里（sglang.egg-info/*、 sglang/_version.py、jit_kernel/csrc/**.cuh …）。COPY 只覆盖同路径文件，不会删掉它们。 新增： - deploy/b300/glm53/Dockerfile.rebuild：FROM v3 + COPY sglang-src/ + 构建期自检 （import 路径必须落在源码树、compileall、_version.py/.egg-info 仍在、mooncake engine.so md5 未变、python/ 下 .so 计数 = 0），可选 --with-p2p-ar 编译 #24 的 P2P AR 扩展； - deploy/b300/glm53/rebuild-image.sh：git archive 目标 ref → docker build → 自检 （镜像里 python/** 的全量 sha256 清单必须与该 commit 逐字节一致；探针逐项对比基础镜像： .so 数、jit .cuh 数、/opt/p2p_ar 与 flag 一致、mooncake md5）。**全程没有 docker push**， 结尾直接打印 docker save → scp → docker load 的分发命令与回滚方式； - README §7：流程、自检表、上机与回滚（LMS_ENGINE_IMAGE 换 tag + 两端配对重启）。 实现细节：清单用 find -print0 | xargs -0 sha256sum —— 树里有含空格的路径 （".../block_shape=[128, 128].json"），普通 xargs 会拆坏（干跑时踩到才修）。自测： 源⊂镜像（含多余构建产物）→ 通过；任一文件被旧版覆盖 → 失败。
Chronostasys
pushed a commit
that referenced
this pull request
Sep 18, 2026
* feat(lora): base 不再占池槽 —— N 个槽位给 N 个 adapter（省 11.05GB/引擎）
base 模型不需要 LoRA delta，但旧实现会在 pool 里给它分一个完整的 buffer 槽
（rank 16 下 11.05GB，B300 实测），于是 N 个槽只能驻留 N-1 个 adapter；而且 base
是会被 LRU 驱逐的（见 mem_pool 里移除 prefer-LoRA hack 的那段注释），驱逐后 slot 0
归属某个 adapter，此时若 base 请求的 metadata 还是默认 slot 0，就会静默套上那个
adapter 的 delta（输出错但不报错）。
改动：
- mem_pool.prepare_lora_batch：uid is None 直接 continue —— 不分槽、不分配 buffer，
也不进入驱逐候选集；
- lora_manager.validate_lora_batch：base(None) 不计入 per-batch 上限与 pinned 记账
（否则 base+adapter 的批次会被多算一个槽而误拒）；
- lora_manager.prepare_lora_batch：weight_indices 默认 0 → -1（"no adapter" 哨兵；
这已经是 CP padding 行在用的约定，见 chunked_backend 的 [PAD-NO-DELTA]），并给
local_active 加 wi >= 0 守卫（-1 会从尾部索引 lora_ranks，把 base-only 批次误判
为活跃）。
契约：uid=None 的请求其 weight_index 恒为 -1，任何消费它的内核必须对负值
early-return（下一条 commit 补齐守卫）。
* fix(lora): 补齐「-1 = no adapter」哨兵的内核/回退守卫
上一条让 base 请求的 weight_indices = -1，这要求每个消费该值的内核在索引
lora_ranks 之前对负索引 early-return（chunked_backend.py 里已写明这条约定）。
GPU 上的 csgmv / absorbed / MoE 路径本来就有守卫（chunked_sgmv_*, kv_b_lora_absorbed,
_compute_moe_lora_info_kernel），但这些还没有，负索引会 OOB 读 lora_ranks 之前的显存：
- chunked_embedding_lora_a.py（csgmv 的 embedding / lm_head 分支）
- embedding_lora_a.py、qkv_lora_b.py、gate_up_lora_b.py、
sgemm_lora_a.py、sgemm_lora_b.py（标准 triton / torch 路径）
另外 _compute_moe_lora_info 的 CPU 回退分支直接拿 weight_indices 去 scatter_，
负索引会被 torch 拒绝或绕到最后一个槽，改为先过滤负值。
守卫与既有 rank == 0 分支同语义（早退、不写 delta），只是把检查提到 lora_ranks
之前。GPU 上未跑这份改动，需 CI / 上游确认。
* fix(lora): fetch_new_loras 也要按「base 不占槽」记账（否则起不来/整机退出）
base 不再注册池槽位之后，fetch_new_loras 里这两处必须跟着改，否则：
1) `new_uids` 会把 base（None）算成「新 uid」，紧接着的
`changed_slots = {uid_to_buffer_id[uid] for uid in new_uids}` 就对 base 取槽位 →
**KeyError**。而启动路径正是 `fetch_new_loras({None})`（初始化 pool 用），
所以引擎会在初始化 LoRA manager 时直接崩（这条只在把 base 从池里拿掉之后才成立，
旧实现里 base 是有槽位的）。
2) `assert len(cur_uids) <= max_loras_per_batch` 把 base 计入 → base + N 个 adapter
的批次在 N 槽配置下必然触发断言，而它在请求路径上，会走 crash-dump 后
`kill_process_tree` 整机退出（AWS B300 对 2026-09-12 实测过这个形态）。
改成只数真 adapter（active_uids）后，N 槽 = N adapter 的语义才自洽，
同时断言带上消息，现场能直接看懂。
pinned 的 `max_loras_per_batch - 1` 余量**保留**（它的第二个理由「避免 unpinned
适配器饿死」仍然成立），只在本 commit 里不动，避免扩大改动面。
* chore(deploy): v4 镜像重建配方（叠加式、本地构建**不推 registry**）+ 逐项自检
背景：分支上有了新修复（PD 隐式重载守卫、base 不占池槽…）之后，上机有两条路 —— 挂
overlay 单文件（少挂一个挂载点会被 Docker 建成目录 → 引擎起不来），或重建整镜像。
这里给第三条、也是最短的一条：在 v3 上**叠加目标 commit 的代码**。
为什么叠加就够（2026-09-17 实测，详见 README §7）：v3 的 python/ 树 == commit 0524363
—— 3544 个文件里 3445 个 sha256 一致，8 个不同（3 个是我们自己的 overlay 文件、5 个是被
构建期改写的 pyproject*.toml），另有 91 个构建期产物不在 git 树里（sglang.egg-info/*、
sglang/_version.py、jit_kernel/csrc/**.cuh …）。COPY 只覆盖同路径文件，不会删掉它们。
新增：
- deploy/b300/glm53/Dockerfile.rebuild：FROM v3 + COPY sglang-src/ + 构建期自检
（import 路径必须落在源码树、compileall、_version.py/.egg-info 仍在、mooncake engine.so
md5 未变、python/ 下 .so 计数 = 0），可选 --with-p2p-ar 编译 #24 的 P2P AR 扩展；
- deploy/b300/glm53/rebuild-image.sh：git archive 目标 ref → docker build → 自检
（镜像里 python/** 的全量 sha256 清单必须与该 commit 逐字节一致；探针逐项对比基础镜像：
.so 数、jit .cuh 数、/opt/p2p_ar 与 flag 一致、mooncake md5）。**全程没有 docker push**，
结尾直接打印 docker save → scp → docker load 的分发命令与回滚方式；
- README §7：流程、自检表、上机与回滚（LMS_ENGINE_IMAGE 换 tag + 两端配对重启）。
实现细节：清单用 find -print0 | xargs -0 sha256sum —— 树里有含空格的路径
（".../block_shape=[128, 128].json"），普通 xargs 会拆坏（干跑时踩到才修）。自测：
源⊂镜像（含多余构建产物）→ 通过；任一文件被旧版覆盖 → 失败。
* Revert "chore(deploy): v4 镜像重建配方（叠加式、本地构建**不推 registry**）+ 逐项自检"
This reverts commit 78d20cb.
---------
Co-authored-by: san-tian <bot@local>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景（实测归因）
B300 decode 节点 60 步 profiler 窗口（bs=23，kernel-only，TP0）的 launch 归因显示：decode step 的 94.8% kernel / 92.9% busy 在
step[TARGET_VERIFY]内，每层 5 次fused_moe_kernel（base 2 次 + MoL LoRA 3 次）：⑤ 的真实工作量 =
M×6144输出 +K=rank=32窄 GEMM（有效流量 ~14 MB，应 ≤2 µs），实测 160.6 µs/层，即 ~85 GB/s ≈ HBM 峰值 1%，且是 153,984 个 CTA（148 SM ≈ 1040 波）⇒ tile/CTA 开销受限。整层 MoL LoRA = 223 µs/层 ≈ 16.7 ms/迭代 ≈ 19.6% 步时，⑤ 一项占 14.1%。根因
virtual_experts.py::_get_stage_config()对这些形状（num_experts = max_loras × n_routed、K = rank）无 autotune 条目 → 走 fallbackBLOCK_SIZE_M=64 / BLOCK_SIZE_N=64；_get_routing()又把BLOCK_SIZE_M交给moe_align_block_size⇒ 每个被触及的虚拟专家桶 pad 成整 64 行：grid = blocks × cdiv(N, BLOCK_SIZE_N)三处 grid 与公式逐一对账（同一窗口）：
153984 = 1604×96（N=6144）、6416 = 1604×4（N=256）、1604 = 1604×1（rank 2r=64）——且 N 比 6144:256 = 24 恰等于 grid 比 153984:6416 = 24 ⇒ 代价 ∝ N-tile 数。修复（统一 flag）
SGLANG_OPT_MOL_LORA_STAGE_CFG（environ.py，默认 False）打开后，MoL 的 shrink/expand stage 只改一项：BLOCK_SIZE_N：grid_m = cdiv(padded_rows, BLOCK_SIZE_M) ≡ blocks（align 按BLOCK_SIZE_M对齐），该值与BLOCK_SIZE_M无关，调小反而把桶切得更碎 ⇒ 刻意不动。BLOCK_SIZE_K（≥ rank，expand 只有 1 次 K 迭代）与 K-loop 不变 ⇒ 每个输出元素仍在同一 tile 内完成全部 K 归约，无 split-K、无跨 tile 累加。n_dim由调用点显式传入（shrink 权重视图[E,K,N]、expand[E,N,K]，不能靠 shape 猜）：shrink 传 rank（≤64 ⇒ 本 flag 下 no-op）、expand 传输出宽（256/6144 ⇒ N-tiles 4× 更少）。预期：⑤ 153,984 → 38,496 CTA ⇒ 160.6 µs → ~40 µs/层 ⇒ 省 ~9-10 ms/迭代 ≈ 11% 步时；②③ 另外省 ~10 µs/层。
验证
py_compile两个文件BLOCK_SIZE_N变化（bit-identical 前提）、两个调用点传的是 stage 真实 Nfused_moe_kernel的args.grid与 ms/迭代grep MOL-LORA-STAGE-CFG（one-shot 打印BLOCK_SIZE_N 64 -> 256）文件
python/sglang/kernels/ops/moe/virtual_experts.py：flag helper +_get_stage_config(n_dim)+ 两个调用点python/sglang/srt/environ.py：SGLANG_OPT_MOL_LORA_STAGE_CFG = EnvBool(False)AGENTS.md（§8 陷阱 + §9 索引）、docs/agent/mol-lora-stage-cfg.md（证据/算术/验收协议）CI States
Latest PR Test (Base): ❌ Run #34950476804
Latest PR Test (Extra): ❌ Run #34950476146