ggml adapter: SM120/Qwen3.6-35B target — binding-driven LLM decode windows, MoE catalog family, packless artifacts - #186
Open
LiangSu8899 wants to merge 37 commits into
Open
ggml adapter: SM120/Qwen3.6-35B target — binding-driven LLM decode windows, MoE catalog family, packless artifacts#186LiangSu8899 wants to merge 37 commits into
LiangSu8899 wants to merge 37 commits into
Conversation
Ports the two Pi0.5 artifacts that had not reached this repository. The guide is the canonical usage and performance reference: what runs on the device, how to instantiate and calibrate each precision tier, the latency and accuracy matrix across FP8 / NVFP4 / INT4 / INT4+RHT at one, two and three views, tier selection, the knob reference, the frame-time breakdown, and the approaches that were measured and rejected. The harness measures every tier against a common FP16 reference in one locked-clock batch, reporting latency and accuracy together, which the strict FP8-referenced suite cannot do — it leaves FP8's own error unmeasured and has no unquantized anchor.
The compact GeGLU store node writes the down-projection input itself, but the collective still performs its ordinary D store into a buffer nothing reads. At encoder shape that is 12.8 MB per call, 21.8% of the kernel's traffic. Aliasing every output row onto row zero keeps the store instructions but collapses their footprint to a single row, so the write-back leaves L2 instead of DRAM. Encoder FFN GEMM 369.9 -> 356.5 us per call at the production shape (781x32768x2048), about 0.23 ms per three-view frame. The decoder shape is unaffected, as expected: at ten tokens the dummy is 40 KB. Fusion kernel contract tests pass unchanged, since no code reads the buffer. Eliminating the store outright is worth roughly 1.2 ms more but needs the epilogue's D descriptor remapped to half width.
The half-width epilogue's compact store node writes the real output, yet the collective still stages the unread D tile through smem and TMA-stores it. Fork the Sm100TmaWarpSpecialized collective (CollectiveEpilogueNoD) with the D store elided behind the same is_destination_supported pattern Sm90 and the Sm100 ptr-array epilogues already use. Encoder combiner 'epilogue_hw_nod' is the new default: a five-leg alternating sandwich measures 32.513/32.550/32.547 (hw) vs 31.956/31.967 (nod) at 3 views, -0.57 ms with <=0.037 ms drift, outputs bit-identical. Formal 3v/2v agree and pass all gates. Two measurement caveats are recorded in the guide: the isolated kernel benchmark scores the elision as a regression, and under nsys node tracing the variants converge, so the win is only observable unprofiled. The decoder-tile variant is a wash (its dummy store is 0.04 MB) and ships opt-in as --decoder-fused-geglu-nod.
…oundaries Up fuses bias + tanh-GELU with FP4 packed + SFD output; Down fuses bias + fp32 residual add with fp32 output (AlignmentCD=4), plus a beta=0 wrapper for plain bias GEMM. The CUTLASS workspace is cached per shape so steady-state calls are graph-capture safe. New symbols only; the existing fp16-boundary siglip_ffn kernels are untouched.
Relocates the ggml-cuda FlashRT layer here as the third host adapter: pure-CUDA kernels (repack, fused norm/modulation/rope, activation quantize, NVFP4 GEMM wrapper) behind a C header, plus the ggml-facing dispatch half (subgraph window matchers and caches) compiled inside the host's build tree. Fused-epilogue GEMMs are consumed from csrc/gemm/fp4.
…ml pi0.5 host Adds the jetson_pi_edge_pi05 pipeline binding (vla_tick_pipeline, complete hot-path coverage, 11 segments mapping the host's fused windows onto catalog structures) and a qualification runner: manifest validation and structure-version pins offline, plus an opt-in on-device gate that holds the steady-state action chunk to a stored golden (exact match; the adapter is bitwise deterministic after warmup). Verified all gates green on the live host and red under a deliberate pin bump and a kernel-set change.
…late kernel
{RMS_NORM, MUL(w), ADD(mul, norm)} == rms_norm(x)*(1+w) runs as one kernel
via ada_rms_mod with scale = w and shift = 0 (cached zero vector). ggml's
own fused rms_norm cannot express this form because the add operand is the
norm output itself. Opt-out via GGML_FLASHRT_NO_RMS_GEMMA.
Fuses the prefix layers' q/k/v projections, RoPE, scale, KV padding to the FA KQ stride, and the permuted f16 materializations into the fused QKV GEMM plus qkv_post. qkv_post_full additionally re-emits the rope'd K and plain V as f32 rows so graph-tail persistent-KV stores keep reading their tensors; pad rows are zeroed to preserve the graph's PAD semantics. Bitwise-identical action output; opt-out via GGML_FLASHRT_NO_QKV_PREFILL.
…16 K/V directly gemm_bias_f16out converts once from the fp32 accumulator in the epilogue, bit-equal to an fp32-out GEMM followed by an fp16 cast. The ggml adapter's vision QKV window uses it for the K/V legs when their cast tensors are handed in, eliminating the two full-tensor cast copies per layer.
FLASH_ATTN_EXT with q_tokens <= 16 over a single padded f16 KV head runs as a batched QK^T GEMM (GQA heads share the stride-0 K operand), a masked fp32-accurate softmax, and a batched PV GEMM whose fp32 result lands directly in the node's [hd, n_head, n_tok] layout via a strided C — the same decomposition the torch pipeline uses at these shapes, where it beats the stream-k flash kernel plus its fixup pass. Opt-out via GGML_FLASHRT_NO_DEC_ATTN.
The rms_gemma and decomposed decode attention windows change the action output at the ulp-amplification level (both judged against the FP16 reference on gripper-active real frames); the golden follows the verified current state. All gates green.
The GQA query heads share the one KV head, so the per-head batched QK^T and PV GEMMs collapse into single GEMMs over all n_head*n_tok query rows. Ordering the rows t-major makes the PV output land contiguously in the flash-attention node's [hd, n_head, n_tok] destination, removing the strided-C store penalty. Numerically identical to the batched form.
…tention qkv_post optionally stores the rope'd+scaled Q rows as f16 alongside the f32 tensor; its [M, Nq] layout is exactly the t-major gather order the decomposed decode attention consumes. A one-shot grow-only slot hands the buffer to the following attention window, which then skips its gather kernel (one launch less per layer). The f16 values round from the same fp32 registers the gather would read, so the results are bit-identical.
The host now offloads its action input projection, which moves the final actions at the usual chaotic-amplification scale; distances to the f16 reference were re-judged before this refresh.
The prefill graph ends with one tiny f32->f16 row copy per layer and KV tensor into the persistent encoder-KV buffers. Batch a run of them into a single launch; every copy is still performed with identical rounding, so results are bit-exact.
Export the vendored FA4 SM100-compatible forward (CuTe-DSL) ahead of time for sm_110a at the padded head_dim-80 shape and run the vision FLASH_ATTN_EXT through it. The padded Q/K/V/dst buffers all share one dense (B, S, H, D) linear layout, so the f32 boundaries reduce to dense elementwise converts and the f16 K/V pass straight through. A small shim file provides the _cuda* runtime aliases the AOT object expects, keeping the build free of any CuTe-DSL runtime dependency; the module loads once outside CUDA graph capture and the wrapper is capture-safe. Includes the regeneration script and provenance notes. Refreshes the e2e action golden: the vision attention numerics move, and the distance to the f16 reference improves.
… padding The vision FA node is followed by a head-de-pad view and a CONT whose strided copy costs more than the attention itself. Absorb the pair: the FA4 f16 output converts directly into the CONT's packed f32 destination, skipping the padded f32 store and the strided copy. The same elements go through the same f16->f32 conversion, so results are bit-identical.
Second AOT export of the vendored FA4 forward at the prefill shape (head_dim 256, GQA with one KV head, full attention). The prefill is a prefix-LM: its mask is row-uniform pad-only and the real KV length equals the query count, so passing that length as the KV dynamic shape reproduces the mask exactly and the padded tail is never read. The f32 graph boundaries again reduce to dense elementwise converts. Refreshes the e2e action golden. On the real-frame judge the overall distance to the f16 reference moves -0.0085 (within the swing this metric shows for any attention change; the gripper dimension improves +0.024); GGML_FLASHRT_NO_PREFILL_FA4=1 falls back per site.
Documents building a host against the adapter, the runtime switch matrix, the three validation layers (operator tests, qualification gates, benchmark/parity methodology), the layer architecture with its capture-safety and fusion-window invariants, and the AOT FA4 module mechanics. Adds measured Thor numbers to the overview.
The ggml adapter's build resolves CUTLASS from third_party/cutlass by default; pin it (NVIDIA/cutlass v4.4.2) as a submodule so a recursive clone is self-contained instead of relying on a local checkout.
Second (arch, model-family) target for the native ggml adapter: fused-region NVFP4 W4A4 GEMVs (GDN in_proj, attn qkv, lm head), GDN cell span with M<=4 per-token state snapshots and checkpoint replay, format-native MoE expert span over ggml K-quant blocks (bit-exact q8_1 activation clone), out-proj / router / spec-draft-head windows. All launches join the host's PDL chain; verify batches up to M=4 are first-class (MTP speculative decode). Host integration is a CMake option (GGML_CUDA_FLASHRT_SM120 + GGML_CUDA_FLASHRT_PUBLIC_DIR) compiling this translation unit inside the host's ggml-cuda build, with all call sites ifdef-guarded. Binding and qualification gates recorded for the migration baseline.
Region wire buffers (packed e2m1 + atom-layout SF + per-tensor alpha) are now built on first sight of the weight tensors in an evaluated graph, from a pre-capture hook: ggml dequant -> bf16 -> global amax -> ue4m3-ceil block scales -> e2m1 nibbles, reproducing the offline packer byte-for-byte (FRT_REPACK_CHECK memcmp gate, 40/40 regions identical; perplexity receipt unchanged). This removes the side-band region pack dependency. The lm-head keeps its pack: it is quantized from the BF16 checkpoint, which both scores and drafts measurably better than a rebuild from the GGUF's Q6_K copy; the online head build remains as a fallback.
- new family moe_expert_ffn v1: routed expert GLU-FFN with exact in-boundary top-k routing, optional sigmoid-gated shared expert, format-native weight consumption (host K-quant blocks in place, bit-exact activation-quant clone) and first-class token batches; reference implementation replicates the Qwen3.x softmax/top-k/clamp/renorm convention including tie-break. - gated_delta_core v2: snapshot_per_token state-update and replay_in_region checkpoint variants — the rollback discipline speculative verify batches require of stateful regions, with the failure modes documented. - autoregressive_decode_pipeline v3: optional spec_draft stage plus the speculative conformance set (output distribution matches target, explicit draft-state rollback, throughput judged on text). - rewrite the sm120 binding/pins against these families (previous ones were placeholders); qkv_pack/linear_proj variant additions are recorded as proposals instead of version bumps so the pi0.5 binding's pins stay green. Registry loads all 18 families; the existing pi0.5 pins verify green against the updated catalog.
Add the catalog charter (catalog/README.md): a structure entry is a local boundary expression — dimensions, contracts, variant semantics, executable reference — serving context alignment against native pipelines and the torch-side distribution boundary. It adjudicates nothing: performance numbers, verdicts and case histories are conditional on model/hardware/ host/driver and expire silently, so the only arbiter is a test run against the live system; dated results live in campaign records and per-binding qualification gates that are re-established by re-running them. Scrub the recent entries accordingly: drop campaign evidence and throughput numbers from moe_expert_ffn and gated_delta_core, reduce variant comments to pure semantics, drop the judgment-phrased speculative conformance line from autoregressive_decode_pipeline, keep only the parity-vs-reference gate on the new family.
…oldered) The binding yaml becomes the single source for every model-specific constant in the window translation unit: qkv_pack member tables (names, rows, offsets, leaders), region shapes and layer counts, gated-delta cell dims (conv row, state size), out-proj/head names and dims, layer scan bounds. tools/gen_binding_header.py generates a checked-in constants header from the binding's host_params section; the window logic itself is now family-level — a same-family model is a new binding file plus a header regeneration, with no window-code edit. Framework-wide ggml naming conventions (ffn_gate_exps, ffn_moe_out, ...) stay in the window code deliberately: they are host facts shared by every model on this host, not model facts. Also brings the binding up to the strict loader contract (token_select stage, complete_hot_path coverage with stock-served segments declared as host stages); it now validates through flash_rt.structures.binding with pipeline coverage required. Receipts: online repack byte-identity 40/40 vs the offline packs; tg128 and 24-chunk perplexity unchanged; speculative smoke coherent.
Move the sm120 heavy math out of the ggml window file into csrc as the next generation of the warp-split-K family (additive; the earlier bf16out/runtime-M entry is unchanged): compile-time row-count specialization (a runtime M in the MMA hot loop costs measurable time even at M=1), f32 output, and an optional programmatic-dependent-launch join so hosts that overlap every launch keep their chain. The M-rows activation quantizer ships alongside with its device body exposed for fused producers. The ggml adapter now only translates: it includes the csrc device header for its fused GDN producers and calls the csrc entries for standalone quantize/GEMV launches. Receipts unchanged: repack byte-identity 40/40, duplicated-token selftest bit-exact, tg128 and 24-chunk perplexity flat.
Compiled-in windows now run by default: GGML_CUDA_FLASHRT_DISABLE=1 turns the whole layer off, GGML_FLASHRT_NO_<X>=1 disables one window, and the historic FRT_<X>_SWAP switches remain as explicit A/B overrides. The in-process repack and the spec-draft head serving are part of the default set (the draft head builds lazily, only once a loaded draft model's head copy is actually seen, so plain runs spend nothing on it); the full-tier FP4 lm-head keeps its measured quality increment and stays opt-in, as do the archive windows. The draft model's own qkv projections match the kind-1 pack shapes and would be silently swept in; that is a separate acceptance-only judgment, gated off by default (FRT_DRAFT_REGIONS=1), measured flat here. Receipts: same-binary env vs zero-env tg128 identical; zero-env speculative server median matches the env-driven safe tier; zero-env 24-chunk perplexity bit-identical; whole-layer disable reproduces stock.
Offline gates: binding manifest validation, structure-version pins, and binding-header freshness (the generated constants header must match the binding yaml). On-device gates re-establish their numbers by running: duplicated-token bit-exact selftest, online-repack byte-identity against an offline reference pack, 24-chunk perplexity (pinned bit value), and tg128 against the tier's pinned baseline with a tolerance band. First full run: six greens (selftest PASS, repack 40/40 byte-identical, ppl 6.1106 exact, tg128 within band on the full tier).
…e state discipline)
Add the FlashRT-edition GGUF flow: an NVFP4 lm-head quantized from the BF16 checkpoint is spliced into the shipping quantized body (tools/splice_nvfp4_head.py). The stock nvfp4 kernels then serve the head at pack-tier speed by themselves, the adapter's spec-draft head serving wires up from the same tensor by pure shuffle (scale passthrough, alpha = 1), and the last side-band pack file is gone: the artifact judges faster-tier speed with better perplexity than the pack-served full tier on the same body. Judged along the way, gates recorded in the pins file: serving the NVFP4 head through the fp4-activation GEMV adds no speed over stock's q8_1-activation mmvq and costs measurable perplexity (archive switch, opt-in); a whole-model standard requantization loses to the shipping body both at the floor and in FP4-region cost (region quality follows the source tensor's quantization); the draft model's shape-colliding projections stay gated.
# Conflicts: # .gitignore
The native ggml host adapter is consumed at build time by a llama.cpp tree and takes its heavy math from csrc/ by include; it has no runtime dependency on the Python structures layer. It now lives beside the other native surfaces (exec/, runtime/, cpp/) instead of under flash_rt/structures/adapters/, ahead of the structures layer moving to its own repository. Path-only change: the csrc includes, the repo-root resolution in the qualification and tooling scripts, and the binding comments follow the move. Offline qualification gates (binding manifest, structure pins, binding-header freshness) pass unchanged.
v3 added the optional spec_draft stage; the shared-schedule-family test still pinned version 2 and the v2 optional-stage set.
LiangSu8899
changed the base branch from
feat/structures-ggml-adapter
to
main
September 11, 2026 18:22
…lash_rt/catalog The binding for the ggml pi0.5 host, the Qwen3.6 SM120 binding, the catalog charter and the moe_expert_ffn family land in flash_rt/catalog/; the adapter's qualification tooling reads the catalog from there.
The native ggml host adapter now lives in https://github.com/flashrt-project/FlashRT-llama.cpp, the repository a llama.cpp host mounts at ggml/src/ggml-cuda/flashrt. It vendors the FlashRT kernel sources it needs one-way (pinned commit, hashes) and carries CUTLASS itself. This repository keeps the kernels (csrc) and the structure catalog and bindings the adapter's qualification gates read through the flash-rt distribution.
The submodule was added so the ggml adapter could be built from this checkout. The adapter now lives in FlashRT-llama.cpp and carries its own CUTLASS; this repository keeps its original arrangement (a checkout dropped under third_party/, ignored by git).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second (arch, model-family) target for the native ggml adapter, stacked on #185: an LLM decode window set for Qwen3.6-35B-A3B (hybrid GDN + attention, 256-expert MoE, MTP speculative decode) on RTX 5090 (SM120), plus the catalog and tooling that make the next same-family model a data change instead of a code change.
What's in here
Windows (
fr_win_qwen36_sm120.cu) — fused-region NVFP4 W4A4 GEMV packs (GDN in_proj, attention qkv), the GDN cell span with per-token state snapshots and checkpoint replay for speculative verify batches (M ≤ 4), a format-native MoE expert span that consumes GGUF K-quant blocks in place through ggml's own vec_dot device functions (bit-exact q8_1 activation clone, shared expert folded), out-proj / router / lm-head / spec-draft-head windows. All launches join the host's programmatic-dependent-launch chain.Zero-configuration defaults — compiled-in windows run by default;
GGML_CUDA_FLASHRT_DISABLE=1reproduces stock,GGML_FLASHRT_NO_<X>=1disables one window, historicFRT_<X>_SWAPswitches remain as A/B overrides. Weights repack in-process on first sight of the tensors (pre-capture hook), byte-identical to the offline packer, so a stock GGUF runs as-is with no side files.Binding-driven windows — every model-specific constant (pack member tables, region shapes, layer counts, cell dims, head facts) lives in the binding's
host_paramsand reaches the code through a generated constants header (tools/gen_binding_header.py). The window logic is family-level; a same-family model is a new binding plus a header regeneration.Catalog — new
moe_expert_ffnfamily (routed expert GLU-FFN with exact in-boundary top-k, sigmoid-gated shared expert, format-native consumption, token batches as part of the boundary, executable reference);gated_delta_corev2 adds thesnapshot_per_token/replay_in_regionstate semantics speculative rollback requires;autoregressive_decode_pipelinev3 adds the optionalspec_draftstage and speculative conformance. A catalog charter (catalog/README.md) fixes the layer's role: structures express boundaries and never adjudicate — no performance claims or case histories in catalog entries. The pi0.5 binding's pins stay green against all of this.csrc — the warp-split-K GEMV family gains a next-generation entry (
fp4_w4a4_mma_warpsplit_mrows_f32out_sm120): compile-time row-count specialization (a runtime M in the MMA hot loop costs measurable time even at M = 1), f32 output, optional PDL join; an M-rows f32→NVFP4 activation quantizer ships alongside with its device body exposed for fused producers. Existing entries untouched.Qualification — one-command runner (
run_qualification_qwen36_sm120.py): binding manifest, structure pins, binding-header freshness, duplicated-token bit-exact selftest, online-repack byte-identity, pinned perplexity and throughput gates. Six greens on the recorded baseline.Artifact tooling —
tools/splice_nvfp4_head.pybuilds a packless speed-tier GGUF by splicing an NVFP4 lm-head (quantized from the BF16 checkpoint) into the shipping body; the stock nvfp4 kernels then serve the head at swap-tier speed with better perplexity than the side-band pack it replaces.Receipts (RTX 5090, 24-chunk
-ub 1perplexity, tg128 r=5, greedy server medians with text verification)-mddraft +-bs)Negative results are recorded in the pins file rather than the catalog, per the charter.
Notes for the Thor line
The window/matcher code, the binding-driven constants flow, the qualification form and the M ≤ 4 / snapshot / checkpoint disciplines are the reusable parts; the MoE and GDN spans are arch-portable CUDA. The SM120 MMA GEMV needs an SM110 counterpart (or the existing csrc GEMMs) and every judgment re-establishes on Thor's own roofline — in particular the launch-overlap capability differs, so the launch-count verdicts do not transfer.