Radv rembrandt vk - #4
Open
Jackson57279 wants to merge 85 commits into
Open
Jackson57279 wants to merge 85 commits into
Jackson57279 wants to merge 85 commits into
Conversation
Three sub-IQ1_S data types: IQ1_S's algorithm with the codebook the mapping step may choose from cut to a subset of iq1s_grid, and the index stored in fewer bits. IQ1_XS id 64 46 B / 256 1024-entry grid 10-bit index 1.4375 bpw IQ1_XXS id 65 42 B / 256 512-entry grid 9-bit index 1.3125 bpw IQ1_XXXS id 66 38 B / 256 256-entry grid 8-bit index 1.1875 bpw Dequantization is IQ1_S's rule throughout, y[j] = d*(2*ls + 1)*(grid[idx][j] + delta). GGML_TYPE_COUNT is 67; 42..63 are left for upstream, which already uses 42 for Q2_0, so a stock build rejects these files rather than misreading them. CPU, CUDA dequant, MMVQ and MMQ. The MMQ path targets the current ggml_cuda_mmq_util_funcs / ggml_cuda_mmq_config API and registers the types in every per-architecture config table; Blackwell falls through to Ampere and needs none. scripts/iqxs_gen_tables.py regenerates all six codebook tables in ggml-common.h byte-identically from the three committed .npy grids, so the tables are reproducible rather than pasted. Tested on B200, CUDA 13, sm_100. test-backend-ops -o MUL_MAT passes 2/2 backends both default and with GGML_CUDA_FORCE_MMQ=1, 11 cases per type matching IQ1_S. Note the default run never enters the new MMQ kernels, since small n routes to MMVQ.
Decode on Rembrandt is mul_mat_vec. The generic AMD path left rm_stdq at 1. Co-authored-by: Cursor <cursoragent@cursor.com>
Omarchy llama-bench on the frozen F32 0.6B stayed at 27.97 gen tok/s, inside noise. Co-authored-by: Cursor <cursoragent@cursor.com>
The F32 0.6B GGUF never hit quantized mul_mat_vec, so the first try could not move gen tok/s. Co-authored-by: Cursor <cursoragent@cursor.com>
Omarchy Q4_K_M decode was faster with GGML_VK_DISABLE_MMVQ (143.26 vs 141.54 tok/s). Prefill still uses integer-dot MMQ. Co-authored-by: Cursor <cursoragent@cursor.com>
Omarchy decode spends about 1.87ms per token on mul_mat_vec q6_K m=151936. AMD always picked the subgroup workgroup. Co-authored-by: Cursor <cursoragent@cursor.com>
This reverts commit 4b271a5.
Vocab Q6_K GEMV dominates decode. A larger DMMV workgroup made it worse, so try more rows instead. Co-authored-by: Cursor <cursoragent@cursor.com>
This reverts commit abd18f5.
Prefill is integer-dot MMQ. Decode GEMV is already at the best row counts we have kept. Co-authored-by: Cursor <cursoragent@cursor.com>
This reverts commit 4793618.
Vocab decode was paying a workgroup barrier per output row. Co-authored-by: Cursor <cursoragent@cursor.com>
This reverts commit 74b9b1c.
Vocab GEMV is wide and short; four-row tiles under-occupancy that dispatch. Co-authored-by: Cursor <cursoragent@cursor.com>
The earlier revert used F32 harness numbers; vocab GEMV is Q6_K. Co-authored-by: Cursor <cursoragent@cursor.com>
This reverts commit 2db8c52.
Scalar fma loops on the vocab matvec leave RADV packing work on the ALU. Co-authored-by: Cursor <cursoragent@cursor.com>
FFN decode GEMVs are the remaining time after vocab Q6_K. Co-authored-by: Cursor <cursoragent@cursor.com>
This reverts commit f6fe0ef.
Rembrandt advertises mixed-float packed dots; vocab matvec was scalar fma. Co-authored-by: Cursor <cursoragent@cursor.com>
This reverts commit 2575fbb.
FFN GEMVs at m=3072 dominate remaining decode time after vocab. Co-authored-by: Cursor <cursoragent@cursor.com>
This reverts commit 87d0082.
This reverts commit 65e0c74.
Match the partials kernel so decode rows of 1024 do not launch 512 threads. Co-authored-by: Cursor <cursoragent@cursor.com>
This reverts commit 2d8e746.
Keep 512-wide workgroups for single-row decode and take the prefill win. Co-authored-by: Cursor <cursoragent@cursor.com>
Id 0 is already norepeat in the shared binary header. Co-authored-by: Cursor <cursoragent@cursor.com>
Let independent decode shaders overlap without a fake copy hazard. Co-authored-by: Cursor <cursoragent@cursor.com>
Skip the per-superblock scale barrier on the vocab GEMV. Co-authored-by: Cursor <cursoragent@cursor.com>
This reverts commit a1e4a17.
Prefill RMS already likes smaller groups; try the same on GLU. Co-authored-by: Cursor <cursoragent@cursor.com>
Match the GLU occupancy that raised prefill on Rembrandt. Co-authored-by: Cursor <cursoragent@cursor.com>
This reverts commit 1010578.
Keep the 512-wide dispatch tile that matches get_idx. Co-authored-by: Cursor <cursoragent@cursor.com>
This reverts commit c12cf3b.
Leave vocab Q6_K on the small subgroup size that already won. Co-authored-by: Cursor <cursoragent@cursor.com>
This reverts commit ed76f0a.
Raise occupancy on the vocab GEMV instead of pairing rows. Co-authored-by: Cursor <cursoragent@cursor.com>
This reverts commit 07d5aba.
128-wide SwiGLU left most FFN outputs uninitialized. Co-authored-by: Cursor <cursoragent@cursor.com>
Keep get_idx and workgroup counts on the same 128x128xZ packing. Co-authored-by: Cursor <cursoragent@cursor.com>
This reverts commit bb79661.
Vocab Q6 stays at 2; FFN 3072-wide GEMVs dominate after lm_head. Co-authored-by: Cursor <cursoragent@cursor.com>
This reverts commit d27b53a.
Keep vocab and attention on the 32-thread subgroup size. Co-authored-by: Cursor <cursoragent@cursor.com>
This reverts commit e3fb007.
This reverts commit ffe323f.
This reverts commit 42eb3a1.
This reverts commit 3832d0d.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Additional information
Requirements