server: fix speculation after an image - #28715
Merged
ggerganov merged 2 commits intoSep 11, 2026
Merged
Conversation
Pass the actual position to the drafter after an image, instead of the token count. Affects every drafter, not just DFlash.
Member
|
This is correct, but we also need to rename |
n_past is used to denote number of tokens and this parameter is meant to be a position
Contributor
Author
|
@ggerganov renamed |
ruixiang63
approved these changes
Sep 10, 2026
ngxson
approved these changes
Sep 11, 2026
ggerganov
approved these changes
Sep 11, 2026
Patt92
pushed a commit
to Patt92/llama.cpp
that referenced
this pull request
Sep 11, 2026
* server: fix speculation after an image Pass the actual position to the drafter after an image, instead of the token count. Affects every drafter, not just DFlash. * rename draft n_past to pos0 n_past is used to denote number of tokens and this parameter is meant to be a position (cherry picked from commit b0dcb81)
roverdrubber
pushed a commit
to roverdrubber/llama.cpp
that referenced
this pull request
Sep 12, 2026
* server: fix speculation after an image Pass the actual position to the drafter after an image, instead of the token count. Affects every drafter, not just DFlash. * rename draft n_past to pos0 n_past is used to denote number of tokens and this parameter is meant to be a position (cherry picked from commit b0dcb81)
6 tasks
1 task
pl752
pushed a commit
to pl752/llama.cpp
that referenced
this pull request
Sep 15, 2026
* server: fix speculation after an image Pass the actual position to the drafter after an image, instead of the token count. Affects every drafter, not just DFlash. * rename draft n_past to pos0 n_past is used to denote number of tokens and this parameter is meant to be a position
liquidspikes
added a commit
to liquidspikes/llama.cpp
that referenced
this pull request
Sep 16, 2026
AvitusA
pushed a commit
to AvitusA/llama.cpp
that referenced
this pull request
Sep 16, 2026
…ft origin ggml-org#28715 switched the draft origin to the target position (pos_next()), which is right for position-mirroring drafters (MTP) but wrong for DFlash/DSpark in this branch: those keep one dense draft row per target token, so after an M-RoPE image the position undershoots the row index and every draft step fails (turns run undrafted at ~33 t/s). Use the token count for them, as the pre-rebase server did, and keep pos_next() for the others. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014XyG2LrZogxN6UW2yy4tfq
quimmedes
pushed a commit
to quimmedes/cafe-llama.cpp
that referenced
this pull request
Sep 16, 2026
* server: fix speculation after an image Pass the actual position to the drafter after an image, instead of the token count. Affects every drafter, not just DFlash. * rename draft n_past to pos0 n_past is used to denote number of tokens and this parameter is meant to be a position
zsogitbe
pushed a commit
to zsogitbe/llama.cpp
that referenced
this pull request
Sep 17, 2026
* server: fix speculation after an image Pass the actual position to the drafter after an image, instead of the token count. Affects every drafter, not just DFlash. * rename draft n_past to pos0 n_past is used to denote number of tokens and this parameter is meant to be a position
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Pass the actual position to the drafter after an image, instead of the token count. Affects every drafter, not just DFlash. Follow up after #28587, a user reported degraded performance after vision input (ROCm). I was able to reproduce similar results locally.
Additional information
RTX 5090, Qwen3.8-27B-UD-Q5_K_S + mmproj-qwen3.8-27b-F16, master
df03399b8:DFlash seems to have a much larger impact but regular MTP also seems to get slightly lower values.
Requirements