Repository navigation
fix(engine): preserve greedy sampling in mixed batches - #471
Merged
jason-fxz merged 1 commit intoSep 19, 2026
Merged
Conversation
taking-lying-flat
force-pushed
the
fix/greedy-mixed-batch
branch
from
September 14, 2026 16:57
d27f090 to
2fffff5
Compare
Collaborator
|
LGTM, merging. |
This was referenced Sep 19, 2026
trcwebdesign
pushed a commit
to trcwebdesign/FreeToken
that referenced
this pull request
Sep 21, 2026
danielamadori
pushed a commit
to danielamadori/DFlash-FreeToken
that referenced
this pull request
Sep 22, 2026
lucaspirola
pushed a commit
to lucaspirola/FreeToken
that referenced
this pull request
Sep 28, 2026
KarrAcaRn
pushed a commit
to KarrAcaRn/FreeToken-ByAI
that referenced
this pull request
Oct 3, 2026
…gprobs for chat and legacy completions Conflicts, all kept on both sides: - sampler: next's penalties (FlashML-org#527) and greedy mask (FlashML-org#471) next to the logprob rows; prepare() passes both, sample() keeps the greedy-mask path and the PR's compute_logprobs reads the raw logits. - SamplingParams / ChatCompletionRequest: penalty fields stay as on next (None = model default); the logprobs fields are added. - openai_api: next's default thinking mode and _resolve_sampling feed the logprobs fields; the legacy completions drain keeps FlashML-org#222's disconnect watch and collects the logprob entries. - engine/scheduler/detokenizer/generation: next's MoE stats (FlashML-org#231) and metrics timings (FlashML-org#504) beside the logprob plumbing. Assisted-by: Claude Opus 5.5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A greedy request with fixed logits
[1.0, 1.0, 0.0]returns token 0 alone, but can return token 1 when batched with a random request: the mixed path approximates greedy with temperature1e-6, leaving the tied maxima equally likely. Also,temperature=0, top_p=0.9unnecessarily enters probability sampling.Keep a per-request
greedy_maskand explicitly selectargmaxfor greedy rows. Treattemperature <= 0ortop_k == 1as greedy regardless oftop_p, and use neutral backend parameters for those rows. The all-greedy fast path is preserved. Mixed batches still run probability sampling across all rows; skipping that computation is outside this correctness fix.Related: #307. This fixes sampling-policy differences for identical logits; it does not establish or resolve the cause of the serial model-level divergence reported there.
Validation:
[1, 2, 2]instead of[0, 2, 1], and zero-temperature requests withtop_p < 1entered probability sampling. The validation file is retained locally and is not included in this PR.e0886ccplus this patch: 73 passed, 4 skipped, including real Triton CUDA sampling. The four FlashInfer cases were skipped because FlashInfer is not installed.git diff --checkpassed.Exact local validation command, from the repository root, using the untracked test file: