Conversation
…ng default
OpenAI-compatible clients that never send chat_template_kwargs (Vercel AI
SDK openai-compatible provider, llama-swap, LiteLLM proxies) currently get
the model's template default, which for reasoning models like Qwen3.6 means
thinking is always on — the token budget goes to reasoning_content and
content comes back empty.
--default-thinking-mode {auto,chat,thinking} lets the operator pick the
server-wide default:
- auto (default): current behavior, template decides
- chat: fills enable_thinking=False into every request that does not set
any explicit thinking control
- thinking: fills enable_thinking=True the same way
An explicit per-request value (enable_thinking / thinking / thinking_mode
in chat_template_kwargs, or the DeepSeek thinking.type wire toggle) always
wins over the server default.
Applied to all three frontends: OpenAI chat completions, Anthropic
messages, and Responses API.
Closes FlashML-org#472
… for server-wide thinking default
gdevenyi
force-pushed
the
5080/13-admission-clamp-118
branch
from
September 19, 2026 21:31
4584957 to
0ab1764
Compare
gdevenyi
force-pushed
the
5080/14-default-thinking-mode-476
branch
from
September 19, 2026 21:31
fd37df5 to
bc13040
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #51. Merges upstream FlashML-org#476 (head
7a15be2) into5080deploy, rebuilt on upstream maincac247a(v0.1.3).Why it is in scope for this deployment: Generic, opt-in: a server-wide default for
enable_thinking; nothing changes unless the flag is passed (the production script does not pass it).Merge: Clean merge.
Testing (RTX 5080 16 GB, production flags: FTW-fp8, nvfp4 KV, 262,144 tokens, 1,280 expert slots):
chat_template_kwargs.enable_thinking=falserequests answer directly, the 250K recall request with reasoning on still returnsreasoning_content.pytest tests/ -m "not slow"at69a5efa(the tip before the middleware commit8adde91, which touches onlyserver/api_server.pyand adds one test;tests/serverrerun at8adde91: 871 passed): 2412 passed, 206 skipped; the 4 failures are environment-only (3xtest_muse_glimmer_vision.pyfail identically on pristine upstreamcac247awith transformers 5.15.1, andtest_async_pull_coalescedis anasync deftest that needspytest-asyncio, which the project does not declare).Rebased 2026-09-19 onto upstream main
cc1f5c2(4 commits pastcac247a: FlashML-org#471 greedy sampling in mixed batches, FlashML-org#518 WeightLoadError, FlashML-org#521 tvm-ffi jit arch, FlashML-org#524 install index). Same replay as before viagit rerere; the replayed stack differs from the previous tip8adde91by exactly thecac247a..cc1f5c2file set. This PR's head is nowbc13040.🤖 Generated with Claude Code
https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt