fix(server): honor configured output default across APIs - #411
Merged
jason-fxz merged 1 commit intoSep 12, 2026
Merged
Conversation
Collaborator
|
Tested end-to-end on top of current main: with |
Artemowka22
added a commit
to Artemowka22/FreeToken
that referenced
this pull request
Sep 15, 2026
One conflict, in python/freetoken/server/openai_api.py: upstream made the default output budget configurable (default_max_tokens, FlashML-org#411) on the same lines where this branch wraps the non-streaming completion drain for disconnect delivery. Kept the drain wrapper; its sampling resolution now passes default_max_tokens through, as upstream does elsewhere. Assisted-by: Claude
Artemowka22
added a commit
to Artemowka22/FreeToken
that referenced
this pull request
Sep 15, 2026
One conflict, in python/freetoken/server/generation.py: upstream made the default output budget configurable (default_max_tokens, FlashML-org#411) in the same SamplingParams construction this branch rewrote to return the validated values. Kept the validated resolved_* fields; the max_tokens fallback now uses the configurable default_max_tokens instead of the fixed constant. Assisted-by: Claude
Artemowka22
added a commit
to Artemowka22/FreeToken
that referenced
this pull request
Sep 15, 2026
Two conflicts: - python/freetoken/server/openai_api.py: upstream threads the configurable default output budget (default_max_tokens, FlashML-org#411) into the resolve_sampling call that this branch had extracted into a variable to attach the logprobs fields. Kept the variable; its construction now passes default_max_tokens. - python/freetoken/tokenizer/server.py: import conflict between upstream's get_mm_processor (image input, FlashML-org#454) and this branch's build_logprobs_entry. Kept both. Assisted-by: Claude
nomanoma121
pushed a commit
to nomanoma121/My-FreeToken
that referenced
this pull request
Sep 16, 2026
nomanoma121
pushed a commit
to nomanoma121/My-FreeToken
that referenced
this pull request
Sep 16, 2026
…r's shape FlashML-org#438 folds qwen3_5_moe's four dense readers into one _DenseReader that asks the checkpoint's QuantConfig what each Linear stores, and deletes _iter_weights_attn_fp8 -- which is where --spec-mtp's weight reading lived. The head is read again on the new shape: _rename keeps mtp.* when the engine asks for it, the head's two pre_fc norms and its own final norm join the (1+w) list, and its per-expert bf16 experts are gathered on the host and yielded last as the two stacked tensors the engine quantizes into a bank layer. Two things the old path needed code for are now free. mtp* is off every quantizer's list, so scheme_for_name returns None for the head and it reads as stored -- the table that said "the head's q|k|v are bf16 where the decoder's are fp8" is gone. And the up-front refusal ("MIXED_PRECISION only") went with attn_quant, which FlashML-org#438 stops parsing for this family: the reader is layout-general now, so what is left to refuse is a head that is not in the checkpoint (the stacker's assert) or one the exporter quantized (a loud NotImplementedError -- the engine quantizes these itself, so a pre-quantized head would be stacked into the wrong format). FlashML-org#428 reads qwen4_exp's dense projections through the same QuantConfig. Its _DenseFuser replaces the fork's fuse_buf; the progress bar keeps asking the pipeline engine's rank rather than TP's. FlashML-org#367 reserves paged KV at allocation granularity -- the same page-span accounting this fork already does for the SWA pool, on the other currency. Admission is stricter than the token math by up to one page per request, which is the point of it. FlashML-org#411 honors the server's max_output_tokens in both APIs' defaults; it lands beside the image parts in chat_request_to_genspec without touching them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
MT-z
added a commit
to MT-z/FreeToken
that referenced
this pull request
Sep 16, 2026
…image input on the Qwen VL families Takes the vision implementation this branch removed in the previous commit, plus the two of its own commits that survive it: the checkpoint-role fix Ornith needs to load, and the bounded uvicorn stop. `/img` comes back with the merge. Nine files conflicted, all of them this branch's own non-vision work meeting FlashML-org#454's: - `stats.py`: kept `effective_context_length` and the three-argument `derive_model_card`; FlashML-org#454's `input_modalities` had already merged into the body. Losing the first would re-advertise the rope ceiling over a smaller KV pool, which is what made Claude Code's first turn 400. - `openai_api.py`, `generation.py`: kept this branch's richer forms (n > 1, echo, FIM suffix, the validated `resolved_*` sampling, presence penalty) and threaded FlashML-org#411's `default_max_tokens` through both API paths -- `/v1/completions` reached `_resolve_sampling` without it, which would have ignored `--max-output-tokens` on exactly the surface FlashML-org#411 exists to fix. - `args.py`: both sides' fields, including FlashML-org#454's media allowlists next to `--sampling-override`, `--template-kwarg`, `--pidfile` and `--no-system-in-place`. - `kvcache/`: `kv_quant` and `mrope` are independent additions to the same signature. - `api_server.py`: kept the `with stack:` teardown around uvicorn.run. - `prefill.py`: took FlashML-org#454's `mm_items`/`mrope_*` admission and dropped the `image_token_id` field the removal left without a reader. The duplicate-implementation check that motivated the removal commit: the four files that carried both towers after a trial merge -- `qwen3_5_moe/model.py`, its `weight.py`, `scheduler/scheduler.py`, `tokenizer/tokenize.py` -- now carry only FlashML-org#454's. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
MT-z
added a commit
to MT-z/FreeToken
that referenced
this pull request
Sep 16, 2026
… tests still drive it FlashML-org#454 reads five config attributes directly. Every one of them exists on a real EngineConfig/ServerArgs, so FlashML-org#454's own suite never notices -- but this branch drives the same code from tests that build the config as a SimpleNamespace, and those stubs predate the fields. Five failures, all AttributeError, none of them about multimodal: kvcache/__init__.py model_config.model_is_mrope test_qsa_pool_fp8 (factory) kvcache/qsa_pool.py same, inside kv_cost test_qsa_pool_fp8 (kv_cost) server/stats.py config.served_modalities test_effective_context_length server/generation.py state.config.mm.max_pixels test_logprobs_api, test_openai_extras server/generation.py the second call site This is the shape FlashML-org#300 flagged for the scheduler hooks and that this branch already carries a fix for: a hook reached from a stub must not require the whole config object. derive_model_card is additionally a metadata route, which must never raise. Only these five. The direct reads in engine.py (nine) and mm/media.py (three) are on paths that see a real config and nothing else -- turning those defensive would hide a genuinely missing field rather than tolerate a deliberately partial stub. Also threads FlashML-org#411's default_max_tokens from _resolve_sampling into resolve_sampling. The merge kept this branch's richer _resolve_sampling body, which did not carry the argument, so /v1/chat/completions and /v1/completions both fell back to the 32k built-in and ignored --max-output-tokens. FlashML-org#411's own test caught it. Full suite against try/all before the vision swap: 8 failed / 2004 passed vs 8 failed / 1996 passed, same failure set -- the three fp8 ones are this branch's standing failures from the FlashML-org#354 take, not new. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
trcwebdesign
pushed a commit
to trcwebdesign/FreeToken
that referenced
this pull request
Sep 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #395.
Threads the server-level
--max-output-tokensfallback through the Chat Completions and Completions adapters, as the Responses adapter already does. Explicit request limits still take precedence; absent server configuration retains the 32k fallback.Validation:
python3 -m py_compilefor changed modulesgit diff --check