fix(models/moondream2): resolve EOS/prompt so real checkpoint generates - #609
Merged
Conversation
Follow-up hardening for #522: the merged Moondream2 port loads its real checkpoint but emits zero tokens because it inherited Moondream3's bos_id = eos_id = 0 convention. Moondream3 uses id 0 with its starmie tokenizer, but Moondream2 ships a GPT-2/CodeGen tokenizer whose special token is <|endoftext|> (id 50256); id 0 there is the literal "!". The loader hardcoded eos_token_id: 0, so the generation stop set was {0} (both generation_config.json and config.json carry no eos, and the nested config object is empty). The decode loop breaks the instant the first sampled token is read as a stop id, before pushing anything, which prints "[Generated 0 tokens]". The BOS prefix embedding also used id 0 instead of the real <|endoftext|>, feeding the decoder a malformed sequence start. Resolve the special-token id from the checkpoint instead: explicit eos_token_id (top-level, then the nested config object), then the tokenizer's declared eos_token looked up in added_tokens_decoder, then the GPT-2 fallback 50256. The resolved id feeds the wrapper eos set, the text-config eos, and the BOS prefix so a normal caption prompt now generates and terminates on the genuine end-of-text. Changes: - src/loading/vlm_special.rs: add resolve_moondream2_eos_token_id plus a tokenizer_config.json reader; moondream2_text_config_value now takes the resolved id and emits both eos and bos; the loader wires the resolved id into the wrapper eos set. - src/models/moondream2.rs: default eos/bos to 50256, add a bos_token_id config field, and have bos_token_id() read it. - src/multimodal/moondream2_prompt.rs: MOONDREAM2_BOS_ID 0 -> 50256. - tests: corrected EOS/BOS defaults, resolver fixtures built from the real config.json / tokenizer_config.json shapes, and a BOS-id lock.
inureyes
added a commit
that referenced
this pull request
Jul 2, 2026
…#616) The round-1 hardening (#609) made the real vikhyatk/moondream2 checkpoint load and generate, but the output was pure garbage (`!NCJNCJ...`). The forward math itself is correct; auditing it layer by layer against the checkpoint's own reference code (text.py, rope.py, layers.py, vision.py) and the working in-tree Moondream3 port turned up no numeric divergence, and a new remap guard test now pins the full 592-tensor key contract. The real divergence is one level up: the 2025-06-21 revision is trained against the moondream/starmie-v1 tokenizer with Moondream3-style control-token templates (query = [1, 15381, 2] + question + [3], bos = eos = 0), but the official repository never removed its legacy GPT-2 tokenizer.json, and the port picked that stale file up, framed the prompt as GPT-2 "Question:/Answer:" text, and prepended BOS 50256. The model saw an out-of-vocabulary word salad, answered with its true EOS (id 0, which decodes to `!` under the wrong vocabulary), was not stopped because round 1 had moved EOS to 50256, and then looped degenerately. Changes: - `moondream2_prompt`: add `Moondream2PromptStyle` (StarmieTemplates vs LegacyQuestionAnswer) with `detect_moondream2_prompt_style`, which reads the checkpoint's bundled moondream.py (it names the tokenizer repo per revision) and falls back to sniffing tokenizer.json for `<|md_reserved_0|>`. Starmie prompts mirror the working Moondream3 template ids exactly; the legacy Question/Answer framing is kept for the 2025-01-09 .. 2025-04-14 revisions where the GPT-2 contract is genuinely correct (verified against those revisions' config.py: bos_id = eos_id = 50256, templates in GPT-2 ids). - `tokenizer::load_tokenizer`: new starmie override ahead of the local tokenizer.json branch. For starmie-era moondream2 checkpoints whose local tokenizer.json is not starmie, resolve moondream/starmie-v1 via hf-hub (the same path the Moondream3 fallback already uses, cached after the first fetch) with an actionable offline error. - `load_moondream2_vlm` / `resolve_moondream2_eos_token_id`: era-aware bos/eos resolution. Starmie era returns 0 and explicitly ignores the stale legacy tokenizer_config.json (which reports 50256); explicit config.json ids still win; legacy behavior is unchanged. - Thread the detected style through the four prompt call sites (CLI generate_vlm, vlm_runtime, both model_worker paths) via a new `Moondream2VLModel::prompt_style` field, and correct the misleading round-1 comments claiming eos 0 was invalid. - Tests: starmie/legacy prompt shaping with exact id sequences, era detection (moondream.py wins over stale tokenizer.json), tokenizer override firing/skipping matrix, era-aware eos resolution against the real config shapes, and a remap guard asserting the real checkpoint's 592 tensor keys map exactly onto the loader-required set with region tensors dropped. Real-checkpoint-gated parity tests confirm era detection on the actual snapshot and that the resolved tokenizer maps "query" to 15381 and id 0 to `<|endoftext|>`; the heavy forward tests stay `#[ignore]`-gated.
16 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up hardening for #522: the merged Moondream2 port loads its real checkpoint (3.48 GB, prepares 16 text + 730 image-prefix tokens) but prints
[Generated 0 tokens]for a normal caption prompt. This fixes the EOS/BOS wiring so generation produces a non-empty description.Root cause
The port inherited Moondream3's
bos_id = eos_id = 0convention. Moondream3 uses id 0 with its starmie tokenizer, but Moondream2 ships a GPT-2/CodeGen tokenizer (tokenizer_config.json/special_tokens_map.json) whose special token is<|endoftext|>= id 50256 and doubles as bos/eos/unk; id 0 there is the literal!.The loader
moondream2_text_config_valuehardcodedeos_token_id: 0, so the generation stop set was exactly{0}(the checkpoint'sconfig.jsonhas an empty nestedconfigobject and no top-level eos, andgeneration_config.jsoncarries no eos). The decode loop breaks the instant the first sampled token is read as a member of the stop set, before any token is pushed, which is exactly the observed[Generated 0 tokens]. Separately, the BOS prefix embedding used id 0 (!) instead of the real<|endoftext|>, feeding the decoder a malformed sequence start.Fix
Resolve the special-token id from the checkpoint instead of hardcoding it. Order: explicit
eos_token_idinconfig.json(top-level, then the nestedconfigobject), then the tokenizer's declaredeos_tokenstring looked up inadded_tokens_decoder, then the GPT-2 fallback50256. The resolved id feeds the wrapper eos set, the text-config eos, and the BOS prefix, so a caption prompt now generates and terminates on the genuine end-of-text.What changed
src/loading/vlm_special.rs: addresolve_moondream2_eos_token_idplus atokenizer_config.jsonreader;moondream2_text_config_valuenow takes the resolved id and emits botheos_token_idandbos_token_id;load_moondream2_vlmwires the resolved id into the wrapper eos set.src/models/moondream2.rs: default eos/bos to50256, add abos_token_idconfig field, and havebos_token_id()read it.src/multimodal/moondream2_prompt.rs:MOONDREAM2_BOS_ID0->50256.config.json/tokenizer_config.jsonshapes, and a BOS-id lock.Test plan
cargo test --lib moondream2(17 passed, 0 failed)cargo check --lib --testscargo clippy --lib --tests -- -D warnings(clean)cargo fmt --check(clean)The bug manifests at GPU generation time; the two real-model checks in
tests/moondream2_parity.rsstay#[ignore]-gated and are validated separately on device.