Skip to content

fix(mcore-adapter): drop mm_token_type_ids before forwarding into GPTModel.forward - #493

Open
AmirF194 wants to merge 1 commit into
alibaba:mainfrom
AmirF194:fix/457-drop-mm-token-type-ids
Open

fix(mcore-adapter): drop mm_token_type_ids before forwarding into GPTModel.forward#493
AmirF194 wants to merge 1 commit into
alibaba:mainfrom
AmirF194:fix/457-drop-mm-token-type-ids

Conversation

@AmirF194

Copy link
Copy Markdown
Contributor

What

Qwen3_5Model.forward (mcore_adapter/src/mcore_adapter/models/qwen3_5/modeling_qwen3_5.py)
pops force_vit_image/force_vit_video out of **kwargs and then forwards the rest of
kwargs unfiltered into super().forward(...), which resolves through McaGPTModel to
Megatron's GPTModel.forward. That method has a fixed keyword signature with no **kwargs
catch-all, so any extra key raises TypeError at argument-binding time, before the method
body runs.

transformers' Qwen3VLProcessor (used for Qwen3.5-VL too) sets return_mm_token_type_ids: True by default and lists mm_token_type_ids in its model_input_names. Nothing in ROLL or
mcore_adapter reads that key (grep -rn "token_type" . returns zero hits repo-wide), so a
Qwen3.5 VLM batch carries it straight through to GPTModel.forward and crashes every
training/inference forward call, matching the traceback in #457.

Fix

Pop mm_token_type_ids next to the two VIT-only kwargs that are already popped, before the
kwargs dict reaches either of the two super().forward() call sites in this method (both
share the same dict, so one pop covers both paths).

Verification

  • Reproduced the exact TypeError: GPTModel.forward() got an unexpected keyword argument 'mm_token_type_ids' in a clean python:3.11-slim container against current HEAD
    (192b1a0), driving a bare Qwen3_5Model instance through forward() the way a
    non-first pipeline-parallel stage calls it (no GPU/NPU needed: the failure is pure
    keyword-argument binding).
  • Added tests/models/test_qwen3_5_forward_kwargs.py (2 tests). Ran both on unmodified
    192b1a0 (both fail with the TypeError above) and on this branch (both pass), in the
    same container.
  • Confirmed against the installed transformers package that mm_token_type_ids is the
    only processor-emitted key beyond what forward() already declares by name.
  • Not verified: an end-to-end Qwen3.5-VL RL training run (needs GPU/NPU and real model
    weights, unavailable here). No CI workflow in this repo currently runs tests/models/
    (ci-npu-test.yml only runs tests/utils and tests/third_party/sglang), so this test
    isn't wired into a gate; happy to add that if you'd like.

Fixes #457

FYI, out of scope here

mcore_adapter/src/mcore_adapter/models/qwen3_vl/modeling_qwen3_vl.py has the identical
pattern (pops force_vit_image/force_vit_video only, forwards the rest of kwargs
unfiltered) and the same processor would hit it the same way. I did not reproduce or touch
it since #457 is scoped to Qwen3.5; flagging in case it's worth a follow-up.

@AmirF194

AmirF194 commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

Bumping this after a week of quiet. The fix is narrow: Megatron's GPTModel.forward chokes on mm_token_type_ids from the multimodal processor, so this drops the unrecognized kwargs before the call. CI's clean if that helps triage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TypeError: GPTModel.forward() got an unexpected keyword argument 'mm_token_type_ids'

1 participant