feat(mm): serve image input on the Qwen families

[jason-fxz]

Image input for the Qwen VL families: Qwen3-VL (dense and MoE), Qwen3.6 (27B dense, 35B-A3B), Qwen3.8-Flash-Next. Images arrive on all three protocols, are encoded by the family's vision tower inside the engine and served through the existing prefix cache, chunked prefill and offload paths. Text-only serving is unchanged: a checkpoint without a vision section, or one started with --text-model-only, builds no tower and takes no extra VRAM.

How it fits together

• Frontend. Images become MMItems in the tokenizer workers; the engine never sees bytes.
  • OpenAI image_url, Anthropic image blocks and Responses input_image; sources are http(s), base64, and file:// behind --allowed-local-media-path; URL hosts are gated by --allowed-media-domains.
  • Images inside Anthropic tool_result are rejected with a 400 rather than dropped: chat templates render tool messages as text.
  • The family's MMProcessor runs the HF image processor, expands the placeholder to the image's token count, and writes a content-derived pseudo id on the image span so the radix cache keys it by content (the scheme sglang uses).
  • Mrope positions are precomputed per request and travel with the message; the tokenizer workers stay free of model code.
  • Image resolution is a server setting, not a per-request one. --image-min-tokens / --image-max-tokens give a per-image token budget; the family's MMProcessor.get_mm_processor_kwargs converts it to its HF image processor's own limits (Qwen VL: one token is a 32x32 patch of the resized image, so the budget becomes pixel areas in size.shortest_edge / longest_edge), and --mm-processor-kwargs hands a JSON object straight to that call for family-specific knobs, applied after the budget. The processor is built once per process with these settings and injected into the tokenizer, so the same image always yields the same tokens, hash and cache key.
• Registration. One registry row per family decides everything downstream.
  • EncoderSpec(kind, config_key, modalities) on ModelSpec: Qwen3-VL, Qwen3.6 and Qwen3.8-Flash-Next register a vision tower serving image; ModelSpec.mm_processor names the processor.
  • EngineConfig.active_encoders = registered by the family, present in the checkpoint config, not disabled by --mm-disable; the parser never sees the config section of a tower that is not built, so a text-only process also has no mrope.
  • served_modalities derives from the built encoders and gates the API in the server process without asking the engine.
• Scheduler and engine. Encoder work is planned per prefill batch and cached per image.
  • The encoder cache holds one [rows, D] embedding per image, owned by the requests that still have rows to gather: an image survives chunk boundaries, repeats within a request and sharing across requests, and dies when the last row is gathered or the request aborts.
  • Each prefill batch gets a plan: which items to encode (cache misses, one job per hash), which embedding rows to gather, and the batch rows they land on (scheduler/mm.py).
  • The engine encodes right before the LM forward, gathers into batch.mm_embeds and passes batch.mm_rows; the model does one index_copy_ and never scans ids.
• Models. The Qwen VL tower reuses the text stack and stays small on the GPU.
  • Built on the text tower's TP-aware projection layers (fused qkv, column/row parallel MLP), so quantization config and TP sharding apply to it like any other layer; DeepStack taps for Qwen3-VL ride along as extra output columns.
  • Block weights stream from pinned host banks two blocks at a time behind the compute by default (--mm-encoder-weights host): about 60 MiB of VRAM instead of the whole tower, at a small-image latency cost.
  • Merger outputs are parked on the host between blocks, so the activation peak at 4096x4096 is the attention working set of one block (0.72 GiB on Qwen3-VL-8B).
  • The ids the model receives are the radix keys: image rows hold the content pseudo id (MM_PAD_SHIFT_VALUE + hash % 2^30, above every real token), so the same image under the same prefix is a cache hit and a different image forks the tree with no extra key. The embedding lookup clamps those ids into the vocab and the copy overwrites the rows, so no layer ever sees them; the one feature that hashes token ids besides the embedding, Qwen3.8's per-layer n-gram embedding, restores <|image_pad|> on image rows first and matches HF's ple_input_ids.
  • SupportsMultimodal (encode, place_encoder_weights) is the engine's contract, checked at start-up; embed_input_ids in models/blocks.py is the shared step every multimodal text forward calls.
• Server. Capability is reported, not inferred from the checkpoint.
  • GET /v1/stats carries model.input_modalities (["text"] or ["text", "image"]) so a client can gate its attachment controls.
  • A request with images against a server that serves none gets a client-facing reason naming the flag that disabled it.

Flags

Flag	Default	Meaning
--text-model-only	off	Build no encoder tower, reject every multimodal input (same as --mm-disable with every kind)
--mm-disable {vision,audio}	none	Leave the named towers unbuilt
--mm-encoder-weights {host,gpu}	host	Stream the tower's block weights from pinned host banks, or keep them resident
--mm-embed-cache-device {cpu,cuda}	cpu	Where encoded embeddings wait between prefill chunks
--image-min-tokens N, --image-max-tokens N	processor defaults	Per-image token budget: every image is resized to take between N_min and N_max tokens, in the family's own units (Qwen VL: N x 1024 pixels into size.shortest_edge / longest_edge; checkpoint defaults 64 to 16384 tokens). Families with fixed tiers honor the maximum only
--mm-processor-kwargs JSON	none	JSON object of extra keyword arguments for the checkpoint's image processor call, for knobs the budget does not cover (Qwen VL: {"size": {"longest_edge": 1048576}}); applied after the budget, so an explicit key wins

Verification

• Vision tower bitwise equal to the HF reference (fp32 truth) on Qwen3-VL-8B at 448x448, 1024x1024 and 4096x4096, with resident and host-streamed weights, and with the parked-merger forward against a naive reference forward.
• Greedy image smokes with identical outputs across every refactor: Qwen3-VL-8B (bf16), Qwen3.6-27B (bf16 and NVFP4), Qwen3.6-35B-A3B (FP8, MoE offload), Qwen3.8-Flash-Next (NVFP4, MoE offload); a request with the same image twice split across a chunk boundary; two image requests admitted into one prefill batch match their single-request outputs; --text-model-only on a VL checkpoint matches the text model.
• --image-max-tokens 2048 caps a 3840x2160 image at 2048 tokens on Qwen3.6-27B and Qwen3-VL-8B where the default takes more; --image-max-tokens 64 on Qwen3-VL-8B still describes the smoke image correctly; the flags land in MultimodalConfig and a minimum above the maximum is rejected at start-up.
• Memory on Qwen3-VL-8B at 4096x4096: activation peak 1.99 -> 0.72 GiB; resident tower weights 1099 -> 373 MiB with host streaming. Cost of host streaming: 448x448 encode 7.5 -> 17.0 ms, no change from 1024x1024 up. A rebuild to the current cache capacity succeeds with host-streamed weights.

Not in this PR

• Video and audio input: the tower already accepts grid_thw with t > 1, the processor does not sample frames yet; EncoderSpec and --mm-disable are ready for an audio tower.
• Tensor-parallel serving of the vision tower is wired (per-rank shards) but untested end to end.
• Gemma-4 image input follows on a stacked branch.