[Qwen3.8 R11] Native image and video understanding

[AtlantisPleb]

R11: Native Vision

Native vision remains a separate roadmap lane after text support.

It requires:

• exact image and video preprocessing
• the 27-layer vision encoder
• spatial and temporal patch handling
• vision-to-text projection into the 5,120-wide decoder
• attachment identity and preprocessing receipts
• image and video reference fixtures
• native output parity against an upstream-supported runtime
• bounded image/video size, frame count, timeout, and memory admission

Until R11 lands, prompt marker projection must not be described as image or
video understanding.

Dependencies

• Blocked on stable R2 prompt marker semantics, R3 multimodal tensor inventory, R8 serving, and a separately qualified mmproj/vision artifact plan.

Scope rules

• This phase owns native image/video preprocessing, the 27-layer vision encoder, spatial/temporal patches, and projection into the 5,120-wide text decoder.
• Download and admit vision projector artifacts separately with immutable source, size, digest, dtype, memory, and processor provenance.
• Prompt marker rendering from R2 is not image/video understanding and must remain a refusal on text-only lanes.
• Keep media attachment identity, preprocessing configuration, runtime/backend identity, fallback truth, and bounded limits machine-legible.

Required evidence

• Synthetic processor/operator tests plus real image/video reference fixtures and native parity against a pinned upstream-supported runtime.
• Admission/refusal rows cover dimensions, frame counts, formats, timeouts, memory, malformed inputs, and missing projectors.
• Serving tests cover chat/responses attachment identity, streaming, tools with media, and text-only refusal behavior.

Validation and closure

• Run focused processor, vision-model, backend, and OpenAI serving tests on retained reference fixtures.
• Merge to main and comment with commit SHA, artifact/source digests, exact commands/results, parity cases, supported bounds, residency, fallback truth, and refusals.
• Close only after origin/main supports native bounded image and video understanding; marker-only behavior cannot close it.

Canonical references

• docs/qwen38/IMPLEMENTATION_ROADMAP.md, R11
• docs/qwen38/MODEL_FACTS.md
• docs/qwen38/UPSTREAM_ARTIFACT_INDEX.md
• docs/INFERENCE_ENGINE.md

[AtlantisPleb]

R11 encoder milestone landed on main in 382a08fb3e1ab60f1565bbf53714ac63f2d72b48; retained evidence landed in 8f2430fc.

Implemented:

• separate admission of the official first shard (3,966,730,552 bytes, SHA-256 ba0ce20aae489ad196733da5064bcdf159a1fe84f53336648196e1ebb7751b1c) with all 333 model.visual BF16 tensors / 921,460,192 tensor bytes and both pinned processor config digests
• exact bounded no-resize RGB8 image preprocessing plus bounded 2 fps video frame sampling/temporal patch packing
• native Candle CPU and feature-gated CUDA graph covering patch projection, learned-position interpolation, rotary attention, all 27 blocks, merger, and 5,120-wide output
• receipt validation, layer-boundary timeout refusal, native residency/fallback receipts, portable tests, a pinned Transformers comparator, and a retained checker

Retained image evidence: fixtures/qwen38/reports/qwen38_vision_parity_v1.json. It was generated from pushed main on an idle RTX 4080, driver 595.58.03, against Transformers 0650ff354501cbdb7cb4138da628cc60f4e0ceed, Transformers 5.16.0.dev0, Torch 2.11.0+cu128, torchvision processor backend. Preprocessing is byte-identical. Encoder output normalized RMSE is 0.0667980101638391, p99 absolute error is 0.1171875, and cosine similarity is 0.9977859258651733. The runtime receipt reports 333 resident tensors, 27 resident layers, fallback_policy = refuse, and hidden_fallback_used = false. One-shot elapsed fields are diagnostic and are not a speed claim.

Validation passed:

• cargo test -p psionic-models qwen38_vision --lib (8 passed)
• cargo test -p psionic-models qwen38_native_vision --lib (2 passed)
• cargo check -p psionic-models --features qwen38-vision-cuda --example qwen38_vision_probe
• scripts/run-qwen38-vision-parity-evidence.sh
• scripts/check-qwen38-vision-parity.sh fixtures/qwen38/reports/qwen38_vision_parity_v1.json

Current admitted image bounds are 65,536 through 262,144 pixels with dimensions divisible by 32. Current sampled video bounds are four through eight frames, at most 65,536 pixels per frame, at most 1,024 combined patches, and 8 MiB decoded attachment bytes. Inputs requiring resize, malformed receipts, identity drift, bound violations, missing tensors, expired deadlines, and unavailable backends refuse.

R11 remains partial and open. The retained claim boundary explicitly leaves video encoder parity, vision-to-text injection, and OpenAI media serving false. Those are the next work items.

[AtlantisPleb]

R11 video parity follow-up landed on main: dual image/video harness 4812efe2679e5dc68c0edd15b5d576d4f745c3f2, retained fixtures/docs 207ae8ca.

The clean runner now checks both media rows with an idle GPU before each upstream and native process. Retained video evidence is fixtures/qwen38/reports/qwen38_vision_video_parity_v1.json. The deterministic 8-frame, 4 fps, 256x256 input matches pinned Qwen3VL sampling at [0, 2, 5, 7], temporal grid [2, 16, 16], and byte-identical 512x1536 preprocessing. The native 128x5120 encoder output has normalized RMSE 0.05989835909389969, p99 absolute error 0.1328125, and cosine similarity 0.9983055591583252 against pinned Transformers CUDA eager attention. The runtime reports all 333 tensors / 27 layers resident and no fallback.

The image fixture was regenerated from the same revision and still passes unchanged numerical parity. scripts/check-qwen38-vision-parity.sh passes for both retained reports.

R11 remains partial: decoder embedding injection and chat/responses media serving are still explicitly false. I am continuing with decoder injection next.

[AtlantisPleb]

CPU decoder integration landed on origin/main in 80ff46ff (following the strict decoder-plan commit da00bf1e).

Implemented:

• exact image/video pad-span expansion and immutable multimodal decoder plans
• full preprocessing-input and native encoder-output revalidation before decoder admission
• one 5,120-wide embedding override per exact image/video pad token
• Qwen3.5-compatible three-axis MRoPE positions through CPU full-attention layers
• generated-token physical cache position plus retained MRoPE delta
• explicit CPU generate_qwen38_multimodal entry point and retained plan receipt
• fail-closed posture for multimodal context truncation and MTP speculative decode

Validation:

• cargo check -p psionic-serve passed
• 4/4 qwen38_multimodal model tests passed, including the pinned official tokenizer when available
• focused interleaved-MRoPE serving test passed

A broader qwen35 name-filtered suite has pre-existing stale tiny-fixture failures: the exact CPU publication test fails identically at untouched baseline da00bf1e (qwen35qwen35 vs expected world). That is not counted as R11 evidence.

R11 remains partial: CUDA/Metal decoder plan consumption, HTTP attachment decoding/admission, and retained end-to-end image/video generation evidence are still pending.

[AtlantisPleb]

CUDA decoder integration and retained end-to-end evidence landed on origin/main.

Commits:

• 36925443: native CUDA embedding injection and three-axis MRoPE
• da55a0a0: shared vision-output digest validation
• 27c50842: restored per-head Q/K normalization in the uncaptured multimodal hybrid path
• 80b8059d: clean-main evidence driver and checker
• 14f8d261: retained report and canonical status update

The normalization fix matters because the uncaptured MRoPE path had duplicated raw Q/K copies. Recurrent delta state grew without bound and reached non-finite values at layer 58. The path now shares the captured text graph's L2-equivalent Q/K normalization. The image and video rows complete with finite state, device argmax, and no fallback.

Retained evidence:
fixtures/qwen38/reports/qwen38_multimodal_cuda_evidence_v1.json

Artifacts:

• official revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
• vision shard: 3,966,730,552 bytes, SHA-256 ba0ce20aae489ad196733da5064bcdf159a1fe84f53336648196e1ebb7751b1c
• decoder GGUF: 13,441,059,904 bytes, SHA-256 00cf92e666c6af6566996c38c89a44ccdb6449ea25ef0f112a452c853b2a71e2

Clean idle RTX 4080 / driver 595.58.03 results:

• image: 64 vision rows, 87 prompt tokens, output [760, 2099] / The image
• video: 128 vision rows, 167 prompt tokens, output [760, 2678] / The video
• both: native CUDA vision and decoder execution, 16-byte device-argmax readback, fallback_policy = refuse, hidden_fallback_used = false
• elapsed fields are diagnostic; no performance claim is made

Validation:

• cargo test -p psionic-models qwen38_multimodal --lib (4 passed)
• cargo test -p psionic-backend-cuda tests::cuda_submission_f16_kv_mrope_uses_each_interleaved_axis_when_available --lib -- --exact --nocapture (1 passed)
• cargo check -p psionic-serve --features qwen38-vision-cuda --example qwen38_multimodal_cuda_smoke
• scripts/run-qwen38-multimodal-cuda-evidence.sh
• scripts/check-qwen38-multimodal-cuda.sh fixtures/qwen38/reports/qwen38_multimodal_cuda_evidence_v1.json

R11 remains partial and open. Metal decoder-plan consumption and OpenAI chat/responses media serving remain incomplete.

[AtlantisPleb]

R11 serving and retained live evidence landed on origin/main.

• 864336de serves bounded Qwen3.8 image/video inputs through Chat Completions and Responses with attachment identity, streamed output, deterministic refusal posture, and media-aware tool execution.
• 4b4547a9 keeps CPU vision execution in F32 while preserving the source BF16 artifact identity.
• 16ae1955 fixes CUDA long-context attention so logits beyond one 256-thread block are initialized; the regression covers a 257-token cache.
• 5b215c5e requires a complete parsed media tool call rather than accepting truncated tool syntax.
• db868187 retains the live report at fixtures/qwen38/reports/qwen38_openai_media_evidence_v1.json and updates the canonical docs.

The retained run used an idle RTX 4080 and pinned model/shard digests. It proves a 374-token image Chat request with a parsed record_visual_summary tool call, streamed GIF input, stored Responses image input, exact identity headers, and deterministic refusals for binary continuation, remote URLs, MP4, malformed base64, and attachment overflow. The decoder plan records 64 image tokens, 310 text tokens, and no fallback.

Validation passed for the retained evidence checker, focused Qwen3.8 serving tests, the 257-token CUDA regression, and the full live media evidence gate. R11 remains partial and open because the retained Metal runtime still does not consume the same multimodal decoder plan.

[AtlantisPleb]

Metal decoder-plan consumption landed in fe41d38.

What changed:

• Metal layer forwarding carries [usize; 3] MRoPE positions; full-attention
  layers apply section-aware apply_rope_neox_mrope to query and key.
• MetalQwen35Model::forward_token_with_embedding_and_mrope accepts an
  optional 5,120-wide embedding override.
• MetalGgufQwen35TextGenerationService::generate_qwen38_multimodal uses plan
  token ids, injects embedding overrides at exact pad-token indices, applies
  plan MRoPE positions during prefill, applies the generated-token position
  delta during decode, refuses context-window prompt truncation and
  out-of-range overrides, and retains the successful plan receipt.
• The generic OpenAI server now permits --qwen38-vision-model-dir with the
  Metal backend, loads Qwen38OpenAiMultimodalRuntime, and dispatches
  multimodal requests to the Metal service. fallback_policy = refuse is
  unchanged; no hidden fallback was added.

Focused validation on this host (Apple Silicon):

• native_qwen38_metal_multimodal_step_matches_cpu_when_available — CPU and
  Metal raw logits agree to 1e-4 under an embedding override and non-uniform
  MRoPE positions [2, 5, 7]; the test also requires the Metal output to
  change under non-uniform positions and under the override, so consumption is
  proven rather than assumed from argmax ties on a zero fixture.
• cargo test -p psionic-serve --lib qwen38 — 26 tests pass.

Remaining for full R11 closure per the roadmap: a retained full-model Metal
multimodal evidence run (vision artifact + decoder artifact, serial execution
on an idle Apple Silicon host). No Metal multimodal evidence script exists
yet; the interactive macOS host currently runs Ollama, which the evidence
idle-host policy refuses on.