fix: pass native image/file parts through AgentRuntimeMessage pipeline

[kwakayama]

Problem

When a user sends a file attachment (image or PDF), two things break:

1. File-only messages cause EMPTY_COMPLETION — the LLM receives only an XML annotation with no user intent, produces a minimal/empty response, and the error panel is shown.
2. The LLM never sees actual file content — even with text + file, the image pixels or PDF text are stripped and replaced with an annotation.

Root cause: convertContentToAgentRuntimeParts in agent-runtime-message-adapter.ts stripped all image/file parts from ProviderModelMessage → AgentRuntimeMessage conversion, replacing them with an XML annotation: "Attached files from earlier conversation context: ...". When converting back (AgentRuntimeMessage → ProviderModelMessage), user messages only received string content, so the LLM never got multimodal input.

Fix

• Add { type: "image"; url: string; mediaType: string } and { type: "file"; url: string; mediaType: string } variants to AgentRuntimeMessagePart and AgentRuntimeMessageLikePart
• In convertStructuredPart: return native image/file parts for URL-based attachments (signed URLs from veryfront-agent)
• In convertContentToAgentRuntimeParts: try native conversion first; fall back to XML annotation only when URL is missing or is a data: base64 URL
• In collectAgentRuntimeProviderContentParts + createProviderMessageFromAgentRuntimeMessage: collect file parts and emit structured ChatUserContentPart[] content for user messages
• Add image/file variants to MessagePartSchema in agent.schema.ts
• Guard hosted-child-fork-step-message-preparation.ts against image/file parts (convert to text annotation for child fork messages)

What's preserved

The XML annotation fallback is kept for:

• Base64 data: URLs (can't pass as URL to provider)
• Parts with no URL (e.g. uploadId-only references before URL resolution)
• Unsupported media types (e.g. DOCX) — rewritten to annotation by upstream rewriteUnsupportedFilePartsAsAnnotations

Providers

All three supported providers (Anthropic, OpenAI, Google) support native URL-based multimodal input. No provider-specific handling required.

Reproduction script

deno run --no-check --allow-all scripts/repro-3499-file-attachment.ts

Evidence

Scenario 1: Text + PDF attachment — file passes natively (not as annotation)
  → resolveFileUrl("upload-pdf") → signed URL
  ✓ Text part preserved
  ✓ File part is native (not annotation text)
  ✓ Signed URL forwarded to provider
  ✓ mediaType preserved

Scenario 2: Image-only message — not dropped (was causing EMPTY_COMPLETION)
  ProviderModelMessages:
  [{ "role": "user", "content": [{ "type": "file", "mediaType": "image/png", "url": "https://signed.example.com/screenshot.png" }] }]
  ✓ Message not dropped (EMPTY_COMPLETION fix)
  ✓ Content is structured array (not empty string)
  ✓ File part present in provider message

Scenario 3: Round-trip ProviderModelMessage → AgentRuntime → ProviderModelMessage
  ✓ Image survives round-trip
  ✓ File survives round-trip
  ✓ Image URL preserved
  ✓ File URL preserved

Scenario 4: Unsupported media type (DOCX) → annotation text (expected fallback)
  ✓ DOCX becomes annotation text (appended to message text)

All checks passed.

Related

• Companion fix (URL error propagation): veryfront/veryfront-agent#1210
• Issue: veryfront/veryfront-studio#3499

Test plan

• agent-runtime-message-adapter.test.ts — 8 tests: native passthrough, annotation fallback, round-trip, file-only not dropped
• runtime-message-preparation.test.ts — updated to expect native file part after URL resolution
• hosted-child-fork-step-message-preparation.test.ts — 3 existing tests pass
• Full src/agent/ suite: 280 passed, 0 failed
• scripts/repro-3499-file-attachment.ts — 4 end-to-end scenarios all pass

[kwakayama]

Code Review — 92/100

Fix is correct and complete. The three-layer change (type → convert → collect) is logically sound and all paths are covered by tests.

What's well done

• toNativeFilePart helper eliminates the duplicated data: URL guard and the as "image" | "file" cast — single source of truth for the URL validation logic
• convertStructuredPart now falls back to null cleanly for data: URLs; annotation collection path is explicit with continue
• collectAgentRuntimeProviderContentParts symmetric with the provider→runtime direction — both sides filter data: URLs consistently
• convertAgentRuntimePartToChildForkMessagePart has explicit image/file guard + never throw for future variants
• Single clean commit, bisectable history
• Tests cover: native passthrough, annotation fallback, file-only not dropped, data: URL dropped on round-trip, resolver returning undefined

Remaining debt (pre-existing, not introduced here)

AgentRuntimeMessageLikePart uses { type: string; toolCallId: ... } instead of `tool-${string}`

This prevents TypeScript from discriminating type === "image" cleanly, which is why toNativeFilePart takes type as an explicit parameter rather than reading part.type after narrowing. The helper is the right workaround. The fix for the root cause would be changing the tool-call variants in the union to type: `tool-${string}`, but that's a separate PR.

Verdict: approve once CI is green.

[kwakayama]

Code Review — 92/100

Fix is correct and complete. The three-layer change (type → convert → collect) is logically sound and all paths are covered by tests.

What's well done

• `toNativeFilePart` helper eliminates the duplicated `data:` URL guard and the `as "image" | "file"` cast — single source of truth for the URL validation logic
• `convertStructuredPart` now falls back to `null` cleanly for data: URLs; annotation collection path is explicit with `continue`
• `collectAgentRuntimeProviderContentParts` symmetric with the provider→runtime direction — both sides filter `data:` URLs consistently
• `convertAgentRuntimePartToChildForkMessagePart` has explicit `image`/`file` guard + `never` throw for future variants
• Single clean commit, bisectable history
• Tests cover: native passthrough, annotation fallback, file-only not dropped, `data:` URL dropped on round-trip, resolver returning `undefined`

Remaining debt (pre-existing, not introduced here)

`AgentRuntimeMessageLikePart` uses `{ type: string; toolCallId: ... }` instead of ```tool-${string}```

This prevents TypeScript from discriminating `type === "image"` cleanly, which is why `toNativeFilePart` takes `type` as an explicit parameter rather than reading `part.type` after narrowing. The helper is the right workaround. The fix for the root cause would be changing the tool-call variants in the union to `type: `tool-${string}``, but that is a separate PR.

Verdict: approve once CI is green.

[kwakayama]

Code Review — 92/100

Fix is correct and complete. The three-layer change (type → convert → collect) is logically sound and all paths are covered by tests.

What's well done

• toNativeFilePart helper eliminates the duplicated data: URL guard and the as "image" | "file" cast — single source of truth for the URL validation logic
• convertStructuredPart falls back to null cleanly for data: URLs; annotation collection path is explicit with continue
• collectAgentRuntimeProviderContentParts symmetric with the provider→runtime direction — both sides filter data: URLs consistently
• convertAgentRuntimePartToChildForkMessagePart has explicit image/file guard + never throw for future variants
• Single clean commit, bisectable history
• Tests cover: native passthrough, annotation fallback, file-only not dropped, data: URL dropped on round-trip, resolver returning undefined

Remaining debt (pre-existing, not introduced here)

AgentRuntimeMessageLikePart uses { type: string; toolCallId: ... } instead of tool-${string} — prevents TypeScript from discriminating type === "image" cleanly after narrowing. The toNativeFilePart helper works around it correctly. Root cause fix belongs in a separate PR.

Verdict: approve once CI is green.