Skip to content

Fix Image Pipeline Diagnostics and Type Safety #642

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ngoiyaeric merged 2 commits into from
Jun 4, 2026
Merged

Fix Image Pipeline Diagnostics and Type Safety #642

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4f17f17
Fix image attachment pipeline: add diagnostics, improve type safety, …
google-labs-jules[bot] Jun 4, 2026
9e27c40
Fix build error and finalize image pipeline improvements
google-labs-jules[bot] Jun 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
143 changes: 100 additions & 43 deletions app/actions.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import {
getAIState,
getMutableAIState
} from 'ai/rsc'
import { CoreMessage, ToolResultPart } from 'ai'
import { CoreMessage, ToolResultPart, TextPart, ImagePart } from 'ai'
import { nanoid } from '@/lib/utils'
import type { FeatureCollection } from 'geojson'
import { Spinner } from '@/components/ui/spinner'
Expand Down Expand Up @@ -242,6 +242,12 @@ async function submit(formData?: FormData, skip?: boolean) {
}

const file = !skip ? (formData?.get('file') as File) : undefined
console.log('File extraction:', {
exists: !!file,
name: file?.name,
type: file?.type,
size: file?.size
})
const userInput = skip
? `{"action": "skip"}`
: ((formData?.get('related_query') as string) ||
Expand Down Expand Up @@ -328,67 +334,118 @@ async function submit(formData?: FormData, skip?: boolean) {
}
}

const messages: CoreMessage[] = [...(aiState.get().messages as any[])].filter(
(message: any) =>
message.role !== 'tool' &&
message.type !== 'followup' &&
message.type !== 'related' &&
message.type !== 'end' &&
message.type !== 'resolution_search_result'
).map((m: any) => {
if (Array.isArray(m.content)) {
return {
...m,
content: m.content.filter((part: any) =>
part.type !== "image" || (typeof part.image === "string" && part.image.startsWith("data:"))
)
} as any
}
return m
let filteredImagesCount = 0
let retainedImagesCount = 0
const messages: CoreMessage[] = [...(aiState.get().messages as any[])]
.filter(
(message: any) =>
message.role !== 'tool' &&
message.type !== 'followup' &&
message.type !== 'related' &&
message.type !== 'end' &&
message.type !== 'resolution_search_result'
)
.map((m: any) => {
if (Array.isArray(m.content)) {
const filteredContent = m.content.filter((part: any) => {
if (part.type === 'image') {
const isValid =
typeof part.image === 'string' &&
(part.image.startsWith('data:') ||
part.image === 'IMAGE_PROCESSED')
if (isValid) {
retainedImagesCount++
} else {
filteredImagesCount++
}
return isValid
}
return true
})
return {
...m,
content: filteredContent
} as any
Comment on lines +348 to +368

@coderabbitai coderabbitai Bot Jun 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🌐 Web query:

For AI SDK version 4.3.19, what values are valid for ImagePart.imageinsideCoreMessage['content']? Is an arbitrary sentinel string like IMAGE_PROCESSED supported in provider requests, or must it be an actual data URL/URL/binary payload?

💡 Result:

For AI SDK version 4.3.19, the ImagePart.image property is designed to hold actual image data or a reference to it, and it does not support arbitrary sentinel strings like IMAGE_PROCESSED [1][2]. The valid types for ImagePart.image are restricted to the following [1][3][4]: - Data payloads: base64-encoded strings, Uint8Array, ArrayBuffer, or Buffer [1][3][4]. - URLs: a URL object or string pointing to an image (e.g., http(s) URLs or base64 data URLs) [1][3][4]. Providing an arbitrary string that does not conform to these formats (i.e., is not a valid data representation or a valid URL) will generally cause the SDK or the underlying model provider to fail, as they expect valid image content to process [3][4]. Note that in recent versions of the AI SDK, ImagePart has been deprecated in favor of FilePart (using { type: 'file', mediaType: 'image', data: ... }) [5][6]. If you need to include metadata or indicate that an image has already been processed, this should be handled through other fields, such as providerOptions, or by managing the file state outside of the ImagePart content array [7][6].

Citations:

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Show the exact image filtering block in app/actions.tsx
echo "=== app/actions.tsx (filtering around retainedImagesCount) ==="
sed -n '320,460p' app/actions.tsx | nl -ba | sed -n '1,200p'

# Show the submit flow where the CoreMessage[] is built / aiState is appended
echo
echo "=== app/actions.tsx (building CoreMessage / appending user messages) ==="
sed -n '460,560p' app/actions.tsx | nl -ba | sed -n '1,220p'

# Show where onSetAIState persists messages
echo
echo "=== app/actions.tsx (onSetAIState persistence) ==="
sed -n '650,780p' app/actions.tsx | nl -ba | sed -n '1,200p'

# Show lib/agents/researcher.tsx where messages are forwarded to nonexperimental_streamText
echo
echo "=== lib/agents/researcher.tsx (forwarding to nonexperimental_streamText) ==="
sed -n '1,220p' lib/agents/researcher.tsx | nl -ba | sed -n '1,220p'

Repository: QueueLab/QCX

Length of output: 180

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== app/actions.tsx (filtering around retainedImagesCount) ==="
sed -n '320,460p' app/actions.tsx | cat -n

echo
echo "=== app/actions.tsx (building CoreMessage / appending user messages) ==="
sed -n '460,560p' app/actions.tsx | cat -n

echo
echo "=== app/actions.tsx (onSetAIState persistence) ==="
sed -n '650,780p' app/actions.tsx | cat -n

echo
echo "=== lib/agents/researcher.tsx (forwarding to nonexperimental_streamText) ==="
sed -n '1,220p' lib/agents/researcher.tsx | cat -n

Repository: QueueLab/QCX

Length of output: 21615

Stop preserving IMAGE_PROCESSED as an ImagePart.image sent to the SDK
app/actions.tsx’s historical message filter keeps { type: 'image', image: 'IMAGE_PROCESSED' }, and lib/agents/researcher.tsx forwards messages unchanged into nonexperimental_streamText (and enables the “has image” path whenever any type: 'image' part exists). ImagePart.image must be real image data/URL (or binary payload), not an arbitrary sentinel—strip/replace this placeholder when building the CoreMessage[] passed to the model, while allowing the sentinel only for persisted/UI history.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@app/actions.tsx` around lines 348 - 368, The historical message filter
currently preserves image parts with ImagePart.image === 'IMAGE_PROCESSED';
update the filter logic (the map over messages that builds filteredContent) to
exclude or replace the 'IMAGE_PROCESSED' sentinel when constructing messages
that will be sent to the model (CoreMessage[]), not just when
persisting/displaying history. Ensure the UI/persistence code can still store
the sentinel but the code path that prepares messages for the model (the place
where messages are forwarded into nonexperimental_streamText in researcher
logic) replaces 'IMAGE_PROCESSED' with a real image payload or removes the image
part so the model never receives the sentinel; reference the ImagePart.image
checks and the nonexperimental_streamText/CoreMessage[] construction to locate
the changes.

}
return m
})
console.log('Historical messages image filter:', {
filteredImagesCount,
retainedImagesCount,
totalMessages: messages.length
})

const groupeId = nanoid()
const useSpecificAPI = process.env.USE_SPECIFIC_API_FOR_WRITER === 'true'
const maxMessages = useSpecificAPI ? 5 : 10
messages.splice(0, Math.max(messages.length - maxMessages, 0))

const messageParts: {
type: 'text' | 'image'
text?: string
image?: string
mimeType?: string
}[] = []
const messageParts: (TextPart | ImagePart)[] = []

if (userInput) {
messageParts.push({ type: 'text', text: userInput })
}

if (file) {
const buffer = await file.arrayBuffer()
if (file.type.startsWith('image/')) {
const dataUrl = `data:${file.type};base64,${Buffer.from(
buffer
).toString('base64')}`
messageParts.push({
type: 'image',
image: dataUrl,
mimeType: file.type
})
} else if (file.type === 'text/plain') {
const textContent = Buffer.from(buffer).toString('utf-8')
const existingTextPart = messageParts.find(p => p.type === 'text')
if (existingTextPart) {
existingTextPart.text = `${textContent}\n\n${existingTextPart.text}`
} else {
messageParts.push({ type: 'text', text: textContent })
const MAX_FILE_SIZE = 10 * 1024 * 1024 // 10MB
if (file.size > MAX_FILE_SIZE) {
console.error('File size exceeds 10MB limit:', file.size)
} else {
try {
const buffer = await file.arrayBuffer()
console.log('File buffer loaded:', { size: buffer.byteLength })
if (file.type.startsWith('image/')) {
const dataUrl = `data:${file.type};base64,${Buffer.from(
buffer
).toString('base64')}`
console.log('Image processed:', {
dataUrlPrefix: dataUrl.substring(0, 50),
totalLength: dataUrl.length
})
const imagePart: ImagePart = {
type: 'image',
image: dataUrl,
mimeType: file.type
}
console.log('Pushing image part (debug shape):', {
...imagePart,
image: dataUrl.substring(0, 50) + '...'
})
Comment on lines +401 to +413

@coderabbitai coderabbitai Bot Jun 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Don’t log raw base64 image bytes.

The dataUrl.substring(0, 50) and imagePart.image.substring(0, 50) logs still emit user-uploaded content into server logs. These attachments can contain sensitive data, and logs usually outlive request bodies. Log MIME type, byte length, or a hash instead.

Safer logging 
           console.log('Image processed:', {
-            dataUrlPrefix: dataUrl.substring(0, 50),
-            totalLength: dataUrl.length
+            mimeType: file.type,
+            payloadBytes: buffer.byteLength,
+            totalLength: dataUrl.length
           })
           const imagePart: ImagePart = {
             type: 'image',
             image: dataUrl,
             mimeType: file.type
           }
           console.log('Pushing image part (debug shape):', {
-            ...imagePart,
-            image: imagePart.image.substring(0, 50) + '...'
+            type: imagePart.type,
+            mimeType: imagePart.mimeType,
+            totalLength: imagePart.image.length
           })
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@app/actions.tsx` around lines 401 - 413, The logs currently leak base64 image
bytes via dataUrl.substring(...) and imagePart.image.substring(...) — remove
those substring logs and replace them with non-sensitive metadata: log image
MIME type (file.type), byte/character length (dataUrl.length), and a short
content fingerprint (e.g., SHA-256 hex or truncated hash computed from dataUrl)
instead; update the console messages around dataUrl and imagePart (the places
where dataUrl, imagePart, and ImagePart are constructed) to output only
mimeType, totalLength, and the computed hash/fingerprint.

messageParts.push(imagePart)
Comment on lines +405 to +414

@coderabbitai coderabbitai Bot Jun 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Don’t persist raw data URLs in chat history.

When an image is attached, the full base64 data URL is pushed into aiState.messages here. onSetAIState later forwards state.messages straight to saveChat() at Lines 707-735, so each upload is retained in chat storage. A 10MB image becomes ~13MB once base64-encoded, and it also stores the raw user attachment indefinitely. Replace stored images with IMAGE_PROCESSED or an asset reference before persistence.

Also applies to: 458-475

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@app/actions.tsx` around lines 405 - 414, The code currently pushes raw base64
data URLs into aiState.messages via the imagePart object (type: 'image', image:
dataUrl) which later flows through onSetAIState into saveChat(), causing large
binary blobs to be persisted; update the logic that constructs and pushes
imagePart (and the similar block referenced at the other location) to avoid
storing the full data URL—replace the image field with either a sentinel string
like 'IMAGE_PROCESSED' or an asset reference/id after uploading the file to your
asset store, and only keep the real base64/data URL transiently for immediate
use (e.g., preview/upload) before substituting the persistent imagePart prior to
calling onSetAIState/saveChat.

} else if (file.type === 'text/plain') {
const textContent = Buffer.from(buffer).toString('utf-8')
const existingTextPart = messageParts.find(
(p): p is TextPart => p.type === 'text'
)
if (existingTextPart) {
existingTextPart.text = `${textContent}\n\n${existingTextPart.text}`
} else {
messageParts.push({ type: 'text', text: textContent })
}
}
} catch (error) {
console.error('Error processing file:', error)
}
Comment on lines 389 to 428

@coderabbitai coderabbitai Bot Jun 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Fail fast when attachment processing yields no usable part.

Oversize files, unsupported MIME types, and arrayBuffer() failures only log and then fall through. If the request is file-only, Line 327 does not short-circuit because file exists, but no new user message is appended at Lines 458-475, so processEvents() runs against stale history instead of returning a user-facing error for the bad upload.

Suggested direction 
   if (file) {
     const MAX_FILE_SIZE = 10 * 1024 * 1024 // 10MB
     if (file.size > MAX_FILE_SIZE) {
-      console.error('File size exceeds 10MB limit:', file.size)
+      throw new Error('File size must be less than 10MB')
     } else {
       try {
         const buffer = await file.arrayBuffer()
         ...
-        }
+        } else {
+          throw new Error(`Unsupported file type: ${file.type || 'unknown'}`)
+        }
       } catch (error) {
-        console.error('Error processing file:', error)
+        throw error
       }
     }
   }

Also applies to: 458-475

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@app/actions.tsx` around lines 389 - 428, The file-processing block that
builds messageParts (the branch handling file, MAX_FILE_SIZE, image/text
conversion, and the catch) currently only logs errors and can leave messageParts
unchanged; update this logic to fail fast by detecting when no usable part was
produced and then either return/throw immediately from the enclosing action or
append a user-facing TextPart error before continuing so processEvents() never
runs on stale history; specifically, after the image/text handling and inside
the catch/oversize/unsupported-MIME branches, set a boolean flag (e.g.,
fileHandled) or push a TextPart like {type: 'text', text: 'Upload failed: <brief
reason>'} and short-circuit the function (or return that error to the caller) so
processEvents() is not invoked with no new message part. Ensure changes touch
the same symbols: the file handling block that uses file.arrayBuffer(),
messageParts, ImagePart/TextPart, and the later processEvents() caller.

}
}

const hasImage = messageParts.some(part => part.type === 'image')
console.log('messageParts structure:', {
parts: messageParts.map(p => ({
type: p.type,
length: p.type === 'text' ? p.text.length : undefined
})),
hasImage
})
const content: CoreMessage['content'] = hasImage
? messageParts as CoreMessage['content']
: messageParts.map(part => part.text).join('\n')
? messageParts
: messageParts.map(part => (part.type === 'text' ? part.text : '')).join('\n')
console.log('Final content structure:', {
hasImage,
contentType: typeof content,
isArray: Array.isArray(content),
partsCount: Array.isArray(content) ? content.length : 'N/A'
})

const type = skip
? undefined
Comment on lines 385 to 451

@qodo-code-review qodo-code-review Bot Jun 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

1. File-only upload dropped 🐞 Bug ≡ Correctness

In submit(), if the user submits only a file (no text) and the file is >10MB or processing throws,
the code only logs and continues, leaving messageParts empty; this makes content an empty string and
the if (content) guard skips appending the user message. The pipeline then runs
taskManager/researcher using only historical messages, so the response ignores the user’s upload
while still consuming tokens.
Agent Prompt 
## Issue description
`submit()` allows file-only requests, but when the file is rejected (size > 10MB) or throws during processing, the request continues without creating a user message. This results in the system running `taskManager()`/`researcher()` against stale `messages`, producing an unrelated response and silently ignoring the user’s upload.

## Issue Context
- The code returns early only when *both* `userInput` and `file` are missing.
- If the file is too large or processing fails, no message part is added and `content` becomes `''`.
- Because message append is guarded by `if (content)`, empty string prevents the user message from being appended.

## Fix Focus Areas
- app/actions.tsx[327-335]
- app/actions.tsx[385-475]
- app/actions.tsx[481-529]

## What to change
- When `file.size > MAX_FILE_SIZE`, stop the pipeline (early return) and surface a user-visible error (e.g., append an assistant error message to `uiStream` and `aiState.done(...)`, and set `isGenerating.done(false)` / `uiStream.done()`).
- In the `catch` for file processing, similarly fail fast with a user-visible error.
- Additionally, guard against `messageParts.length === 0` before building/using `content` (so file-only requests can’t fall through into `processEvents()` with no new user message).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Expand Down
22 changes: 22 additions & 0 deletions lib/agents/researcher.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,28 @@ export async function researcher(
message.content.some(part => part.type === 'image')
)

const lastUserMessage = [...messages].reverse().find(m => m.role === 'user')
console.log('Researcher - Image pipeline trace:', {
hasImage,
totalMessages: messages.length,
messagesWithImages: messages.filter(
m =>
Array.isArray(m.content) && m.content.some(p => p.type === 'image')
).length,
lastUserMessageContentStructure: lastUserMessage
? {
type: typeof lastUserMessage.content,
isArray: Array.isArray(lastUserMessage.content),
parts: Array.isArray(lastUserMessage.content)
? lastUserMessage.content.map(p => ({
type: p.type,
hasImage: p.type === 'image'
}))
: 'string'
}
: 'none'
})

const result = await nonexperimental_streamText({
model: (await getModel(hasImage)) as LanguageModel,
maxTokens: 4096,
Expand Down