Skip to content

fix(agent): preserve attachment context in text-generation prompts - #1475

Merged
kwakayama merged 1 commit into
mainfrom
vf-chat-attachment-runtime-preservation
May 8, 2026
Merged

kwakayama merged 1 commit into
mainfrom
vf-chat-attachment-runtime-preservation

Conversation

@kwakayama

@kwakayama kwakayama commented May 8, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Preserve native user image/file parts while also adding provider-visible uploaded-file text context.
  • Ensure text-generation conversion includes <uploaded_files> metadata for native attachments instead of dropping them.
  • Avoid duplicate <uploaded_files> annotations during provider round trips.
  • Keep attachment annotations readable: separated from user text by a blank line, without leading blank lines for file-only prompts.
  • Add regression coverage for the synthetic PDF path through message preparation, text-generation conversion, and a fake model prompt.
  • Bump package version from 0.1.426 to 0.1.427 in deno.json and src/utils/version-constant.ts.

Addresses https://github.com/veryfront/veryfront-studio/issues/3510 and the upstream attachment-context portion of https://github.com/veryfront/veryfront-studio/issues/3499.

Why this is needed

#1469 preserved native file/image parts through the AgentRuntimeMessage pipeline, but the text-generation runtime path still needed provider-visible attachment context. Without that context, the model can respond as if no attachment was present even though the persisted chat message contains a file part.

Regression coverage

  • Runtime message preparation keeps signed PDF metadata visible.
  • Text-generation conversion includes one readable <uploaded_files> annotation.
  • File-only prompts do not start with leading blank lines.
  • Provider round trips do not duplicate existing attachment annotations.
  • Fake model prompt test proves the final model-facing prompt contains the uploaded PDF metadata.

Test plan

  • deno fmt --check deno.json src/utils/version-constant.ts src/agent/agent-runtime-message-adapter.ts src/agent/agent-runtime-message-adapter.test.ts src/agent/runtime-message-preparation.test.ts src/agent/runtime/text-generation-runtime-message-converter.ts src/agent/runtime/text-generation-runtime-message-converter.test.ts
  • DENO_NO_PACKAGE_JSON=1 deno lint src/utils/version-constant.ts src/agent/agent-runtime-message-adapter.ts src/agent/agent-runtime-message-adapter.test.ts src/agent/runtime-message-preparation.test.ts src/agent/runtime/text-generation-runtime-message-converter.ts src/agent/runtime/text-generation-runtime-message-converter.test.ts
  • deno test --no-check --allow-all --unstable-worker-options --unstable-net src/agent/agent-runtime-message-adapter.test.ts src/agent/runtime-message-preparation.test.ts src/agent/runtime/text-generation-runtime-message-converter.test.ts — 7 passed / 31 steps / 0 failed
  • deno check src/agent/agent-runtime-message-adapter.ts src/agent/runtime-message-preparation.ts src/agent/runtime/text-generation-runtime-message-converter.ts

Follow-up required after merge

After this package version is published, veryfront-agent needs a dependency-only PR bumping veryfront to the published version containing this change, followed by staging rollout and end-to-end attachment retest.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ce97946e07

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines 218 to 220
if (convertedPart) {
parts.push(convertedPart);
continue;
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve single attachment annotation across round trips

Re-processing a provider message that already contains an <uploaded_files> text annotation now appends a second annotation, because converted file/image parts no longer short-circuit before being collected into attachmentReferences. This happens in the child-fork flow (prepareHostedChildForkRuntimeStepMessages converts agent messages to provider messages and back), so attachment metadata can be duplicated on each round trip, inflating prompt size and increasing token pressure for long conversations with files.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in c12f800: provider messages that already contain an <uploaded_files> text annotation now skip adding the fallback attachment-context part, while still preserving the native file/image part. Added a provider round-trip regression test that asserts only one annotation remains.

@kwakayama
kwakayama force-pushed the vf-chat-attachment-runtime-preservation branch from ce97946 to c12f800 Compare May 8, 2026 09:45
@kwakayama kwakayama added the ready-for-review Agent-prepared work is ready for human review label May 8, 2026
@kwakayama
kwakayama force-pushed the vf-chat-attachment-runtime-preservation branch from 8d12544 to a29ba1a Compare May 8, 2026 15:47
@kwakayama
kwakayama force-pushed the vf-chat-attachment-runtime-preservation branch from a29ba1a to dec60bf Compare May 8, 2026 17:15
@kwakayama kwakayama changed the title fix(agent): preserve chat attachment context fix(agent): preserve attachment context in text-generation prompts May 8, 2026
@kwakayama
kwakayama merged commit 981fe93 into main May 8, 2026
18 checks passed
@kwakayama
kwakayama deleted the vf-chat-attachment-runtime-preservation branch May 8, 2026 17:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-review Agent-prepared work is ready for human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant