Skip to content

fix(provider): cache all system messages instead of only first 2 - #38206

Closed
zouhua021 wants to merge 1 commit into
anomalyco:devfrom
zouhua021:feat/prompt-cache-partitioning
Closed

fix(provider): cache all system messages instead of only first 2#38206
zouhua021 wants to merge 1 commit into
anomalyco:devfrom
zouhua021:feat/prompt-cache-partitioning

Conversation

@zouhua021

Copy link
Copy Markdown

Problem

applyCaching() in provider/transform.ts slices system messages to only the first 2:

const system = msgs.filter((msg) => msg.role === "system").slice(0, 2)

This means additional system messages (from plugins via experimental.chat.system.transform, MCP instructions, AGENTS.md content, etc.) are never marked with cacheControl. Since the system prompt is the most stable prefix across turns, not caching the full prefix causes unnecessary cache misses and higher input token costs.

Change

Remove the .slice(0, 2) limit so ALL system messages get cache markers:

const system = msgs.filter((msg) => msg.role === "system")

Impact

  • 1 line changed
  • Maximizes prefix cache reuse for Anthropic, Bedrock, OpenRouter, and other providers that support prompt caching
  • The system prompt is identical across turns within a session, so caching the full prefix is always beneficial
  • No functional change to model behavior

applyCaching() was slicing system messages to only the first 2, which
meant additional system messages (from plugins, MCP instructions, etc.)
were never marked with cacheControl. This caused unnecessary cache
misses on every turn since the full system prompt prefix was not being
cached.

Remove the .slice(0, 2) limit so all system messages get cache markers.
This maximizes prefix cache reuse for providers that support it
(Anthropic, Bedrock, OpenRouter, etc.).
@github-actions github-actions Bot added needs:compliance This means the issue will auto-close after 2 hours. needs:issue labels Jul 22, 2026
@github-actions

Copy link
Copy Markdown
Contributor

This PR doesn't fully meet our contributing guidelines and PR template.

What needs to be fixed:

  • PR description is missing required template sections. Please use the PR template.

Please edit this PR description to address the above within 2 hours, or it will be automatically closed.

If you believe this was flagged incorrectly, please let a maintainer know.

@github-actions

Copy link
Copy Markdown
Contributor

Thanks for your contribution!

This PR doesn't have a linked issue. All PRs must reference an existing issue.

Please:

  1. Open an issue describing the bug/feature (if one doesn't exist)
  2. Add Fixes #<number> or Closes #<number> to this PR description

See CONTRIBUTING.md for details.

@github-actions

Copy link
Copy Markdown
Contributor

The following comment was made by an LLM, it may be inaccurate:

Based on my search, I found a related PR that may be worth reviewing:

Related PR:

This PR also addresses prompt caching optimization in the provider. It may have already tackled similar caching improvements or could provide context on the caching strategy. Worth checking if it covers the same issue or complements this fix.

No other exact duplicates addressing the .slice(0, 2) system message limitation were found.

@github-actions

Copy link
Copy Markdown
Contributor

This pull request has been automatically closed because it was not updated to meet our contributing guidelines within the 2-hour window.

Feel free to open a new pull request that follows our guidelines.

@github-actions github-actions Bot removed the needs:compliance This means the issue will auto-close after 2 hours. label Jul 22, 2026
@github-actions github-actions Bot closed this Jul 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant