fix(provider): cache all system messages instead of only first 2 - #38206
fix(provider): cache all system messages instead of only first 2#38206zouhua021 wants to merge 1 commit into
Conversation
applyCaching() was slicing system messages to only the first 2, which meant additional system messages (from plugins, MCP instructions, etc.) were never marked with cacheControl. This caused unnecessary cache misses on every turn since the full system prompt prefix was not being cached. Remove the .slice(0, 2) limit so all system messages get cache markers. This maximizes prefix cache reuse for providers that support it (Anthropic, Bedrock, OpenRouter, etc.).
|
This PR doesn't fully meet our contributing guidelines and PR template. What needs to be fixed:
Please edit this PR description to address the above within 2 hours, or it will be automatically closed. If you believe this was flagged incorrectly, please let a maintainer know. |
|
Thanks for your contribution! This PR doesn't have a linked issue. All PRs must reference an existing issue. Please:
See CONTRIBUTING.md for details. |
|
The following comment was made by an LLM, it may be inaccurate: Based on my search, I found a related PR that may be worth reviewing: Related PR:
This PR also addresses prompt caching optimization in the provider. It may have already tackled similar caching improvements or could provide context on the caching strategy. Worth checking if it covers the same issue or complements this fix. No other exact duplicates addressing the |
|
This pull request has been automatically closed because it was not updated to meet our contributing guidelines within the 2-hour window. Feel free to open a new pull request that follows our guidelines. |
Problem
applyCaching()inprovider/transform.tsslices system messages to only the first 2:This means additional system messages (from plugins via
experimental.chat.system.transform, MCP instructions, AGENTS.md content, etc.) are never marked withcacheControl. Since the system prompt is the most stable prefix across turns, not caching the full prefix causes unnecessary cache misses and higher input token costs.Change
Remove the
.slice(0, 2)limit so ALL system messages get cache markers:Impact