Skip to content

fix(llm): keep the tail cache breakpoint off reasoning-only parts - #52023

Open
argszero wants to merge 1 commit into
anomalyco:devfrom
argszero:cache-breakpoint-reasoning
Open

argszero wants to merge 1 commit into
anomalyco:devfrom
argszero:cache-breakpoint-reasoning

Conversation

@argszero

Copy link
Copy Markdown

Issue for this PR

Closes #51141

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

applyCachePolicy picks which part carries a cache breakpoint. When the targeted message has no text part it fell back to content.length - 1, and for an assistant turn that ends in reasoning that position is the thinking block.

ReasoningPart declares no cache field, and Message is a schema class that rebuilds each part, so a hint written there is dropped on construction: the breakpoint the policy asked for is silently never delivered. That is also the part Anthropic refuses with thinking.cache_control: Extra inputs are not permitted, which is what this issue ran into.

markMessageAt now walks back from the requested position to the last part that can hold a hint (text, tool-call, tool-result) instead of taking the last part whatever it is. When the message itself has no such part it keeps going into the preceding messages, so the policy still gets its breakpoint — one boundary earlier rather than not at all. I checked I could not reproduce the reported 400 on this branch, so this is the picker defect rather than a claim about the exact request the reporter sent.

How did you verify your code works?

Three tests in packages/llm/test/cache-policy.test.ts, all failing before the change and passing after: an assistant turn ending in reasoning steps back, { tail: 1 } steps back past a reasoning-only turn, and a hint is never written onto a part that drops it. bun test in packages/llm is 301 pass / 30 skip / 0 fail, and tsgo --noEmit plus oxlint are clean.

Screenshots / recordings

Not a UI change.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

@github-actions

Copy link
Copy Markdown
Contributor

The following comment was made by an LLM, it may be inaccurate:

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cache breakpoint placed on a thinking block permanently 400s long Anthropic sessions (thinking.cache_control: Extra inputs are not permitted)

1 participant