fix(middleware): budget aliasing, cache key collisions, retry panics, and gateway errors

[hallelx2]

Stacked on #9, which is stacked on #8. Merge in order; each needs retargeting to main as the one below lands.

Finishes M1. Four defects that all produce plausible-looking wrong behaviour rather than an error.

HAL-526 — budget.New aliased every client it wrapped

shared := &budgetClient{...}
return func(inner llmgate.Client) llmgate.Client {
    shared.inner = inner   // overwritten on every application
    return shared
}

Wrapping two clients returned the same object, pointing at whichever was wrapped last. A router composed the way the README recommends would send every request to its fallback and never call the primary — silently. The write also raced the read in Complete.

Counters now live in a shared Ledger; each application gets its own wrapper. Ledger.Spent() / Remaining() make spend observable, which the dashboard and control plane both need and which previously had no API at all.

Config.RefuseUnpriced closes the related hole: an unpriced call debits zero, so a budget over unpriced models never trips however much is really spent.

HAL-527 — cache keys collided inside tool loops

The key omitted ToolChoice, Message.ToolCalls and Message.ToolCallID. In a tool loop the assistant turn requesting a call usually has empty Content — the calls are the payload — so two different loop states hashed identically and the second request got the first's answer. ToolChoice: "required" and "none" collided for the same reason.

Responses were also only shallow-copied, leaving ToolCalls and their Input bytes aliased to the cached entry — despite the comment on that line claiming it was a defensive copy. A caller unmarshalling and rewriting arguments in place corrupted every later hit.

Key is versioned (v2) so old entries miss rather than collide.

HAL-536 — two reachable panics in sleepBackoff

d := base << attempt          // overflows int64 ~attempt 35, goes negative
if d > max { d = max }        // negative is not > max, so no clamp
jitter := rand.Int63n(int64(d / 2))   // panics on a non-positive bound

Reachable with MaxRetries: 40. Separately, a BaseDelay under 2ns makes d/2 == 0 and panics identically — exactly what someone writing a fast test would set.

Backoff now saturates instead of shifting blind, and the jitter bound is guarded. It also honours Retry-After when present (via a new LLMError.RetryAfterDur and llmgate.RetryAfter), clamped to MaxDelay so a mistaken or hostile header cannot park the caller for an hour.

HAL-528 — gateway 200-with-error-body was retried 4×

Compatible gateways answer HTTP 200 with an error envelope. z.ai returns {"code":500,"msg":"404 NOT_FOUND"} when the base URL lacks its version segment — which reaches the adapter as a successful call with zero choices, became a bare "empty response", classified Unknown, and the retry middleware fails open on Unknown. Four paid attempts on a permanent configuration fault, then an error that doesn't say what's wrong.

New ErrClassGateway (non-retryable), and the message now names the endpoint and model, with an explicit hint when the base URL has no version segment. The nine-line explanatory comment in vectorless-engine/pkg/config/config.go is the evidence this was worth fixing.

HAL-546 — docs drift

README linked a DESIGN.md that doesn't exist and pointed the roadmap at another repo; ROADMAP still described tool calling as unimplemented three commits after it shipped.

Verification

• go build, go vet, golangci-lint, staticcheck — clean
• Full suite run 3× with -shuffle=on, zero failures

Shuffle earned its keep here: it caught an order dependency I had introduced in #9 — TestRegisterBeatsNormalization left a global price override in place, so it could poison TestLookupResolvesVariants depending on order. Fixed on that branch (29eb794) rather than papered over here.

• -race still not runnable locally (no gcc); CI covers it

Closes HAL-526
Closes HAL-527
Closes HAL-528
Closes HAL-536
Closes HAL-546

Summary by Sourcery

Fix middleware and error handling defects affecting budgeting, caching, retries, and gateway error classification, and align roadmap/documentation with current tool-calling support.

New Features:

• Expose budget spend via a shared Ledger with Spent and Remaining APIs so multiple clients can draw down a single observable budget.
• Add configuration to refuse unpriced models so spending caps cannot be silently bypassed.
• Support provider Retry-After hints in retry backoff, clamped by MaxDelay, and introduce a non-retryable gateway error class for 200-with-error-body responses.

Bug Fixes:

• Ensure budget middleware wraps each client separately and shares counters without aliasing, eliminating routing errors and data races.
• Fix cache key collisions by including tool-loop state, tool choice, and related fields and deep-copy cached responses to avoid caller mutations corrupting cache entries.
• Prevent retry backoff panics under extreme MaxRetries and BaseDelay configurations by guarding exponential backoff and jitter calculations.
• Classify empty, choice-less responses from compatible gateways as gateway errors, naming the endpoint and model and avoiding pointless retries.

Enhancements:

• Refactor budget tracking into a reusable Ledger type and add helper constructors for shared-budget setups.
• Improve adapter error messaging for misconfigured gateways, including endpoint naming and version-segment hints.
• Refine retry middleware to compute backoff via dedicated helpers for exponential delay and jitter management.

Documentation:

• Update README and ROADMAP to reflect shipped tool calling support, current streaming status, and modern pricing update strategy.

Tests:

• Add regression and concurrency tests for budget middleware aliasing, shared ledgers, observability, and unpriced-model handling.
• Add cache middleware tests covering tool-related cache keys, sampling presence, and deep-copy behaviour of cached responses.
• Add retry middleware tests for extreme backoff configurations, Retry-After honouring and clamping, and non-retry of gateway errors.
• Add adapter tests to verify gateway empty responses are classified correctly, name endpoints and models, and emit version-segment hints.

[sourcery-ai]

Sorry @hallelx2, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

[coderabbitai]

Warning

Review limit reached

@hallelx2, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 55 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 0413baa0-6584-4837-a268-a371d9ceeb2d

📥 Commits

Reviewing files that changed from the base of the PR and between af5a9ec and 36bbaf4.

📒 Files selected for processing (13)

• README.md
• ROADMAP.md
• errors.go
• internal/adapter/adapter.go
• internal/adapter/adapter_gateway_test.go
• middleware/budget/budget.go
• middleware/budget/budget_shared_test.go
• middleware/cache/cache.go
• middleware/cache/cache_toolkey_test.go
• middleware/retry/retry.go
• middleware/retry/retry_backoff_test.go
• provider/anthropic/anthropic.go
• provider/openai/openai.go

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[sourcery-ai]

Reviewer's Guide

Refactors budget, cache, retry, and adapter middleware to fix subtle correctness bugs (client aliasing, cache key collisions and aliasing, retry backoff panics, and gateway 200-with-error-body handling), introduces observability and configurability for budgets, honors provider Retry-After, and refreshes documentation to match shipped tool support.

Sequence diagram for gateway empty-response handling and non-retryable classification

sequenceDiagram
actor Caller
participant Adapter
participant Provider as llmgate.Provider
participant RetryMiddleware as retryClient
participant llmgate

Caller->>RetryMiddleware: Complete(ctx, req)
RetryMiddleware->>Adapter: Complete(ctx, req)
Adapter->>Provider: Complete(ctx, req)
Provider-->>Adapter: resp (Choices may be empty)
alt len(resp.Choices) == 0
  Adapter->>Adapter: emptyResponseErr(req)
  Adapter-->>RetryMiddleware: *llmgate.LLMError{Class: ErrClassGateway, Message: ..., Provider, RetryAfterDur}
  RetryMiddleware->>llmgate: Classify(err)
  llmgate-->>RetryMiddleware: ErrClassGateway
  RetryMiddleware->>RetryMiddleware: defaultRetryIf(err) == false
  RetryMiddleware-->>Caller: error (no retries)
else choices present
  Adapter-->>RetryMiddleware: resp
  RetryMiddleware-->>Caller: resp
end

Loading

File-Level Changes

Change	Details	Files
Budget middleware now uses a shared Ledger with per-client wrappers, exposes spend/remaining APIs, and can optionally refuse unpriced models to avoid silently unlimited budgets.	Replaces single shared budgetClient instance with NewLedger + per-Client wrappers and adds NewWithLedger helper. Introduces Ledger struct with mutex-protected daily/total counters, rollover logic, and Spent/Remaining methods. Adds RefuseUnpriced flag to Config and ErrUnpriced; Complete checks Usage.Priced before debiting and can fail calls whose cost cannot be enforced. Implements remaining() helper that returns +Inf for unset caps and zero when overspent, and exposes Ledger() from budgetClient.	middleware/budget/budget.go middleware/budget/budget_shared_test.go
Cache middleware’s key derivation and response handling are hardened to avoid tool-loop collisions and aliasing of cached responses.	Versions cache keys via a keyVersion salt and expands cacheKey to hash ToolCalls, ToolCallID, and ToolChoice in addition to existing fields. Adds cloneResponse to deep-copy ToolCalls and their Input bytes for both cache storage and cache hits, eliminating shared mutable state. Updates Complete to use cloneResponse for cached hits and stored entries while returning the original response to the caller. Adds tests to ensure ToolChoice, assistant ToolCalls, ToolCallID, and sampling presence affect the key, and that cached responses are deep-copied.	middleware/cache/cache.go middleware/cache/cache_toolkey_test.go
Retry middleware’s backoff logic is made overflow-safe, jitter is guarded, provider Retry-After is honored and clamped, and gateway errors are classified as non-retryable.	Changes sleepBackoff to take the error, uses backoffFor + expBackoff to compute delays without shifting into int64 overflow, and adds jitterFor with a non-positive-bound guard. Integrates llmgate.RetryAfter so provider-supplied retry delays override exponential backoff but are clamped to MaxDelay. Extends defaultRetryIf to treat ErrClassGateway as non-retryable. Adds tests that exercise extreme MaxRetries/BaseDelay configs for panics, verify Retry-After is honored and clamped, and assert gateway-class errors are not retried.	middleware/retry/retry.go middleware/retry/retry_backoff_test.go errors.go
Adapter and provider wiring now recognize empty, zero-choice responses as gateway errors, enrich error messages with endpoint/model context and hints, and propagate base URLs from providers.	Adapter gains a baseURL field and SetBaseURL; provider constructors (Anthropic/OpenAI) set it from config.BaseURL. Complete now uses emptyResponseErr when Choices is empty, constructing an LLMError classified as ErrClassGateway that names the model and base URL. Introduces hasVersionSegment and versionSegment regex to detect missing API version segments and append a version-hint to the error message when appropriate. Adds tests that verify empty responses are classified as gateway errors, that errors mention the model and endpoint, that the version-segment hint fires only when expected, and that hasVersionSegment behaves correctly.	internal/adapter/adapter.go internal/adapter/adapter_gateway_test.go provider/anthropic/anthropic.go provider/openai/openai.go errors.go
Documentation is updated to reflect shipped tool use, roadmap location, and pricing refresh capabilities, removing stale or misleading references.	Updates ROADMAP phase descriptions to split tool use and streaming, describe shipped tool calling, and document pricing.UseRemote and future embedded-table refresh. Adjusts README status, "coming next" section, and removes non-existent DESIGN.md reference, pointing directly to ROADMAP.md. Clarifies streaming remains deferred while tool use is implemented across providers.	ROADMAP.md README.md

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.