fix(eval): fail fast with one classified error when model access is refused - #4502
Conversation
|
You have reached your Codex usage limits for security reviews. Please try again later. |
|
Warning Review limit reachedNext included review available in 55 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (27)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
|
|
📦 Client bundle boundary
A server module in a client graph aborts hydration in the browser. New leaks fail CI; known leaks are tracked in |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 66c6bb80b5
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
You have reached your Codex usage limits for security reviews. Please try again later. |
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
|
You have reached your Codex usage limits for security reviews. Please try again later. |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6740ab768e
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
You have reached your Codex usage limits for security reviews. Please try again later. |
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f3c86cb6d2
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f3c86cb6d2
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Addresses review on #4502: - Built-in LLM judges rethrow classified model access denials instead of scoring them 0, so judge metrics stop the eval too. - The agent service eval adapter throws the classified denial for a 402 or RUN_ERROR with an account-wide credit code instead of resolving a failed record. - RESOURCE_LIMIT_EXCEEDED, agent run credit limits, and unrecognized 402s stay record failures: they can be request-scoped. - A ProviderError 402 counts only with the gateway problem body (`slug: insufficient-credits`) as provenance, so direct or BYOK provider 402s never get Veryfront billing advice. - The CLI keeps the missing billing group warning when any gateway request in the run got past admission, tracked on the Veryfront Cloud context. Refs veryfront/veryfront-issue-inbox#1444
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
|
You have reached your Codex usage limits for security reviews. Please try again later. |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3299f3e030
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
|
You have reached your Codex usage limits for security reviews. Please try again later. |
|
You have reached your Codex usage limits for security reviews. Please try again later. |
|
@codex review |
A custom provider endpoint can return code gateway_project_required with arbitrary error text, so the detail no longer copies the response's error field. Refs veryfront/veryfront-issue-inbox#1444
|
You have reached your Codex usage limits for security reviews. Please try again later. |
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 144ae44593
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
The spend limit clears with time or an administrator, not with credits, so AI_PROVIDER_SPEND_LIMIT_EXCEEDED now raises eval-model-spend-limit-exceeded with that guidance instead of the credit purchase advice. Refs veryfront/veryfront-issue-inbox#1444
|
@codex review |
|
You have reached your Codex usage limits for security reviews. Please try again later. |
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bbc7e87ded
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…reams A streaming agent service turns the gateway's gateway_project_required 400 into a RUN_ERROR, which dropped the code. The body now maps to the curated provider failure GATEWAY_PROJECT_REQUIRED (fixed wording, status 400), which survives the AG-UI boundary and hosted executor frames, and the eval adapter classifies that RUN_ERROR code as eval-project-required. Refs veryfront/veryfront-issue-inbox#1444
|
@codex review |
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
|
You have reached your Codex usage limits for security reviews. Please try again later. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e80dfb8e1b
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…rvice credit codes An AG-UI RUN_ERROR or a curated failure that crossed a runtime boundary keeps only the code, and the stream can derive INSUFFICIENT_CREDITS from a direct or BYOK provider failure. The agent service adapter now fails fast on billing only for the gateway's 402 problem body; code-only billing failures stay record failures. The project-required code, which comes only from the gateway's structured code, still fails fast. Refs veryfront/veryfront-issue-inbox#1444
|
@codex review |
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
|
You have reached your Codex usage limits for security reviews. Please try again later. |
|
Codex Review: Didn't find any major issues. Keep it up! Reviewed commit: ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
|
…tion #4502 now classifies gateway_project_required: the eval stops with EVAL_PROJECT_REQUIRED and parseProviderError maps it to GATEWAY_PROJECT_REQUIRED for streaming and hosted surfaces. Drop this branch's own provider-http message so the gateway rejection has one wording, and make EVAL_PROJECT_REQUIRED name the same supported remedy as the proactive warning instead of a "linked project directory". Cover the streaming path end to end: a gateway 400 built by buildProviderError parses to GATEWAY_PROJECT_REQUIRED directly and through lastError. Refs veryfront/veryfront-issue-inbox#1444
…tion #4502 now classifies gateway_project_required: the eval stops with EVAL_PROJECT_REQUIRED and parseProviderError maps it to GATEWAY_PROJECT_REQUIRED for streaming and hosted surfaces. Drop this branch's own provider-http message so the gateway rejection has one wording, and make EVAL_PROJECT_REQUIRED name the same supported remedy as the proactive warning instead of a "linked project directory". Cover the streaming path end to end: a gateway 400 built by buildProviderError parses to GATEWAY_PROJECT_REQUIRED directly and through lastError. Refs veryfront/veryfront-issue-inbox#1444
…tion #4502 now classifies gateway_project_required: the eval stops with EVAL_PROJECT_REQUIRED and parseProviderError maps it to GATEWAY_PROJECT_REQUIRED for streaming and hosted surfaces. Drop this branch's own provider-http message so the gateway rejection has one wording, and make EVAL_PROJECT_REQUIRED name the same supported remedy as the proactive warning instead of a "linked project directory". Cover the streaming path end to end: a gateway 400 built by buildProviderError parses to GATEWAY_PROJECT_REQUIRED directly and through lastError. Refs veryfront/veryfront-issue-inbox#1444
Part of https://github.com/veryfront/veryfront-issue-inbox/issues/1444
Root cause
veryfront evalgenerates a billing group id per run (evalrun_...,src/eval/run-report.ts) and sends it on each model request inx-veryfront-billing-group-id. After the run, the CLI callsPOST /ai/gateway/billing/finalizeto reconcile what the gateway recorded under that id.404 gateway_billing_group_not_foundwhen no request was recorded under the id (finalizeGatewayBillingGroupCredits,aggregate.requestCount === 0). That happened because gateway admission rejected every request with 402 before recording any usage (checkGatewayCredits->createInsufficientCreditsResponse, orcheckAiProviderSpendLimit). So the 404 is a side effect of the 402, not a provisioning gap.checkProjectCredits->resolveProjectOwner). Without a project, it checks the caller's own wallet. The 402 body carriesslug: insufficient-creditswithbalanceandrequired, and the CLI kept it on theProviderError(responseBody). The eval runner flattened it toProvider request failed with status 402, then ran the project'scheckon empty output.Change
src/eval/model-access.tsrecognizes typed 402 provider errors, including ones wrapped inlastError,causeorerrors, plus curated provider failures. Free-form error text never counts as a denial. The runner rejectsrunEval()with the neweval-model-access-deniedregistry error at the first refusal from a target or metric.gateway_billing_group_not_foundat debug level, not as a warning. Finalization still runs, because earlier requests may have been served.Eval check could not evaluate the failed target output: ....runEval()behavior change.Output now:
The error registry has no per-error docs URL, so the default
Docs:link stays and the suggestion names the billing settings page and the insufficient-credits docs.Validation (Deno 2.7.7)
deno task test:file src/eval: 20 files, 218 steps passed. New tests: classifier, runner fail-fast (adapter called once, check never called), judge metric refusal, check-after-target-failure wording, suite stop.deno task test:file cli/commands/eval/command.test.ts: 50 steps passed. Includes a realagent()whose model throws a gateway 402. That test confirmsagent.generatepasses the typed error through and the model is called once. It also checks there is no billing-group warning after a refusal, while other failures still warn.deno task test:file src/errors: 607 steps passed.tests/docs/error-docs-links.test.tspassed.deno check(eval, CLI main, touched tests),deno lint, anddeno fmt --checkon the touched files all pass.docs:errors:check,docs:api-reference:check,lint:anti-slop,lint:cli-boundary,lint:dependency-boundaries,lint:module-boundaries,lint:check-awaits,lint:test-semantic-dispositionsandlint:barrel-jsdocalso pass.No API or gateway change: the API behaved as designed. Whether this user's wallet, or the project owner's, should have had credits is a billing question and is tracked on the inbox issue.
Summary by CodeRabbit
New Features
Bug Fixes
Documentation