feat(eval,cli): print why a gated metric failed, not just that it did

[kwakayama]

A failing metric line said that something failed, never why:

  ● LLM as a judge passed: 0/1 passed (0%)

Everything needed to act on that already existed in summary.json, unprinted:

The answer correctly states the $15.21 tip and $99.71 total, but it does not split the total into $33.24, $33.24, and $33.23; it incorrectly claims $33.24 each.

For a judge rubric the verdict is the reason the eval failed. Reading it should not mean opening a JSON artifact.

After

  Eval:   Assistant smoke test
  Target: agent:assistant
  Result: 0/1 passed (0%)

  ● Agent called tool "calculator": 1/1 passed (100%)
  ● Agent had no failed tool calls: 1/1 passed (100%)
  ● LLM as a judge passed: 0/1 passed (0%)
      calculator: The answer correctly states the $15.21 tip and $99.71 total, but ...

Explanations print under the metric that produced them, three per metric, then and N more, see the report. The cap keeps a large dataset from burying the summary; the pointer says where the rest live.

Which record errors are suppressed

summary.gateFailures also carries a record.error entry whenever a record did not complete. Printing it unconditionally would restate the judge verdict directly above it as a second, independent problem.

The rule keys on the explanation, not the severity:

if (
  failure.name === "record.error" &&
  failure.explanation === RECORD_INCOMPLETE_EXPLANATION &&
  derived.has(failure.recordId)
) continue;

• isBlockingFailure (src/eval/runner.ts:232) clears completed for both gate and budget severities, so either one yields the synthesized Record did not complete. stand-in. Both are suppressed when another failure already explains the record.
• A record.error carrying text of its own — an adapter that threw, say — always prints. It is the only line explaining why the agent never answered, and nothing else reports it.

The stand-in string was written out at both sites that build it (report.ts, run-report.ts). It is now an exported RECORD_INCOMPLETE_EXPLANATION, so the suppression rule cannot drift from the text it matches.

Review follow-ups

• Bounded suppression to the stand-in explanation (#discussion_r3736104367). CodeRabbit proposed filtering derived to severity === "gate". I did not take that: budget failures clear completed too, so it would have reintroduced the duplicate line for every budget metric. It did surface a real bug next door — a thrown adapter error being swallowed — which is what the explanation check fixes.
• Env restore in tests (#discussion_r3736104363) — skipped with reasoning. Ten tests in this file share the pattern; changing one would make the file inconsistent. Offered as a separate cleanup.

Testing

• src/eval/, cli/commands/eval/, src/extensions/eval/, extensions/ext-eval-report-mlflow/: 21 passed, 255 steps, 0 failed
• deno lint, deno fmt --check (58 files): clean
• deno check on every touched file: clean

Both new tests were verified failing against the previous code first:

• prints why a gated metric failed, without restating it as a record error
• keeps a record error that carries detail of its own — drives a rejecting adapter plus a failing gate, asserts the upstream message survives

Not from this branch: deno check src/eval/judges.test.ts reports three TS18048 'call' is possibly 'undefined' errors. They reproduce unchanged on origin/main.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added explanations for gated evaluation metric failures in CLI output.
  • Grouped failure reasons by metric and summarized additional reasons when more than three apply.
  • Preserved independent upstream errors alongside gate failures.

• Bug Fixes

  • Removed redundant incomplete-record messages when a more specific metric explanation is available.
  • Standardized incomplete-record explanations across evaluation and report outputs.

• Documentation

  • Updated evaluation API reference links and documented the shared incomplete-record explanation.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Repo admins can enable using credits for code reviews in their settings.

[coderabbitai]

Warning

Review limit reached

@kwakayama, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 32 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 1f482e42-ef02-4efb-9f5f-ee3597fb9418

📥 Commits

Reviewing files that changed from the base of the PR and between 5c5b51b and 59325e3.

📒 Files selected for processing (2)

• cli/commands/eval/command.ts
• docs/api-reference/veryfront/eval.md

📝 Walkthrough

Walkthrough

The eval CLI now groups gate-failure explanations by metric, suppresses duplicate incomplete-record errors, limits displayed reasons, and prints unattached failures. A shared explanation constant is used across reports and JUnit output. Integration coverage verifies the output.

Changes

Eval gate output

Layer / File(s)	Summary
Incomplete-record explanation contract src/eval/report.ts, src/eval/index.ts, src/eval/run-report.ts, docs/api-reference/veryfront/eval.md	Defines and exports RECORD_INCOMPLETE_EXPLANATION. Eval reports and JUnit output use the constant. The API reference lists the export and corrected source links.
Failure reason rendering and regression coverage cli/commands/eval/command.ts, cli/commands/eval/command.test.ts	The CLI groups gate failures by metric, separates unattached failures, suppresses redundant incomplete-record errors, limits output to three reasons with an overflow message, and prints metric explanations. Integration tests validate gated metric failures and upstream agent errors.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant EvalReport
  participant EvalCLI
  participant CLIOutput
  EvalReport->>EvalCLI: Provide gate-failure summary
  EvalCLI->>EvalCLI: Group reasons and suppress redundant incomplete-record errors
  EvalCLI->>CLIOutput: Print metric and unattached failure explanations

Loading

Possibly related PRs

• veryfront/veryfront-code#3453: Both PRs modify eval CLI reporting and related output tests.

Suggested reviewers: kojiwakayama

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 20.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: printing reasons for gated metric failures in the eval CLI.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/eval-print-failure-reasons

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@cli/commands/eval/command.test.ts`:
- Around line 1268-1304: Update the test around runEvalCommand to save the
original values of VERYFRONT_API_TOKEN, VERYFRONT_PROJECT_SLUG,
VERYFRONT_EVAL_EXPORT, VERYFRONT_EVAL_EXPORTERS, and XDG_CONFIG_HOME before
modifying them, then restore each value in the finally block, deleting only
variables that were originally unset. Keep the existing temporary-directory
cleanup intact.

In `@cli/commands/eval/command.ts`:
- Around line 856-860: Update the derived duplicate-record calculation near
const derived so it includes only failures whose severity is "gate", while
retaining the exclusion of "record.error". Ensure the suppression logic at line
866 therefore removes record.error only when the record also has a gate failure,
and add a regression case covering a budget failure alongside an independent
record error.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 3b27b18e-21d3-4d37-b32a-95c23d2f161a

📥 Commits

Reviewing files that changed from the base of the PR and between 9acfc01 and dda8c29.

📒 Files selected for processing (2)

• cli/commands/eval/command.test.ts
• cli/commands/eval/command.ts