[evals] Daily Evals Feature Report - 2026-08-11

[github-actions]

Executive Summary

Analyzed 36 evals-producing runs across 28 workflows over the last 7 days. The evals pipeline itself is stable with a 100.0% job success rate, but the aggregate YES rate is only 50.0%, so the feature is DEGRADED. 23 answers were emitted as UNKNOWN rather than binary YES/NO and were counted as non-passing for this report.

Note

Status: DEGRADED — evals artifacts are being produced consistently, but judge outcomes remain below the 60% HEALTHY threshold and include 23 UNKNOWN answers.

Key Metrics

Metric	Value
Workflows with evals	28
Runs analyzed	36
Runs with evals results	36
Evals job success rate	100.0%
Overall YES rate	50.0%

Per-Workflow Pass Rates

Workflow	Runs	Evals Job Success	Run Pass Rate	Lowest-Scoring Question
PR Sous Chef	6	100.0%	0.0%	"Does the agent output show a specific reason why the selected PR needs a nudge toward maintainer investigation?" (0.0% YES)
Issue Monster	3	100.0%	0.0%	"Does the agent output show that at most one issue was assigned to Copilot per run?" (0.0% YES)
Avenger	2	100.0%	0.0%	"Did the agent assess the current CI state and determine if intervention was needed? [2 UNKNOWN]" (0.0% YES)
[aw] Failure Investigator (6h)	1	100.0%	100.0%	"Did the agent investigate agentic workflow failures from the last 6 hours and produce findings?" (100.0% YES)
AI Moderator	1	100.0%	0.0%	"Did the agent apply at least one label (spam, ai-generated, link-spam, or ai-inspected) or call noop? [1 UNKNOWN]" (0.0% YES)
Auto-Triage Issues	1	100.0%	0.0%	"Did the agent label a non-report issue describing unexpected behavior, expected versus actual results, or a reproducible failure with bug and the relevant component label (or needs-triage if no component is clear)?" (0.0% YES)
CLI Version Checker	1	100.0%	100.0%	"Did the agent check for new versions of agentic CLI tools (Claude Code, GitHub Copilot CLI, Codex, MCP servers, etc.)?" (100.0% YES)
Code Scanning Fixer	1	100.0%	0.0%	"Did the agent analyze code scanning alerts and identify at least one fixable alert, or correctly skip when no fixable alerts were found?" (0.0% YES)
Contribution Check	1	100.0%	0.0%	"Did the agent dispatch PRs to the contribution-checker subagent and produce evaluation results for each PR? [1 UNKNOWN]" (0.0% YES)
Copilot Session Insights	1	100.0%	100.0%	"Was a report produced with usage patterns, success rates, and performance metrics?" (100.0% YES)
Daily AgentRx Trace Optimizer	1	100.0%	0.0%	"Does the agent output show that the objective for experiment sub_agent_strategy was successfully completed? [1 UNKNOWN]" (0.0% YES)
Daily Assign Issue To User	1	100.0%	0.0%	"Did the agent assign one or more unassigned issues to a user, or correctly call noop when no unassigned issues were found?" (0.0% YES)
Daily Cli Tools Tester	1	100.0%	0.0%	"Did the agent run exploratory tests on the audit, logs, and compile CLI tools?" (0.0% YES)
Daily Container Image Security Scan	1	100.0%	100.0%	"Did the agent analyze container images for vulnerabilities, updates, and rejected licenses?" (100.0% YES)
Daily GitHub Docs SEO Optimizer	1	100.0%	0.0%	"Did the agent report an actionable documentation recommendation or explain why no update was needed? [1 UNKNOWN]" (0.0% YES)
Daily Go Test Parallelizer	1	100.0%	100.0%	"Did the agent create a pull request for a safe test change, or use noop when no safe change was available?" (100.0% YES)
Daily Safe Outputs Conformance Checker	1	100.0%	100.0%	"Did the agent run a conformance check against the Safe Outputs specification implementation?" (100.0% YES)
Daily VulnHunter Scan	1	100.0%	100.0%	"Was a security issue created for verified exploitable findings, or was noop used when VulnHunter found nothing actionable?" (100.0% YES)
Daily Workflow Updater	1	100.0%	0.0%	"Did the agent check GitHub Actions versions for available updates?" (0.0% YES)
Design Decision Gate 🏗️	1	100.0%	0.0%	"Does the agent output confirm that it checked for existing ADRs before deciding on an action?" (0.0% YES)
ESLint Refiner	1	100.0%	100.0%	"Did the agent analyze ESLint diagnostics trends to identify rule refinement opportunities?" (100.0% YES)
Issue Arborist	1	100.0%	0.0%	"Did the agent analyze recent issues and identify related issue relationships? [1 UNKNOWN]" (0.0% YES)
Multi-Device Docs Tester	1	100.0%	100.0%	"Did the agent test the documentation site across the requested device form factors?" (100.0% YES)
PR Code Quality Reviewer	1	100.0%	0.0%	"Does the agent output show that the review findings are limited to changes in the pull request diff rather than unrelated code? [1 UNKNOWN]" (0.0% YES)
PR Triage Agent	1	100.0%	0.0%	"Does the agent output include a triage report summarizing the PRs processed? [1 UNKNOWN]" (0.0% YES)
Sub-Issue Closer	1	100.0%	100.0%	"Did the agent check parent issues for the completion status of all their sub-issues?" (100.0% YES)
Test Quality Sentinel	1	100.0%	0.0%	"Does the agent output show that the objective for experiment model_size was successfully completed? [1 UNKNOWN]" (0.0% YES)
Tidy	1	100.0%	100.0%	"Was a pull request created with formatting and tidying fixes, or was noop used when no changes were needed?" (100.0% YES)

Per-Question Breakdown per Workflow

PR Sous Chef

Question ID	Question	YES	NO	YES Rate
comment-added	Did the agent add a comment to at least one pull request?	5	1	83.3%
nudge-targeted	Does the agent output show a specific reason why the selected PR needs a nudge toward maintainer investigation?	0	6	0.0%
pr-evaluated	Does the agent output confirm that it evaluated at least one open PR for nudge eligibility?	6	0	100.0%

Issue Monster

Question ID	Question	YES	NO	YES Rate
issue_assigned	Did the agent assign at least one issue to the Copilot coding agent, or correctly skip when no suitable issues were found?	3	0	100.0%
single_issue_scoped	Does the agent output show that at most one issue was assigned to Copilot per run?	0	3	0.0%

Avenger

Question ID	Question	YES	NO	YES Rate
ci_state_assessed	Did the agent assess the current CI state and determine if intervention was needed?	0	2 (2 UNKNOWN)	0.0%
pr_created_or_skipped	Was a PR created with CI fixes, or was the run correctly skipped because CI was already passing?	0	2 (2 UNKNOWN)	0.0%

[aw] Failure Investigator (6h)

Question ID	Question	YES	NO	YES Rate
failures_investigated	Did the agent investigate agentic workflow failures from the last 6 hours and produce findings?	1	0	100.0%
issues_created_or_closed	Were fix sub-issues created for unresolved failures, or were resolved tracking issues closed?	1	0	100.0%

AI Moderator

Question ID	Question	YES	NO	YES Rate
action-taken	Did the agent apply at least one label (spam, ai-generated, link-spam, or ai-inspected) or call noop?	0	1 (1 UNKNOWN)	0.0%
no-unsupported-action	Does the agent output show that only allowed safe-output actions (add-labels, hide-comment, noop) were used?	0	1 (1 UNKNOWN)	0.0%
rationale-provided	Does the agent output include a rationale explaining why the label(s) were applied or why noop was called?	0	1 (1 UNKNOWN)	0.0%

Auto-Triage Issues

Question ID	Question	YES	NO	YES Rate
human-bug-labeled	Did the agent label a non-report issue describing unexpected behavior, expected versus actual results, or a reproducible failure with bug and the relevant component label (or needs-triage if no component is clear)?	0	1	0.0%
labels-applied	Did the agent apply at least one label to an unlabeled issue, or correctly call noop when no unlabeled issues were found?	0	1	0.0%
report-created	Was a summary discussion created listing the issues processed and the labels applied?	0	1	0.0%

CLI Version Checker

Question ID	Question	YES	NO	YES Rate
cli_versions_checked	Did the agent check for new versions of agentic CLI tools (Claude Code, GitHub Copilot CLI, Codex, MCP servers, etc.)?	1	0	100.0%
docker_images_checked	Did the agent check for new versions and digest changes of Docker images in pkg/cli/docker_images.go (actionlint, syft, grype, grant, zizmor, poutine, runner-guard, yamllint)?	1	0	100.0%
updates_applied_or_noop	Were version or digest updates applied and a PR created, or was noop used when all tools were already up to date?	1	0	100.0%

Code Scanning Fixer

Question ID	Question	YES	NO	YES Rate
alerts_analyzed	Did the agent analyze code scanning alerts and identify at least one fixable alert, or correctly skip when no fixable alerts were found?	0	1	0.0%
pr_created_or_noop	Was a pull request created with a remediation for a code scanning alert, or was noop used when no fixable alerts existed?	0	1 (1 UNKNOWN)	0.0%

Contribution Check

Question ID	Question	YES	NO	YES Rate
prs-evaluated	Did the agent dispatch PRs to the contribution-checker subagent and produce evaluation results for each PR?	0	1 (1 UNKNOWN)	0.0%
report-created	Was a report issue created summarizing PR compliance with the contributing guidelines?	0	1 (1 UNKNOWN)	0.0%

Copilot Session Insights

Question ID	Question	YES	NO	YES Rate
insights_report_produced	Was a report produced with usage patterns, success rates, and performance metrics?	1	0	100.0%
sessions_analyzed	Did the agent analyze GitHub Copilot coding agent sessions?	1	0	100.0%

Daily AgentRx Trace Optimizer

Question ID	Question	YES	NO	YES Rate
sub_agent_strategy_goal_met	Does the agent output show that the objective for experiment sub_agent_strategy was successfully completed?	0	1 (1 UNKNOWN)	0.0%

Daily Assign Issue To User

Question ID	Question	YES	NO	YES Rate
assignment-made	Did the agent assign one or more unassigned issues to a user, or correctly call noop when no unassigned issues were found?	0	1	0.0%
comment-posted	Did the agent post a comment explaining the assignment decision?	0	1 (1 UNKNOWN)	0.0%
contributor-selected	Does the agent output show that a relevant contributor was selected from recent merged PRs?	0	1 (1 UNKNOWN)	0.0%

Daily Cli Tools Tester

Question ID	Question	YES	NO	YES Rate
cli_tools_tested	Did the agent run exploratory tests on the audit, logs, and compile CLI tools?	0	1	0.0%
test_results_reported	Were the test results reported with any issues or anomalies identified?	1	0	100.0%

Daily Container Image Security Scan

Question ID	Question	YES	NO	YES Rate
container_images_scanned	Did the agent analyze container images for vulnerabilities, updates, and rejected licenses?	1	0	100.0%
findings_reported_or_noop	Did the agent report actionable image findings, or use noop when no findings required action?	1	0	100.0%

Daily GitHub Docs SEO Optimizer

Question ID	Question	YES	NO	YES Rate
recommendation_reported	Did the agent report an actionable documentation recommendation or explain why no update was needed?	0	1 (1 UNKNOWN)	0.0%
seo_opportunities_analyzed	Did the agent analyze GitHub Docs for opportunities to improve Agentic Workflows recommendations in Copilot CLI?	0	1 (1 UNKNOWN)	0.0%

Daily Go Test Parallelizer

Question ID	Question	YES	NO	YES Rate
pr_created_or_noop	Did the agent create a pull request for a safe test change, or use noop when no safe change was available?	1	0	100.0%
tests_analyzed	Did the agent analyze Go tests to identify a safe candidate for t.Parallel?	1	0	100.0%

Daily Safe Outputs Conformance Checker

Question ID	Question	YES	NO	YES Rate
conformance_checked	Did the agent run a conformance check against the Safe Outputs specification implementation?	1	0	100.0%
tasks_created_or_noop	Were agentic tasks created for Copilot to address critical/high/medium/low issues, or was noop used when implementation was compliant?	1	0	100.0%

Daily VulnHunter Scan

Question ID	Question	YES	NO	YES Rate
issue_created_or_noop	Was a security issue created for verified exploitable findings, or was noop used when VulnHunter found nothing actionable?	1	0	100.0%
scan_completed	Did the agent download the prepared VulnHunter bundle artifact, load the vulnhunt skill instructions, and scan the pre-ranked candidate files?	1	0	100.0%

Daily Workflow Updater

Question ID	Question	YES	NO	YES Rate
action_versions_checked	Did the agent check GitHub Actions versions for available updates?	0	1	0.0%
update_pr_created_or_noop	Did the agent create a pull request for required updates, or report that no changes were needed?	0	1 (1 UNKNOWN)	0.0%

Design Decision Gate 🏗️

Question ID	Question	YES	NO	YES Rate
action-taken	Did the agent add a PR comment, push a draft ADR, or call noop?	1	0	100.0%
adr-check-performed	Does the agent output confirm that it checked for existing ADRs before deciding on an action?	0	1	0.0%
decision-justified	Does the agent output explain why an ADR is required or why no ADR gate was triggered for this PR?	1	0	100.0%

ESLint Refiner

Question ID	Question	YES	NO	YES Rate
eslint_trends_analyzed	Did the agent analyze ESLint diagnostics trends to identify rule refinement opportunities?	1	0	100.0%
refinements_reported	Did the agent report actionable ESLint rule refinements or explain why no refinement was needed?	1	0	100.0%

Issue Arborist

Question ID	Question	YES	NO	YES Rate
issues_analyzed	Did the agent analyze recent issues and identify related issue relationships?	0	1 (1 UNKNOWN)	0.0%
sub_issues_linked_or_noop	Were related issues linked as sub-issues, or was noop used when no linkable relationships were found?	0	1 (1 UNKNOWN)	0.0%

Multi-Device Docs Tester

Question ID	Question	YES	NO	YES Rate
device_tests_completed	Did the agent test the documentation site across the requested device form factors?	1	0	100.0%
results_reported	Did the agent report the multi-device test results and any responsive design or functionality findings?	1	0	100.0%

PR Code Quality Reviewer

Question ID	Question	YES	NO	YES Rate
findings_scoped	Does the agent output show that the review findings are limited to changes in the pull request diff rather than unrelated code?	0	1 (1 UNKNOWN)	0.0%
review_posted	Did the agent post a code review comment on the pull request?	0	1 (1 UNKNOWN)	0.0%

PR Triage Agent

Question ID	Question	YES	NO	YES Rate
labels-applied	Did the agent apply triage labels to at least one pull request?	1	0	100.0%
report-produced	Does the agent output include a triage report summarizing the PRs processed?	0	1 (1 UNKNOWN)	0.0%
triage-data-set	Does the agent output confirm that category, risk, and action data were determined for each processed PR?	0	1 (1 UNKNOWN)	0.0%

Sub-Issue Closer

Question ID	Question	YES	NO	YES Rate
issues_checked	Did the agent check parent issues for the completion status of all their sub-issues?	1	0	100.0%
issues_closed_or_noop	Were completed parent issues closed with a comment, or does the agent output confirm no issues were ready to close?	1	0	100.0%

Test Quality Sentinel

Question ID	Question	YES	NO	YES Rate
model_size_goal_met	Does the agent output show that the objective for experiment model_size was successfully completed?	0	1 (1 UNKNOWN)	0.0%

Tidy

Question ID	Question	YES	NO	YES Rate
pr_created_or_noop	Was a pull request created with formatting and tidying fixes, or was noop used when no changes were needed?	1	0	100.0%
tidy_completed	Did the agent run code formatting and tidying tools on the codebase?	1	0	100.0%

Quality Signals

• nudge-targeted is the weakest current question at 0.0% YES (0 YES / 6 NO) in PR Sous Chef.
• single_issue_scoped is the weakest current question at 0.0% YES (0 YES / 3 NO) in Issue Monster.
• ci_state_assessed is the weakest current question at 0.0% YES (0 YES / 2 NO; 2 UNKNOWN) in Avenger.

Recommendations

• Tighten or simplify the judge questions in PR Sous Chef, Issue Monster, and Avenger; those workflows account for the highest-volume zero-YES questions in the 7-day window.
• Investigate why 23 answers were emitted as UNKNOWN in otherwise successful evals jobs. That usually points to missing agent context, ambiguous question wording, or prompts that do not surface enough observable evidence for the judge.
• Keep monitoring infrastructure separately from quality. The evals job path is currently healthy, so iteration effort should focus on workflow prompts and BinEval question design rather than push_evals_state reliability.

References

§31469629212 §31468615174 §31468388964

Generated by 🧪 Daily Evals Feature Report · gpt54 · 156.2 AIC · ⌖ 10.8 AIC · ⊞ 12K · ◷

• expires on Aug 17, 2026, 11:58 PM UTC-08:00

[github-actions]

This issue is being closed as outdated. A newer issue has been created: #52258

View newer issue

---

This action was performed automatically by the Daily Evals Feature Report workflow.