[Outcome Report] Outcome Report — 2026-05-26: 54.5% acceptance rate, 🔴 regression from 100%

[github-actions]

Outcome Scorecard — 2026-05-26

Metric	Value	Status
Acceptance rate	54.5%	🔴 <60%
Zero-touch rate	0%	🔴 0%
Waste rate	20%	🟡 10-25%
Median time to resolution	343 sec (5.7 min)	—
Accepted	6 / 25	—
Rejected	5	—
Pending	14	—
Runs checked	12	—

🔴 Action Items

1. Chaos PR Bundle Fuzzer — Critical — 50% rejection rate (5 rejected, 5 pending). This workflow's output quality needs immediate review. Recommend auditing the prompt and testing with smaller sample before re-enabling at scale.

2. Stuck pending items — 3 items pending >12 hours:

   • PR [instructions] Sync inline skills instructions with v0.74.8 #34905 (Instructions Janitor) — pending 14,512 sec
   • PR [chaos-test] [chaos/r66] mutation-mapper: minor-rename+two-commits #34907-34911 (Chaos PR Bundle Fuzzer) — pending 14,340-14,356 sec
   • Comments on PR feat(setup): add Copilot CLI toolcache resolver with scoped release-mode opt-in #34918 (Smoke CI) — pending 3,352 sec
   • Add labels on issue (AI Moderator) — pending 13,459 sec (no URL)

   Recommend setting timeout policies for pending outcomes.

3. Missing URLs — 2 pending items with no URL (Contribution Check, AI Moderator). These may be silently failing. Investigate error handling.

4. Zero-touch rate at 0% — All 6 accepted items required human review/edits. This suggests the workflows are not generating production-ready output. Consider whether these should be rejected instead.

Per-Workflow Breakdown

Workflow	Accepted	Rejected	Pending	Acceptance	Waste Rate
Chaos PR Bundle Fuzzer	0	5	5	0%	50%
Contribution Check	0	0	3	0%	0%
Smoke CI	0	0	4	0%	0%
Instructions Janitor	0	0	1	0%	0%
AI Moderator	0	0	1	0%	0%
Agent Performance Analyzer	2	0	0	100%	0%
Claude Code User Documentation Review	1	0	0	100%	0%
Terminal Stylist	1	0	0	100%	0%
Workflow Normalizer	1	0	0	100%	0%
Dev	1	0	0	100%	0%

Trend Analysis

Comparing to 2026-05-25:

Metric	2026-05-25	2026-05-26	Change
Acceptance Rate	100%	54.5%	⬇️ -45.5pp
Zero-touch Rate	0%	0%	➡️ Stable
Waste Rate	0%	20%	⬇️ +20pp
Runs Checked	14	12	-2 runs
Total Outcomes	29	25	-4 outcomes

⬇️ Regressing — Acceptance rate dropped 45.5 percentage points, primarily due to Chaos PR Bundle Fuzzer's 50% rejection rate. This is the single largest factor driving today's degradation.

Rejected Items

All 5 rejections came from Chaos PR Bundle Fuzzer (run #26451165321):

• PR [chaos-test] phase-shifter: amend+two-commits (run 67) #34922 — closed (16 changed, 628 additions, 560 deletions)
• PR [chaos-test] code-cartographer: minor-rename+staged-subset (run 67) #34923 — closed (2 changed, 5 additions)
• PR [chaos-test] eol-sculptor: line-ending-variant+amend (run 67) #34924 — closed (1 changed, 3 additions)
• PR [chaos-test] branch-alchemist: two-commits+staged-subset (run 67) #34925 — closed (2 changed, 5 additions)
• PR [chaos-test] graph-gardener: octopus-merge (run 67) #34926 — closed (16 changed, 628 additions, 560 deletions)

All were auto-closed by the system, suggesting policy violations or test failures.

Reaction Summary & Recommendations

No positive or negative reactions recorded on any outcomes. This may indicate:

• Items not yet reviewed by team
• Reactions not being captured in evaluation
• Low visibility of the outcomes

Recommendations

1. Immediate: Pause or reduce Chaos PR Bundle Fuzzer runs pending prompt audit
2. Short-term: Add URL validation and error handling for missing-URL outcomes (Contribution Check, AI Moderator)
3. Medium-term: Investigate why zero-touch rate is 0% — consider whether accepting-but-edited items should be classified differently
4. Ongoing: Set TTL policies for pending outcomes to flag stuck items automatically

📊 Measured by Outcome Collector · haiku45 39K

• expires on Jun 2, 2026, 1:58 PM UTC

[github-actions]

Outcome Scorecard — 2026-05-26

Metric	Value	Status
Acceptance rate	17.9%	🔴 <60%
Zero-touch rate	0%	🔴 <25%
Waste rate	0%	🟢 <10%
Median time to resolution	—	—
Accepted	5 / 28	—
Rejected	0	—
Zero-touch	0 / 5	—
Pending	23	—
Runs checked	13	—

🔴 Action Items

1. High pending volume (23/28 = 82% of all outcomes) — The majority of safe-output items are stuck pending. These items need investigation to determine:

   • Are they awaiting human review that has not happened yet?
   • Is there a workflow timeout or retry mechanism needed?
   • Do any pending items exceed 48 hours? (Currently most items pending ~1.1+ hours)

2. Zero-touch rate is 0% — None of the 5 accepted items were created without requiring human edits or verification. This suggests:

   • Agent outputs are complete but may not match exact user expectations
   • Safe-output configurations may need refinement
   • Consider reviewing the most recent accepted item to understand quality gap

3. Low acceptance rate (18%) — 82% of outcomes are pending rather than accepted/rejected. This is not a quality signal yet (outcomes have not been evaluated), but when they are reviewed, watch for:

   • Whether pending items eventually move to accepted (indicates workflows are functioning)
   • Whether any pending items are actually invalid and should be rejected

Per-Workflow Breakdown

Workflow	Type	Accepted	Pending	Rejected	Total	Acceptance	Zero-touch
Daily Testify Uber Super Expert	Discussion	1	0	0	1	100%	—
Daily Secrets Analysis Agent	Discussion	1	0	0	1	100%	—
Daily Code Metrics and Trend Tracking Agent	Discussion	1	0	0	1	100%	—
Daily Cache Strategy Analyzer	Discussion	1	0	0	1	100%	—
Copilot Agent PR Analysis	Discussion	1	0	0	1	100%	—
Test Quality Sentinel	Comments	0	2	0	2	0%	—
PR Code Quality Reviewer	Reviews	0	5	0	5	0%	—
PR Sous Chef	Comments	0	7	0	7	0%	—
Matt Pocock Skills Reviewer	Reviews	0	7	0	7	0%	—
Linter Miner	PR	0	1	0	1	0%	—
Auto-Triage Issues	Labels	0	1	0	1	0%	—

Observations:

• Discussion-creating workflows (5 total): All 5 accepted. Output: GitHub discussions created successfully. No pending.
• PR comment/review workflows (7 total pending): Matt Pocock Skills Reviewer and PR Code Quality Reviewer are awaiting safe-output creation. These may be failing to submit due to missing URLs or authentication.
• PR operation workflows (3 total pending): Linter Miner (PR creation), Test Quality Sentinel (comments), Auto-Triage Issues (labels) all pending.

Pending Items by Workflow (age descending)

High Priority (>1 hour pending):

• Linter Miner: create_pull_request — 1.2h old
• PR Code Quality Reviewer: create_pull_request_review_comment — 1.1h old (×2)
• PR Code Quality Reviewer: submit_pull_request_review — 1.1h old
• Matt Pocock Skills Reviewer: create_pull_request_review_comment — 1.1h old (×2)
• Matt Pocock Skills Reviewer: submit_pull_request_review — 1.1h old
• PR Sous Chef: 6 items (comments & updates) — 50-52 min old
• Test Quality Sentinel: 2 items (comment & review) — 55 min old
• Auto-Triage Issues: add_labels — 16 min old

Recommendation: Most pending items are under 2 hours old, which is expected for recent workflow runs. Monitor these to ensure they eventually transition to accepted or rejected status. If any remain pending >24 hours, investigate for stuck/failed safe-output submissions.

Detailed Outcome Summary

Runs Evaluated: 13 workflow runs

• Auto-Triage Issues (1 run): 1 pending
• Copilot Agent PR Analysis (1 run): 1 accepted
• Daily Cache Strategy Analyzer (1 run): 1 accepted
• Daily Code Metrics and Trend Tracking Agent (1 run): 1 accepted
• Daily Secrets Analysis Agent (1 run): 1 accepted
• Daily Testify Uber Super Expert (1 run): 1 accepted
• Linter Miner (1 run): 1 pending
• Matt Pocock Skills Reviewer (2 runs): 7 pending
• PR Code Quality Reviewer (2 runs): 5 pending
• PR Sous Chef (1 run): 7 pending
• Test Quality Sentinel (1 run): 2 pending

Summary

This report reflects outcomes from 13 workflow runs producing 28 safe-output items. Five discussion items were successfully created (100% acceptance). Twenty-three items are currently pending human review (PR comments, reviews, PR creation, label additions). No items were rejected so far.

Next Steps:

1. Monitor pending items for resolution (target: move to accepted by next report)
2. Review why PR operation workflows have 0% acceptance (vs 100% for discussion workflows)
3. Verify safe-output credentials for PR comment/review submissions
4. Check if any pending items >48 hours require intervention

📊 Measured by Outcome Collector · haiku45 40.9K

• expires on Jun 2, 2026, 7:39 PM UTC

[github-actions]

This issue is being closed as outdated. A newer issue has been created: #35099

View newer issue

---

This action was performed automatically by the Outcome Collector workflow.