[prompt-clustering] Prompt Clustering Analysis - 2026-10-04

[github-actions[bot]]

Summary

Analysis Period: Last 30 days (2026-09-04 → 2026-10-04)
Total Tasks Analyzed: 592
Clusters Identified: 10
Overall Success Rate: 75.3% (446 merged / 592 total)

Full Analysis Report

General Insights

• Most Common Task Type: Cluster 8 — general workflow runtime fixes (133 tasks, 22.5% of total)
• Highest Success Rate: Cluster 3 — firewall/network egress logging (96.2% merged)
• Most Complex Tasks: Cluster 6 — usage artifacts & coverage reporting (avg. 98.5 files changed per PR, among tasks with available file data)
• Outliers: 1 outlier cluster identified (Cluster 5 — operational-value grader additions), with a 0% merge rate across 60 PRs
• Data limitation: Workflow run logs (gh-aw logs) were not available in this environment, so turn counts and cost/duration metrics could not be computed this run — findings below rely on PR metadata, comments, and reviews only. File/comment/review averages are computed only over the subset of PRs with cached full detail (coverage noted per cluster).

Cluster Analysis

Cluster 8: General workflow runtime fixes

• Size: 133 tasks (22.5% of total)
• Success Rate: 79.7%
• Top Keywords: workflow, runtime, coverage, fixes, workflows, copilot, add, agent
• Characteristics: Broad catch-all of runtime bug fixes and small feature additions across workflow execution
• Example PRs: Include WorkQ claims in the admission prompt #65192, Refactor coordinator smoke verification to JavaScript #65126, Instrument dispatch coordinator queue and preserve smoke failure reporting #65103, Use the gh.io/gh-aw short URL in docs homepage social cards #65095, Add TLC execution witnesses for dispatch coordinator ADR review #65075, Clarify explicit PR targets rejected by triggering-only push configuration #65065, Add self-hosted OpenObserve to the local devcontainer #65045, Implementing feature related to ADR-64955 #65028

---

Cluster 0: PR Sous Chef automation tasks

• Size: 84 tasks (14.2%)
• Success Rate: 89.3%
• Top Keywords: sous chef, pr sous, aic, run
• Characteristics: Tasks driven by the "PR Sous Chef" automation bot — generally high success rate
• Example PRs: Update lethal trifecta landing page copy #65115, Prevent base config restore from clobbering an external root checkout #65069, Fail add-comment when the triggering target cannot be resolved #65068, Handle transient stdin reads in the safe-output transport #65067, Clarify PR branch target errors in scheduled runs #65066

---

Cluster 3: Firewall / network egress logging

• Size: 78 tasks (13.2%)
• Success Rate: 96.2% (highest of all clusters)
• Top Keywords: firewall, network, details, summary, blocked, domain
• Characteristics: Firewall log summaries, blocked-domain reporting — narrowly scoped, well-understood changes with the best outcomes
• Example PRs: Normalize unified agent session metadata and AIC accounting #65344, Claude: default to strict MCP configuration #64970, Add Daily Ecosystem Explorer agentic workflow #64096, Prevent false blocked-domain warnings for successful firewall requests #63246, Filter logs to agentic workflows #63177

---

Cluster 5: Operational-value grader additions — ⚠️ OUTLIER

• Size: 60 tasks (10.1%)
• Success Rate: 0.0%
• Top Keywords: operational, operational value, evidence, study, metric
• Characteristics: Each PR adds a single operational-value grader/evaluator for a specific workflow. See root-cause section below.
• Example PRs: Fix operational-value grader created-at acquisition and preserve grader result contract in gh aw logs #60635, Record blocked operational-value study for daily-choice-test #58559, Add operational-value grader for daily-caveman-optimizer #58557, Add operational-value grader for cache strategy remediation #58555, Record blocked operational-value study for daily-byok-ollama-test #58554, Record blocked operational-value study for AWF spec surfacing #58553, Add operational-value grader for daily cross-repository compile checks #58552, Add operational-value grading for daily Markdown spellcheck #58550

---

Cluster 9: Safe-outputs & agent tool handling

• Size: 58 tasks (9.8%)
• Success Rate: 84.5%
• Top Keywords: safe, issue, output, failure, agent, tool
• Characteristics: Changes to the safe-outputs pipeline and agent tool wiring

---

Cluster 7: Engine/model integration (Codex, Copilot)

• Size: 56 tasks (9.5%)
• Success Rate: 82.1%
• Top Keywords: model, codex, copilot, workflow, cli, fixes

---

Cluster 2: gh-aw CLI & workflow tooling

• Size: 39 tasks (6.6%)
• Success Rate: 69.2%
• Top Keywords: aw, gh, gh aw, workflow, path, repository, cli
• Characteristics: Second-lowest success rate among non-outlier clusters; larger average file footprint (61 files changed)

---

Cluster 4: Logs & cache target handling

• Size: 32 tasks (5.4%)
• Success Rate: 78.1%
• Top Keywords: logs, target, cache, runs, workflow, preserve, json

---

Cluster 6: Usage artifacts & coverage reporting

• Size: 31 tasks (5.2%)
• Success Rate: 80.6%
• Top Keywords: usage, artifact, daily, coverage, detection, job
• Characteristics: Highest average files-changed footprint (98.5 files/PR) — largest, most complex changes in this period

---

Cluster 1: CI / GitHub Actions failure fixes

• Size: 21 tasks (3.5%)
• Success Rate: 85.7%
• Top Keywords: actions, job, fix, github, logs, failure

Success Rate by Cluster

Cluster	Theme	Tasks	Success Rate	Avg Comments	Avg Reviews	Avg Files Changed*
8	General workflow runtime fixes	133	79.7%	5.1	2.6	36.9
0	PR Sous Chef automation	84	89.3%	12.7	6.0	13.0
3	Firewall/network egress logging	78	96.2%	10.2	4.8	67.8
5	Operational-value grader (outlier)	60	0.0%	0.3	0.2	2.5
9	Safe-outputs & agent tool handling	58	84.5%	6.2	3.6	34.5
7	Engine/model integration	56	82.1%	3.4	1.7	24.3
2	gh-aw CLI & workflow tooling	39	69.2%	5.0	2.4	61.2
4	Logs & cache target handling	32	78.1%	6.9	3.9	11.1
6	Usage artifacts & coverage reporting	31	80.6%	6.4	3.3	98.5
1	CI/GitHub Actions failure fixes	21	85.7%	2.3	1.5	33.9

*Avg files changed is computed only over PRs with cached full detail available (coverage ranges 15–88 PRs per cluster; Cluster 5 has full 60/60 coverage).

Full Data Table

Sample PRs by cluster (3 per cluster, 592 total analyzed)

PR #	Title	Cluster	Outcome
#65115	Update lethal trifecta landing page copy	0 — PR Sous Chef automation	merged
#65069	Prevent base config restore from clobbering an external root checkout	0 — PR Sous Chef automation	merged
#65068	Fail add-comment when the triggering target cannot be resolved	0 — PR Sous Chef automation	merged
#65537	Publish AW issue clusters with actions/github-script	1 — CI/Actions failure fixes	open
#65493	Fix PR Sous Chef inference and setup-action cleanup failures	1 — CI/Actions failure fixes	merged
#64709	Consolidate ledger compaction into two Agentic Maintenance jobs	1 — CI/Actions failure fixes	merged
#65386	Route ESLint factory tasks through an Issues-backed queue	2 — gh-aw CLI & workflow tooling	closed
#65320	Include runtime versions in the unified agent session	2 — gh-aw CLI & workflow tooling	merged
#65106	Version and upgrade dispatch coordinator ledger messages	2 — gh-aw CLI & workflow tooling	merged
#65344	Normalize unified agent session metadata and AIC accounting	3 — Firewall/network egress logging	merged
#64970	Claude: default to strict MCP configuration	3 — Firewall/network egress logging	merged
#64096	Add Daily Ecosystem Explorer agentic workflow	3 — Firewall/network egress logging	merged
#65463	Use public permissions for extracted ZIP logs	4 — Logs & cache target handling	open
#65065	Clarify explicit PR targets rejected by triggering-only push	4 — Logs & cache target handling	closed
#64499	Document SEC-005 exemption for ledger pushes	4 — Logs & cache target handling	merged
#60635	Fix operational-value grader created-at acquisition and preserve result contract	5 — Operational-value grader (outlier)	closed
#58559	Record blocked operational-value study for daily-choice-test	5 — Operational-value grader (outlier)	closed
#58557	Add operational-value grader for daily-caveman-optimizer	5 — Operational-value grader (outlier)	closed
#65407	Prevent stack overflow in secret-redaction file scans	6 — Usage artifacts & coverage	merged
#65368	Skip private MCP cache during artifact redaction	6 — Usage artifacts & coverage	merged
#65323	Preserve detection AIC in unified session accounting	6 — Usage artifacts & coverage	closed
#65529	Recognize explicitly named regex-fragment constants	7 — Engine/model integration	open
#65406	Surface zero-token driver exits and repair Avenger and PR Triage	7 — Engine/model integration	merged
#65404	Resolve Codex model rejections in eight workflows	7 — Engine/model integration	closed
#65530	Fix timenowsub detection of parenthesized time.Now calls	8 — General workflow runtime fixes	merged
#65528	Allow PR Sous Chef to refresh externally authored PRs	8 — General workflow runtime fixes	merged
#65526	Fix PR Sous Chef's targeted eligibility refresh	8 — General workflow runtime fixes	merged
#65501	Avoid false safe-output rate-limit warnings	9 — Safe-outputs & agent tool handling	merged
#65494	Add selectable issue-backed work queue storage	9 — Safe-outputs & agent tool handling	open
#65492	Keep the essential-issue dashboard current when discussion edits	9 — Safe-outputs & agent tool handling	merged

Full per-PR cluster assignments (592 rows) are retained in this run's cached data (pr-clusters.jsonl) for deeper drill-down.

Key Findings

1. Operational-value graders are a systemic dead end in this pipeline: 60 PRs (10.1% of all copilot tasks this period) attempted to add a standalone operational-value grader/evaluator and none were merged — a 75.3-point gap below the overall merge rate, the single largest failure concentration in the dataset.
2. Scoped, single-purpose changes succeed best: Firewall/network logging (Cluster 3, 96.2%) and PR Sous Chef automation (Cluster 0, 89.3%) — both narrowly-scoped, template-following tasks — have the highest success rates.
3. Broad CLI/workflow-tooling changes are comparatively weaker: Cluster 2 (gh-aw CLI & workflow tooling, 69.2%) has the lowest success rate among non-outlier clusters, correlating with a large average files-changed footprint (61.2) — broader, more invasive changes appear to face more friction.

Lowest-Merge Cluster Root Cause

• Cluster: 5 — Operational-value grader additions
• Merge Rate Gap: 0.0% vs 75.3% overall (gap: 75.3 points)
• Primary Blocker: Scope/complexity mismatch — specifically, an automated operational-value contract gate rejecting grader-only submissions that modify an evaluator independent of its associated workflow (per this repo's AGENTS.md operational-value-contract rule, enforced via verify-operational-value-contract-change.sh)
• Confidence: Medium — no CI check-run data was available in the cached dataset to directly confirm the gate firing, but the behavioral signature is very strong and consistent across all 60 PRs
• Evidence PRs: Add operational-value grading for Duplicate Code Detector #58452, Record operational value study for archivx-agentic-workflows-analyzer #58501, Add operational-value study record for copilot-pr-prompt-analysis #58540, Fix operational-value grader created-at acquisition and preserve grader result contract in gh aw logs #60635

Supporting evidence (all 60 PRs in this cluster, full comment/review data available):

• 54/60 (90%) PRs have zero comments and zero reviews — no human reviewer ever engaged
• Only 1/60 PRs received an actual code review
• Median time from PR creation to close was 15 minutes; 53/60 (88%) closed within 1 hour, and all 60 closed within 4 hours
• Each PR touches only 1–4 files (avg 2.5), consistent with a single grader/evaluator file added in isolation from its workflow
• The few bot-generated comments present (PR Triage Agent) tag these as Batch: operational-value-grading with routine batch_review scoring — not substantive rejections

This pattern (near-instant close, no human review, no CI friction visible, narrow uniform file scope) is inconsistent with organic review rejection or CI test failures, and consistent with an automated admission/contract check rejecting these submissions in bulk shortly after creation.

Blocker Category	Count	Share	Notes
Scope/complexity mismatch (contract-gate pattern)	54	90%	Zero comments/reviews, closed in minutes — matches automated contract-gate rejection signature
Review friction	1	2%	#60635 — only PR in cluster with an actual review/discussion thread
Other/unknown	5	8%	Has 1–2 bot/triage comments but no human review; cause not confirmed
CI/test failure	0	0%	No direct evidence of CI-driven rejection found in available data

Recommendations

1. Stop generating standalone operational-value grader tasks until they're paired with their owning workflow in the same PR, per the repo's operational-value contract rule — this would eliminate the single largest source of wasted agent effort (60 PRs, 10% of total volume, 0% success).
2. Favor narrowly-scoped, template-following tasks (firewall logging, Sous Chef automation) when assigning new work — these show the most consistent success (89–96%).
3. Investigate Cluster 2's lower success rate (69.2%): gh-aw CLI/workflow-tooling tasks tend to touch more files (avg 61.2) and may benefit from being broken into smaller, more targeted PRs.
4. Outlier Cluster Recommendation: Route operational-value grader requests through .github/skills/operational-value-designer/SKILL.md from the start (as mandated by AGENTS.md), bundling the evaluator and its workflow in one change, and validate locally with verify-operational-value-contract-change.sh before opening the PR — this directly targets the contract-gate mismatch identified above.

---

Generated by Prompt Clustering Analysis (Run: 37197804983)

Generated by 📊 Copilot Agent Prompt Clustering Analysis · claude · agent · 152.2 AIC · ⌖ 9.62 AIC · ⊞ 6.8K · ◷

• expires on Oct 5, 2026, 3:27 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #65832.