Skip to content

Prompt Clustering: add mandatory root-cause drilldown for low-merge outlier clusters - #51141

Merged
pelikhan merged 2 commits into
mainfrom
copilot/deep-report-investigate-merge-rate
Aug 7, 2026
Merged

Prompt Clustering: add mandatory root-cause drilldown for low-merge outlier clusters#51141
pelikhan merged 2 commits into
mainfrom
copilot/deep-report-investigate-merge-rate

Conversation

Copilot AI commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

The prompt-clustering workflow identified cluster-level merge-rate variance but did not require a blocker analysis for low-performing clusters. This left outlier clusters (notably image/container/digest-themed groups) without a concrete, evidence-backed root cause.

  • Root-cause trigger for outlier clusters

    • Added explicit outlier detection criteria in clustering analysis:
      • cluster size threshold (>=15 PRs)
      • merge-rate gap threshold (>=10 percentage points below overall)
    • Requires focused investigation of the lowest-merge qualifying cluster.
  • New investigation phase (Phase 4.5)

    • Added a mandatory blocker-classification pass over PR evidence (comments, reviews, reviewDecision, changed files).
    • Standardized blocker taxonomy:
      • CI/test failure (including container/image build failures)
      • review friction (including digest/security/compliance concerns)
      • scope/complexity mismatch
      • other/unknown
    • Requires evidence sampling (minimum 10 PRs when available), comparison against merged PRs in-cluster, and a primary-blocker conclusion with confidence.
  • Report output upgrades

    • Added a dedicated Lowest-Merge Cluster Root Cause section to the report template.
    • Added blocker distribution table, merge-rate gap summary, evidence PR references, and targeted mitigation recommendations.
    • Updated success criteria to require this outlier investigation when applicable.
### Phase 4.5: Root-Cause Analysis for Low-Merge Clusters

When an outlier cluster is detected (>=15 PRs and merge rate >=10 points below overall rate), perform a focused blocker investigation for the lowest-merge cluster.
...
| Blocker Category | Count | Share | Notes |
|------------------|-------|-------|-------|
| CI/test failure  | X     | Y%    | ...   |
| Review friction  | X     | Y%    | ...   |

… report

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Investigate low merge rate of image/container/digest Copilot PRs Prompt Clustering: add mandatory root-cause drilldown for low-merge outlier clusters Aug 7, 2026
Copilot AI requested a review from pelikhan August 7, 2026 17:26
@pelikhan
pelikhan marked this pull request as ready for review August 7, 2026 17:29
Copilot AI balanced review requested due to automatic review settings August 7, 2026 17:29
@pelikhan
pelikhan merged commit cfad684 into main Aug 7, 2026
@pelikhan
pelikhan deleted the copilot/deep-report-investigate-merge-rate branch August 7, 2026 17:29

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds mandatory root-cause analysis for statistically significant low-merge prompt clusters.

Changes:

  • Defines outlier thresholds and blocker taxonomy.
  • Adds evidence, comparison, and mitigation reporting requirements.
  • Extends the report template and success criteria.
Show a summary per file
File Description
.github/workflows/prompt-clustering-analysis.md Adds outlier investigation and reporting instructions.
.github/workflows/prompt-clustering-analysis.lock.yml Updates the compiled workflow body hash.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Suppressed comments (1)

.github/workflows/prompt-clustering-analysis.md:595

  • The investigation phase requires three concrete mitigation actions, while the report template requests only one outlier recommendation. Agents following the template can therefore omit two required mitigations. Provide three dedicated outlier mitigation slots.
4. **[Outlier Cluster Recommendation]**: [Action specifically targeting the lowest-merge cluster blocker]
  • Files reviewed: 2/2 changed files
  • Comments generated: 4
  • Review effort level: Balanced

Comment on lines +367 to +368
2. Use full PR data (`comments`, `reviews`, `reviewDecision`, changed files) to classify merge blockers per PR into categories:
- CI/test failure (including container/image build failures)
2. **[Finding 2]**: [Description and data supporting this finding]
3. **[Finding 3]**: [Description and data supporting this finding]

### Lowest-Merge Cluster Root Cause
Comment on lines +581 to +586
| Blocker Category | Count | Share | Notes |
|------------------|-------|-------|-------|
| CI/test failure | X | Y% | [notable failure pattern] |
| Review friction | X | Y% | [notable review concern] |
| Scope mismatch | X | Y% | [complexity signal] |
| Other/unknown | X | Y% | [notes] |
- **Merge Rate Gap**: [cluster rate]% vs [overall rate]% (gap: [X] points)
- **Primary Blocker**: [CI failures / review friction / scope mismatch / other]
- **Confidence**: [high/medium/low]
- **Evidence PRs**: #[id], #[id], #[id]
@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v0.86.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[deep-report] Investigate why the image/container/digest Copilot PR cluster has the lowest merge rate (58.8% vs 74.3% fleet average)

3 participants