Prevent AI credits rate-limit false positives from MCP echoes by Copilot · Pull Request #61425 · github/gh-aw

Copilot · 2026-09-16T23:18:23Z

Rate-limit detection scanned echoed MCP payloads and could self-seed from prior failure issue titles, causing failure handling despite successful agent conclusions.

Detection hardening

Bound related-keyword matching to 80 characters.
Skip lines containing MCP tool_result payload echoes.

line => !/\btool[_\s-]*result\b/i.test(line) &&
  AI_CREDITS_RATE_LIMIT_PATTERNS.some(pattern => pattern.test(line))

Conclusion handling
- Only treat an AI credits rate-limit signal as actionable when the agent conclusion is failure.
Failure guidance
- Replace the Copilot-specific HTTP 429 assertion with engine-neutral AI credits rate-limit wording.
Regression coverage
- Cover echoed failure titles, distant keywords, nearby genuine signals, and successful conclusions carrying the signal.

Fixes AI-credits rate-limit detector false-positives on echoed MCP tool results, and self-seeds via its own issue title #61141

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

github.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "github.com"

See Network Configuration for more information.

Generated by 👨‍🍳 PR Sous Chef · pi · gpt54 · 31.6 AIC · ⊞ 9.1K · ◷
_{Comment /souschef to run again}

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

This reverts commit 005a093. Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot

🟢 Approval recommended

The implementation addresses the reported false-positive paths with focused regression coverage.

Pull request overview

Fixes #61141 by preventing echoed MCP payloads from falsely triggering AI credits rate-limit failure handling.

Changes:

Bounds keyword proximity and excludes MCP tool-result lines.
Requires an agent failure before acting on detected signals.
Adds engine-neutral guidance and regression tests.

File summaries

File	Description
`actions/setup/md/ai_credits_rate_limit_throttle.md`	Makes guidance engine-neutral.
`actions/setup/js/parse_mcp_gateway_log.cjs`	Hardens rate-limit detection.
`actions/setup/js/parse_mcp_gateway_log.test.cjs`	Tests echo filtering and keyword bounds.
`actions/setup/js/handle_agent_failure.cjs`	Gates signals on failed conclusions.
`actions/setup/js/handle_agent_failure.test.cjs`	Tests successful-conclusion handling.

Review details

Files reviewed: 5/5 changed files
Comments generated: 0
Review effort level: Balanced

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

gh-aw-bot · 2026-09-16T23:48:17Z

@copilot this PR looks close, but it has not moved since the latest review. Please do a quick branch refresh if needed, re-run the pr-finisher skill, and confirm there are no remaining follow-ups before merge.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

github.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "github.com"

See Network Configuration for more information.

Generated by 👨‍🍳 PR Sous Chef · pi · gpt54 · 50.1 AIC · ⊞ 9.4K · ◷
_{Comment /souschef to run again}

Copilot · 2026-09-17T00:03:47Z

@copilot this PR looks close, but it has not moved since the latest review. Please do a quick branch refresh if needed, re-run the pr-finisher skill, and confirm there are no remaining follow-ups be...

Branch refresh confirmed it is 4 commits ahead and 0 behind main; no update was needed. pr-finisher found no review threads or code follow-ups. Local make fmt, make lint, make test-unit, make test, and make recompile passed; the remaining Copilot check is in progress.

github-actions · 2026-09-17T02:29:42Z

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel skipped because pre-fetch PR data was unavailable: unable to fetch test file diff

🧪 Test quality analysis by Test Quality Sentinel

github-actions · 2026-09-17T02:29:49Z

⚠️ Security scanning failed for Design Decision Gate 🏗️. Review the logs for details.

No ADR enforcement needed: PR does not have the 'implementation' label and has ≤100 new lines of code in business logic directories.

🏗️ ADR gate enforced by Design Decision Gate 🏗️

github-actions · 2026-09-17T02:29:51Z

⚠️ PR Code Quality Reviewer failed during code quality review.

Warning

Threat Detection Engine Failure — The analysis engine could not complete. This is a tooling failure, not a security finding.

What happened

The threat detection engine failed to produce results.

Review the workflow run logs for details.

🔎 Code quality review by PR Code Quality Reviewer

github-actions · 2026-09-17T02:29:53Z

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "registry.npmjs.org"

See Network Configuration for more information.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

github-actions · 2026-09-17T02:30:38Z

✅ Ponytail Reviewer completed successfully!

Lean already. Ship.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

ab.chatgpt.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "ab.chatgpt.com"

See Network Configuration for more information.

Generated by Ponytail Reviewer for #61425

github-actions

Reviewed with a correctness/security focus (Go/JS backend change; the Impeccable UI modes don't apply here since there's no UI in this diff).

Findings: No blocking issues. The changes are small, well-scoped, and covered by new regression tests:

Bounding AI_CREDITS_RATE_LIMIT_PATTERNS to 80 chars and excluding tool_result echo lines correctly prevents false positives from MCP payload echoes without narrowing genuine nearby signals.
Gating aiCreditsRateLimitError on agentConclusion === "failure" matches the existing pattern used by sibling flags (maxCacheMissesExceeded, missingModelPricingError, shellExpansionGuardRejected), so it's consistent with the codebase's conventions.
The engine-neutral wording change in ai_credits_rate_limit_throttle.md is a reasonable, low-risk copy fix.
New tests in both .test.cjs files directly cover the fixed scenarios (echoed titles, distant keywords, success conclusion, nearby genuine signal).

Note: I was unable to execute vitest in this sandbox (dependency install did not complete), so test results are based on static review of the added test cases rather than a live run.

Warning

Firewall blocked 2 domains

The following domains were blocked by the firewall during workflow execution:

codeload.github.com
github.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "codeload.github.com"
    - "github.com"

See Network Configuration for more information.

🧵 Reviewed using Impeccable skills by Impeccable Skills Reviewer · copilot · sonnet50 · 31 AIC · ⌖ 14.6 AIC · ⊞ 8.4K

github-actions

Skills-Based Review 🧠

Applied /diagnosing-bugs and /tdd — the fix correctly hardens parse_mcp_gateway_log.cjs, but the same false-positive pattern class remains unpatched in a sibling detection path.

📋 Key Themes & Highlights

Key Themes

Root cause only partially addressed: hasAICreditsRateLimitError in parse_mcp_gateway_log.cjs now bounds keyword distance to 80 chars and filters tool_result echoes. However ai_credits_context.cjs's AI_CREDITS_RATE_LIMIT_PATTERNS (used by resolveAICreditsFailureState(), the function handle_agent_failure.cjs gates on agentConclusion === "failure") still uses unbounded .* matching and has no echo filtering — the same false-positive risk can leak through the firewall audit-log path.
Test coverage gap: the new regression test only covers the agentConclusion === "success" case; there's no companion test pinning down that the rate-limit signal is still honored when agentConclusion === "failure".

Positive Highlights

✅ Clean, minimal conclusion-gating change in handle_agent_failure.cjs
✅ Good regression coverage for the echoed-title and distant-keyword false positives in parse_mcp_gateway_log.test.cjs
✅ Engine-neutral wording update in the throttle template is a nice correctness improvement independent of the bug fix

Posted 2 inline comments with specific suggestions.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "registry.npmjs.org"

See Network Configuration for more information.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · sonnet50 · 197.1 AIC · ⌖ 15.2 AIC · ⊞ 10.4K
_{Comment /matt to run again}

github-actions · 2026-09-17T02:46:11Z

+      delete process.env.GH_AW_AI_CREDITS_RATE_LIMIT_ERROR;
+      delete process.env.GH_AW_AIC;
+    }
+  });


[/tdd] Good regression test for the success case, but there's no test asserting the opposite: that aiCreditsRateLimitError is still honored when agentConclusion === "failure". Without it, a future refactor could silently invert or drop the agentConclusion === "failure" guard and this suite would still pass.

💡 Suggested test

it("still handles an AI credits rate-limit signal when the agent failed", async () => { // ...same setup as the success test but with: process.env.GH_AW_AGENT_CONCLUSION = "failure"; process.env.GH_AW_AI_CREDITS_RATE_LIMIT_ERROR = "true"; process.env.GH_AW_AIC = "1"; // assert a failure issue / ai_credits_rate_limit_error context IS produced });

This closes the loop on the fix and pins down both directions of the new conclusion gate as a spec, per the red-green-refactor principle.

@copilot please address this.

Addressed in 7f2a775: added a failed-conclusion regression that verifies the AI credits rate-limit signal remains actionable.

gh-aw-bot · 2026-09-17T05:41:02Z

pr-sous-chef
@copilot this PR still needs forward progress before a maintainer can investigate quickly.

Please address the unresolved review feedback:

github-actions[bot]: Prevent AI credits rate-limit false positives from MCP echoes #61425 (comment)

Also review the failing check from the candidate scan:

agent: https://github.com/github/gh-aw/actions/runs/35162496291/job/105053539404

If the branch is behind or stale, refresh it, then continue with the pr-finisher skill to close out review feedback and validation.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

github.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "github.com"

See Network Configuration for more information.

Generated by 👨‍🍳 PR Sous Chef · pi · gpt54 · 31.6 AIC · ⊞ 9.1K · ◷
_{Comment /souschef to run again}

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot · 2026-09-17T05:56:58Z

pr-sous-chef
@copilot this PR still needs forward progress before a maintainer can investigate quickly.
...

Addressed the GitHub Actions review in 1358663: firewall audit-log matching now applies the same bounded proximity and MCP tool_result filtering, with regression coverage. The candidate scan’s agent log reports a threat-detection engine tooling failure, not a code finding; local validation passes.

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

github-actions · 2026-09-19T03:47:57Z

🎉 This pull request is included in a new release.

Release: v0.89.17

Initial plan

1959727

Copilot AI assigned Copilot and pelikhan Sep 16, 2026

Copilot started work on behalf of pelikhan September 16, 2026 23:18 View session

Copilot AI linked an issue Sep 16, 2026 that may be closed by this pull request

AI-credits rate-limit detector false-positives on echoed MCP tool results, and self-seeds via its own issue title #61141

Closed

Copilot AI and others added 3 commits September 16, 2026 23:22

Plan AI credits rate-limit false-positive fix

005a093

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Revert "Plan AI credits rate-limit false-positive fix"

c6b139f

This reverts commit 005a093. Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Fix AI credits rate-limit false positives

74694f5

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Fix false-positive rate-limit detection in AI credits~~ Prevent AI credits rate-limit false positives from MCP echoes Sep 16, 2026

Copilot finished work on behalf of pelikhan September 16, 2026 23:28

Copilot AI requested a review from pelikhan September 16, 2026 23:28

pelikhan marked this pull request as ready for review September 16, 2026 23:29

Copilot AI balanced review requested due to automatic review settings September 16, 2026 23:29

Copilot started reviewing on behalf of pelikhan September 16, 2026 23:29 View session

Copilot AI reviewed Sep 16, 2026

View reviewed changes

github-actions Bot mentioned this pull request Sep 16, 2026

[pr-sous-chef] PR Sous Chef run report (2026-09-15T23:58Z) #61229

Closed

Copilot started work on behalf of gh-aw-bot September 16, 2026 23:48 View session

github-actions Bot mentioned this pull request Sep 17, 2026

[pr-sous-chef] PR Sous Chef run report #61445

Closed

Copilot finished work on behalf of gh-aw-bot September 17, 2026 00:04

Copilot AI requested a review from gh-aw-bot September 17, 2026 00:04

pelikhan unassigned Copilot Sep 17, 2026

github-actions Bot added the needs-review label Sep 17, 2026

github-actions Bot approved these changes Sep 17, 2026

View reviewed changes

github-actions Bot reviewed Sep 17, 2026

View reviewed changes

Merge branch 'main' into copilot/fix-ai-credits-rate-limit

790fb1d

Copilot started work on behalf of gh-aw-bot September 17, 2026 05:41 View session

Copilot AI and others added 2 commits September 17, 2026 05:46

Plan rate-limit review follow-up

6657d88

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Harden audit-log rate-limit detection

1358663

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Cover failed rate-limit issue handling

93dc438

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

pelikhan assigned Copilot Sep 17, 2026

pelikhan merged commit ef7f1e2 into main Sep 17, 2026
1 check failed

pelikhan deleted the copilot/fix-ai-credits-rate-limit branch September 17, 2026 05:58

Copilot AI restored the copilot/fix-ai-credits-rate-limit branch September 17, 2026 05:58

Copilot finished work on behalf of gh-aw-bot September 17, 2026 05:58

github-actions Bot deleted the copilot/fix-ai-credits-rate-limit branch September 25, 2026 02:02

Conversation

Copilot AI commented Sep 16, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

gh-aw-bot commented Sep 16, 2026

Uh oh!

Copilot AI commented Sep 17, 2026

Uh oh!

github-actions Bot commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Skills-Based Review 🧠

Key Themes

Positive Highlights

Uh oh!

github-actions Bot Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

gh-aw-bot commented Sep 17, 2026

Uh oh!

Copilot AI commented Sep 17, 2026

Uh oh!

Uh oh!

github-actions Bot commented Sep 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Copilot AI commented Sep 16, 2026 •

edited by github-actions Bot

Loading

github-actions Bot commented Sep 17, 2026 •

edited

Loading

github-actions Bot commented Sep 17, 2026 •

edited

Loading

github-actions Bot commented Sep 17, 2026 •

edited

Loading

github-actions Bot commented Sep 17, 2026 •

edited

Loading

github-actions Bot commented Sep 17, 2026 •

edited

Loading