Skip to content

Prevent AI credits rate-limit false positives from MCP echoes #61425

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pelikhan merged 8 commits into from
Sep 17, 2026
Merged

Prevent AI credits rate-limit false positives from MCP echoes #61425

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
1959727
Initial plan
Copilot Sep 16, 2026
005a093
Plan AI credits rate-limit false-positive fix
Copilot Sep 16, 2026
c6b139f
Revert "Plan AI credits rate-limit false-positive fix"
Copilot Sep 16, 2026
74694f5
Fix AI credits rate-limit false positives
Copilot Sep 16, 2026
790fb1d
Merge branch 'main' into copilot/fix-ai-credits-rate-limit
github-actions[bot] Sep 17, 2026
6657d88
Plan rate-limit review follow-up
Copilot Sep 17, 2026
1358663
Harden audit-log rate-limit detection
Copilot Sep 17, 2026
93dc438
Cover failed rate-limit issue handling
Copilot Sep 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions actions/setup/js/ai_credits_context.cjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ const AI_CREDITS_RATE_LIMIT_ERROR_FIELDS = new Set(["ai_credits_rate_limit_error
// Note: these text fields are intentionally broad (common field names like "error", "message") because
// rate-limit signals can appear in any of them. This asymmetry vs parseMaxAICreditsFromAuditLog is deliberate.
const AI_CREDITS_RATE_LIMIT_TEXT_FIELDS = new Set(["error", "message", "reason", "details", "detail", "type", "code"]);
const AI_CREDITS_RATE_LIMIT_PATTERNS = [/ai[\s_-]*credits?.*(?:rate[\s-]*limit|limit exceeded|budget exceeded|exceeded)/i, /(?:rate[\s-]*limit|too many requests).*(?:ai[\s_-]*credits?)/i, /\bai_credits_limit_exceeded\b/i];
const AI_CREDITS_RATE_LIMIT_PATTERNS = [/ai[\s_-]*credits?.{0,80}(?:rate[\s-]*limit|limit exceeded|budget exceeded|exceeded)/i, /(?:rate[\s-]*limit|too many requests).{0,80}(?:ai[\s_-]*credits?)/i, /\bai_credits_limit_exceeded\b/i];
const MAX_AI_CREDITS_EXCEEDED_FIELDS = new Set(["max_ai_credits_exceeded", "maxAiCreditsExceeded"]);
const AI_CREDITS_TOTAL_FIELDS = new Set(["ai_credits_total", "aiCreditsTotal"]);
/** @type {{ aiCredits: string, maxAICredits: string, rateLimitError: boolean, maxAICreditsExceeded: boolean }} */
Expand Down Expand Up @@ -217,7 +217,7 @@ function parseAICreditsErrorInfoFromAuditEntry(entry) {
if (parsed) aiCredits = parsed;
}
if (AI_CREDITS_RATE_LIMIT_ERROR_FIELDS.has(key) && isTrueLike(value)) rateLimitError = true;
if (AI_CREDITS_RATE_LIMIT_TEXT_FIELDS.has(key) && typeof value === "string") {
if (AI_CREDITS_RATE_LIMIT_TEXT_FIELDS.has(key) && typeof value === "string" && !/\btool[_\s-]*result\b/i.test(value)) {
if (AI_CREDITS_RATE_LIMIT_PATTERNS.some(pattern => pattern.test(value))) rateLimitError = true;
}
});
Expand Down
15 changes: 15 additions & 0 deletions actions/setup/js/ai_credits_context.test.cjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -611,6 +611,21 @@ describe("ai_credits_context parseAICreditsErrorInfoFromAuditLog", () => {
expect(parseAICreditsErrorInfoFromAuditLog().rateLimitError).toBe(true);
});

it("ignores rate-limit text echoed in an MCP tool result", () => {
writeAuditLog([{ message: 'tool_result: {"title":"[aw] Workflow hit AI credits rate limit"}' }]);
expect(parseAICreditsErrorInfoFromAuditLog().rateLimitError).toBe(false);
});

it("ignores rate-limit keywords farther than 80 characters apart", () => {
writeAuditLog([{ message: `AI credits ${"x".repeat(79)} rate limit exceeded` }]);
expect(parseAICreditsErrorInfoFromAuditLog().rateLimitError).toBe(false);
});

it("detects nearby rate-limit keywords", () => {
writeAuditLog([{ message: `AI credits ${"x".repeat(78)} rate limit exceeded` }]);
expect(parseAICreditsErrorInfoFromAuditLog().rateLimitError).toBe(true);
});

it("detects rate limit from code field with ai_credits_limit_exceeded", () => {
writeAuditLog([{ code: "ai_credits_limit_exceeded" }]);
expect(parseAICreditsErrorInfoFromAuditLog().rateLimitError).toBe(true);
Expand Down
3 changes: 2 additions & 1 deletion actions/setup/js/handle_agent_failure.cjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3442,7 +3442,8 @@ async function main() {
const codePushFailureCount = process.env.GH_AW_CODE_PUSH_FAILURE_COUNT || "0";
const checkoutPRSuccess = process.env.GH_AW_CHECKOUT_PR_SUCCESS || "";
const timeoutMinutes = process.env.GH_AW_TIMEOUT_MINUTES || "";
const { aiCredits, maxAICredits, aiCreditsRateLimitError, maxAICreditsExceeded } = resolveAICreditsFailureState();
const { aiCredits, maxAICredits, aiCreditsRateLimitError: detectedAICreditsRateLimitError, maxAICreditsExceeded } = resolveAICreditsFailureState();
const aiCreditsRateLimitError = agentConclusion === "failure" && detectedAICreditsRateLimitError;
const inferenceAccessError = process.env.GH_AW_INFERENCE_ACCESS_ERROR === "true";
const copilotOrgBillingError = detectCopilotOrgBillingErrorFromLog();
const mcpPolicyError = process.env.GH_AW_MCP_POLICY_ERROR === "true";
Expand Down
62 changes: 62 additions & 0 deletions actions/setup/js/handle_agent_failure.test.cjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,68 @@ describe("handle_agent_failure", () => {
});
});

it("does not handle an AI credits rate-limit signal when the agent succeeded", async () => {
const fs = require("fs");
const os = require("os");
const path = require("path");
const agentOutputPath = path.join(fs.mkdtempSync(path.join(os.tmpdir(), "aw-agent-output-")), "output.json");
fs.writeFileSync(agentOutputPath, JSON.stringify({ items: [{ type: "create_discussion" }] }));
process.env.GH_AW_AGENT_OUTPUT = agentOutputPath;
process.env.GH_AW_AGENT_CONCLUSION = "success";
process.env.GH_AW_AI_CREDITS_RATE_LIMIT_ERROR = "true";
process.env.GH_AW_AIC = "1";

try {
await main();
expect(global.core.info).toHaveBeenCalledWith(expect.stringContaining("skipping failure handling"));
} finally {
fs.rmSync(path.dirname(agentOutputPath), { recursive: true, force: true });
delete process.env.GH_AW_AGENT_OUTPUT;
delete process.env.GH_AW_AGENT_CONCLUSION;
delete process.env.GH_AW_AI_CREDITS_RATE_LIMIT_ERROR;
delete process.env.GH_AW_AIC;
}
});

@github-actions github-actions Bot Sep 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/tdd] Good regression test for the success case, but there's no test asserting the opposite: that aiCreditsRateLimitError is still honored when agentConclusion === "failure". Without it, a future refactor could silently invert or drop the agentConclusion === "failure" guard and this suite would still pass.

💡 Suggested test 
it("still handles an AI credits rate-limit signal when the agent failed", async () => {
  // ...same setup as the success test but with:
  process.env.GH_AW_AGENT_CONCLUSION = "failure";
  process.env.GH_AW_AI_CREDITS_RATE_LIMIT_ERROR = "true";
  process.env.GH_AW_AIC = "1";
  // assert a failure issue / ai_credits_rate_limit_error context IS produced
});

This closes the loop on the fix and pins down both directions of the new conclusion gate as a spec, per the red-green-refactor principle.

@copilot please address this.

Copilot AI Sep 17, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 7f2a775: added a failed-conclusion regression that verifies the AI credits rate-limit signal remains actionable.


it("handles an AI credits rate-limit signal when the agent failed", async () => {
const fs = require("fs");
const os = require("os");
const path = require("path");
const agentOutputPath = path.join(fs.mkdtempSync(path.join(os.tmpdir(), "aw-agent-output-")), "output.json");
fs.writeFileSync(agentOutputPath, JSON.stringify({ items: [{ type: "create_discussion" }] }));
process.env.GH_AW_AGENT_OUTPUT = agentOutputPath;
process.env.GH_AW_AGENT_CONCLUSION = "failure";
process.env.GH_AW_AI_CREDITS_RATE_LIMIT_ERROR = "true";
process.env.GH_AW_AIC = "1";
const createIssueMock = vi.fn(async () => ({
data: { number: 99, html_url: "https://github.com/owner/repo/issues/99", node_id: "I_99" },
}));
global.github = {
rest: {
search: {
issuesAndPullRequests: vi.fn(async () => ({ data: { total_count: 0, items: [] } })),
},
issues: {
create: createIssueMock,
createComment: vi.fn(),
},
pulls: { get: vi.fn() },
},
graphql: vi.fn(),
};

try {
await main();
expect(createIssueMock).toHaveBeenCalledWith(expect.objectContaining({ title: "[aw] unknown hit AI credits rate limit" }));
} finally {
fs.rmSync(path.dirname(agentOutputPath), { recursive: true, force: true });
delete process.env.GH_AW_AGENT_OUTPUT;
delete process.env.GH_AW_AGENT_CONCLUSION;
delete process.env.GH_AW_AI_CREDITS_RATE_LIMIT_ERROR;
delete process.env.GH_AW_AIC;
}
});

describe("buildFailureIssueTitle", () => {
const baseOptions = {
workflowName: "Test Workflow",
Expand Down
10 changes: 4 additions & 6 deletions actions/setup/js/parse_mcp_gateway_log.cjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,9 +45,9 @@ function getRpcMessageType(entry) {
return "";
}
const AI_CREDITS_RATE_LIMIT_PATTERNS = [
/ai[\s_-]*credits?.*(?:rate[\s-]*limit|limit exceeded|budget exceeded|exceeded)/i,
/(?:rate[\s-]*limit|too many requests).*(?:ai[\s_-]*credits?)/i,
/\b429\b.*(?:rate[\s-]*limit|too many requests|ai[\s_-]*credits?)/i,
/ai[\s_-]*credits?.{0,80}(?:rate[\s-]*limit|limit exceeded|budget exceeded|exceeded)/i,
/(?:rate[\s-]*limit|too many requests).{0,80}(?:ai[\s_-]*credits?)/i,
/\b429\b.{0,80}(?:rate[\s-]*limit|too many requests|ai[\s_-]*credits?)/i,
];
// Detects the AWF API proxy HTTP 400 error emitted when maxAiCredits is active and
// the requested model is not in the built-in pricing table.
Expand Down Expand Up @@ -435,9 +435,7 @@ async function writeStepSummaryWithTokenUsage(coreObj) {
* @returns {boolean}
*/
function hasAICreditsRateLimitError(contents) {
const joined = contents.filter(Boolean).join("\n");
if (!joined) return false;
return AI_CREDITS_RATE_LIMIT_PATTERNS.some(pattern => pattern.test(joined));
return contents.filter(Boolean).some(content => content.split(/\r?\n/).some(line => !/\btool[_\s-]*result\b/i.test(line) && AI_CREDITS_RATE_LIMIT_PATTERNS.some(pattern => pattern.test(line))));
}

/**
Expand Down
11 changes: 11 additions & 0 deletions actions/setup/js/parse_mcp_gateway_log.test.cjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ const {
generateTokenUsageSummary,
formatDurationMs,
writeStepSummaryWithTokenUsage,
hasAICreditsRateLimitError,
} = require("./parse_mcp_gateway_log.cjs");

describe("parse_mcp_gateway_log", () => {
Expand Down Expand Up @@ -2413,3 +2414,13 @@ not-json
});
});
});
describe("hasAICreditsRateLimitError", () => {
test("ignores echoed MCP tool results and distant keywords", () => {
expect(hasAICreditsRateLimitError(['... tool_result ... "title":"[aw] Weekly Research hit AI credits rate limit" ...'])).toBe(false);
expect(hasAICreditsRateLimitError([`AI credits ${"x".repeat(81)} rate limit`])).toBe(false);
});

test("detects a nearby AI credits rate-limit error", () => {
expect(hasAICreditsRateLimitError(["CAPIError: AI credits rate limit exceeded"])).toBe(true);
});
});
2 changes: 1 addition & 1 deletion actions/setup/md/ai_credits_rate_limit_throttle.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
> [!WARNING]
> **AI Credits Rate Limit**
>
> The Copilot API returned a rate limit response (HTTP 429), but the workflow did not report the explicit AI credits budget-exceeded guardrail signal.{metrics_summary}
> The workflow encountered an AI credits rate-limit signal, but did not report the explicit AI credits budget-exceeded guardrail signal.{metrics_summary}

<details>
<summary>Tips for reducing rate limit issues</summary>
Expand Down
Loading