Skip to content

[aw-failures] P2: Daily-AIC guardrail marks budget-blocked runs as failure (core.setFailed) — false-failure noise + auto-issue [Content truncated due to length] #39079

Description

@github-actions

Recommendation

Stop the daily-AIC guardrail from calling core.setFailed when the budget is exceeded — emit a neutral/skipped conclusion instead. The agent job is already gated on the daily_ai_credits_exceeded output, so the setFailed is redundant for control flow and only serves to paint budget-blocked runs red — which dominates the failure signal and spawns per-workflow auto-failure issues.

Problem statement

In the 2026-06-13 13:17Z 6h window, 14 of 20 "failed" agentic runs were not real failures — they were scheduled reviewer workflows correctly blocked by the daily-AIC guardrail after the shared 5000 daily-AIC budget was exhausted (~11:00Z). Each blocked run exits the activation job via core.setFailed, so the run conclusion is failure even though the design intent is "skip this run." Side effects: (1) reliability/failure dashboards over-count failures, (2) auto-failure issues are filed per workflow (#39059 Test Quality Sentinel, #39050 Matt Pocock Skills Reviewer), and (3) this Failure Investigator is triggered on non-actionable noise.

Root cause

The guardrail script hard-fails the step on exceedance:

  • actions/setup/js/check_daily_aic_workflow_guardrail.cjs:593core.setFailed(\Daily workflow AIC guardrail exceeded for ${workflowName}: ${totalAIC}/${threshold}.`)`
  • It also already sets the control output at line 586 → core.setOutput("daily_ai_credits_exceeded", "true").

The downstream agent job is gated on that output, independent of step status:

  • pkg/workflow/compiler_main_job.go:69needs.activation.outputs.daily_ai_credits_exceeded != 'true'

Observed guardrail log (representative run §27465717773):

##[warning]Daily workflow AIC guardrail exceeded for Test Quality Sentinel: 5017.38/5000.
##[error]Daily workflow AIC guardrail exceeded for Test Quality Sentinel: 5017.38/5000.

Because the agent job already keys on daily_ai_credits_exceeded, the core.setFailed at line 593 is redundant for skipping the agent — its only effect is the red failure conclusion.

Affected workflows and run IDs

Workflow Runs (in-window)
Test Quality Sentinel §27465717773, §27465421432, §27465299831, §27464789069, §27464680872
PR Code Quality Reviewer §27465717777, §27465421426, §27465299828, §27464789060, §27464680882
Matt Pocock Skills Reviewer §27465421436, §27464789067, §27464680862, §27463928154

Representative run: §27465717773 (clearest 5017.38/5000 exceedance log). Comparator: same workflows succeeded earlier in the day before the shared budget was exhausted; no in-window successful comparator exists because every post-exhaustion run is blocked.

Why this is distinct from existing AIC issues

Proposed remediation

  1. In check_daily_aic_workflow_guardrail.cjs, on exceedance keep core.setOutput("daily_ai_credits_exceeded", "true") and core.warning(...), but replace core.setFailed(...) (line 593) with a non-failing exit so the activation job concludes success (the agent still skips via the output gate).
  2. Optionally surface the block as a neutral GitHub conclusion (e.g. a notice + a step-summary line) so the run is visibly "budget-skipped," not green-as-if-ran.
  3. Suppress the per-workflow auto-failure issue path ([aw] Test Quality Sentinel exceeded daily AI credits budget #39059/[aw] Matt Pocock Skills Reviewer exceeded daily AI credits budget #39050 class) when daily_ai_credits_exceeded == 'true', since the run is a deliberate skip, not a failure.

Success criteria / verification

  1. A workflow blocked by the daily-AIC guardrail produces a run whose conclusion is not failure (skipped/neutral/success-with-skip).
  2. The agent job still does not execute when daily_ai_credits_exceeded == 'true' (existing gate unchanged; verify via daily_aic_workflow_guardrail_test.go).
  3. No [aw] ... exceeded daily AI credits budget auto-failure issue is filed for a guardrail-skipped run.
  4. A subsequent Failure Investigator window over a budget-exhaustion period reports these runs as skipped, not as P-tier failures.

References

Generated by 🔍 [aw] Failure Investigator (6h) · 191.6 AIC · ⌖ 12.9 AIC · ⊞ 5.1K ·

  • expires on Jun 20, 2026, 5:26 AM UTC-08:00

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions