Skip to content

[aw-failures] pull_request max-stack filter crashes non-PR workflow runs — 29 workflows hit startup_failure #49445

Description

@github-actions

Executive Summary

Fix the max-stack PR filter now — it silently breaks every workflow with a pull_request trigger. Commit f9181ad ("Add stacked PR max-stack trigger protection") regenerated 30 .lock.yml files with a new activation if: clause that performs arithmetic on github.event.pull_request.stack.position/.size — fields that don't exist outside a real stacked-PR event. On any non-pull_request trigger (push, workflow_dispatch, etc.) this crashes GitHub Actions' expression evaluator, and the entire run fails at startup_failure before a single job executes. This is a repo-wide P0 regression, not an isolated flake.

Failure Cluster Table

Cluster Severity Workflows affected Representative run Comparator run Signature
C1 — Stack-filter arithmetic crash P0 ~29 of the 32 failures in this window: all smoke-*.lock.yml (Pi, Antigravity, agent-public-approved, agent-public-none, agent-all-merged, agent-all-none, agent-scoped-approved, call-workflow, ci, claude, codex, copilot-arm, create-cross-repo-pr, gemini, multi-pr, opencode, project, temporary-id, test-tools, update-cross-repo-pr), plus ai-moderator, changeset, design-decision-gate, firewall-escape, impeccable-skills-reviewer, mattpocock-skills-reviewer, pr-code-quality-reviewer, pr-description-caveman, refiner, test-quality-sentinel, visual-regression-checker 30668964442 (Smoke Pi) 30576616018 (last green Smoke Pi) startup_failure: 0 jobs run, 0s duration, created_at == updated_at
C2 — Detection continue-on-error still broken P1 (tracked) Detection Analysis Report 30672938234 — GH_AW_DETECTION_CONTINUE_ON_ERROR=true set but job still marked failure
C3 — Ollama BYOK backend failure P2 (tracked) Daily BYOK Ollama Test 30670580818 — Failure in "Execute GitHub Copilot CLI" step, ollama orphan process terminated in cleanup

Evidence

Fix the null/undefined .stack dereference in applyPullRequestStackFilter — it's the confirmed single point of failure. pkg/workflow/filters.go (added in f9181ad) generates, for every workflow whose on: includes pull_request (default max-stack: 1, no opt-in required):

github.event_name != 'pull_request' || github.event.pull_request.stack == null || github.event.pull_request.stack.position + 1 > github.event.pull_request.stack.size

This is spliced into the job-level activation if: via BuildConditionTree/RenderCondition (see diff to smoke-pi.lock.yml, activation and pre_activation jobs). github.event.pull_request.stack is not a real GitHub webhook field for ordinary PRs — on any event where github.event.pull_request itself is absent (push, workflow_dispatch, schedule...), GitHub Actions' expression evaluator hits an unsupported arithmetic operation (null + 1) while evaluating the job condition, which invalidates the whole run before job scheduling.

Verified via audit-diff between the failing run and the last successful Smoke Pi run:

  • Failing run (30668964442): 0 GitHub API calls, 0 token usage — the run never reached the agent/activation logic.
  • Comparator (30576616018): 12 API calls, normal core-quota consumption — a normal execution.
  • run_metrics_diff.github_rate_limit_details.api_calls_change: "+∞" confirms the failing run made no calls at all, consistent with a pre-job startup_failure, not an in-job error.

This is not new as of this window: an earlier Smoke Pi run (30606865041, 2026-07-31T05:27:45Z, ~17h before this window) shows the identical zero-duration/zero-job signature — the underlying filter bug predates this 6h lookback; commit f9181ad expanded its blast radius by regenerating 30 additional lock files with the same broken clause.

Existing Issue Correlation

Fix Roadmap

  • P0 — Ship now: Guard applyPullRequestStackFilter's generated condition so it never dereferences .stack.position/.size unless github.event_name == 'pull_request' is already true (short-circuit via nested &&/fromJSON default, not a flat || chain that GitHub's evaluator still type-checks). See sub-issue below.
  • P1 — Same PR: Add a compiler-level regression test that renders the activation/pre_activation if: for a workflow with on: [push, pull_request] and asserts the expression evaluates without error under a synthetic push event context.
  • P2 — Follow-up: Re-run gh-aw compile across the repo once fixed and confirm all 30 previously-regenerated lock files no longer contain the broken clause; re-enable/verify the smoke-test matrix goes green.

Sub-Issues Created

  • Sub-issue opened against pkg/workflow/filters.go's applyPullRequestStackFilter for the concrete arithmetic-on-null fix (linked below).

Generated by 🔍 [aw] Failure Investigator (6h) · agent · 102.6 AIC · ⌖ 32.8 AIC · ⊞ 6.8K · ◷

  • expires on Aug 7, 2026, 5:27 PM UTC-08:00

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions