[aw-failures] Failure Report 2026-05-01: Gemini API Key Invalid + Crush EROFS + Branch-Deleted Race Condition

[github-actions]

Executive Summary

4 failures in the last 6 hours (2026-05-01 01:18–07:38 UTC), all on PR branch copilot/guard-bot-filtering-against-dependabot-attacks. Two are P0 engine-level failures (Gemini API key invalid, Crush EROFS installation failure) and two are P1 safe_outputs failures caused by the PR branch being deleted before the workflow completed.

Failure Clusters

Run ID	Workflow	Engine	Failure Stage	Root Cause	Priority
§25203411975	Smoke Gemini	gemini	agent	API_KEY_INVALID from generativelanguage.googleapis.com	P0
§25203411962	Smoke Crush	crush	agent	EROFS: read-only file system installing Crush CLI binary	P0
§25203411946	Changeset Generator	codex	safe_outputs	Branch deleted before push_to_pull_request_branch ran	P1
§25203411945	Smoke Copilot	copilot	safe_outputs	Branch deleted → haiku-printer dispatch failed; unresolvable PR review comment line	P1

Evidence

Cluster A — Gemini API Key Invalid (P0)

Run §25203411975 produced 0 turns before dying in the agent job.

Key error from gemini-client-error-generateJson-api-*.json:

ApiError: {"error":{"code":400,"message":"API key not valid. Please pass a valid API key.",
  "status":"INVALID_ARGUMENT","details":[{"`@type`":"type.googleapis.com/google.rpc.ErrorInfo",
  "reason":"API_KEY_INVALID","domain":"googleapis.com","metadata":{"service":"generativelanguage.googleapis.com"}}]}}

The firewall log shows 1 blocked request to (unknown) domain alongside 1 allowed call to generativelanguage.googleapis.com (the one that returned 400).

All other smoke tests on the same PR (Claude, Codex, OpenCode) succeeded, confirming this is Gemini-specific.

Cluster B — Crush EROFS Installation Failure (P0)

Run §25203411962 produced 0 turns.

Key error from agent-stdio.log:

Error: Installation failed: EROFS: read-only file system, mkdir
  '/opt/hostedtoolcache/node/24.14.1/x64/lib/node_modules/`@charmland/crush`/bin'
    at install ((redacted)

The binary download completes successfully (crush_0.59.0_Linux_x86_64.tar.gz) but the @charmland/crush npm package tries to extract it into the read-only hostedtoolcache node_modules path.

Cluster C — PR Branch Deleted Before safe_outputs (P1)

Both runs §25203411946 (Changeset Generator) and §25203411945 (Smoke Copilot) succeeded in the agent job but failed in safe_outputs.

Changeset Generator error:

✗ Message 1 (push_to_pull_request_branch) failed: Branch
  copilot/guard-bot-filtering-against-dependabot-attacks no longer exists on origin
  (it may have been deleted), can't push to it.
⏭ Message 2 (update_pull_request) cancelled — Cancelled: code push operation failed

Smoke Copilot errors:

Failed to dispatch workflow "haiku-printer": No ref found for:
  refs/heads/copilot/guard-bot-filtering-against-dependabot-attacks
  
PR review submission failed: "Line could not be resolved"

Smoke Copilot also showed high resource consumption (28 turns, write_heavy, 11m duration), flagged with severity=high resource_heavy_for_domain agentic assessment.

Existing Issue Correlation

Unable to query open issues (GitHub API returned 403 from the workflow runner). Sub-issues created below are fresh; please close or link if duplicates exist.

Proposed Fix Roadmap

Priority	Issue	Owner	Effort
P0	Rotate/renew GOOGLE_GEMINI_API_KEY secret	infra	Minutes
P0	Fix Crush CLI install path (avoid read-only hostedtoolcache)	platform	Hours
P1	Graceful handling when PR branch deleted before safe_outputs	platform	Days
P2	Investigate Smoke Copilot resource consumption (28 turns, write_heavy)	workflow	Hours

Sub-Issues Created

References:

• §25203411975 Smoke Gemini
• §25203411962 Smoke Crush
• §25203411946 Changeset Generator
• §25203411945 Smoke Copilot

Generated by [aw] Failure Investigator (6h) · ● 263.7K · ◷

• expires on May 8, 2026, 7:43 AM UTC

[github-actions]

This issue was automatically closed because it expired on 2026-05-08T07:43:39.580Z.

Closed by Workflow