[aw-failures] [aw] Failure Report 2026-05-14 (6h): PR Sous Chef Copilot field hallucination + 3 distinct workflow failures

[github-actions]

Executive Summary

Investigated the 6-hour window ending 2026-05-14 01:25 UTC. Out of 51 total agentic workflow runs, 7 failed (13.7%) across 5 distinct workflows. The dominant signal is PR Sous Chef, which fails ~37.5% of the time because the Copilot CLI hallucinates field names that the safe-outputs validator rejects (pr_number instead of pull_request_number, etc.). One Design Decision Gate failure is a continuation of #31178 — that issue has been updated with the recurrence; no new tracker is needed for that cluster.

Failure Clusters

#	Workflow	Failures / Total	Root cause	Priority	Tracking
1	PR Sous Chef	3 / 8 (37.5%)	Copilot emits pr_number instead of pull_request_number; pr / pr_number instead of item_number / issue_number	P0	NEW — see sub-issue #32040
2	Design Decision Gate 🏗️	1 / 3	Bash permission denials → 21-turn max-turns loop ($1.04 wasted)	P1	Existing #31178 (updated with recurrence note)
3	Daily Model Inventory Checker	1 / 1	Copilot CLI exits with exitCode=1 stdout=0B stderr=0B 1s after spawn in chroot — silent startup crash	P1	NEW (this report)
4	Agentic Workflow Audit Agent	1 / 1	pip install pandas matplotlib seaborn install hang in chroot → 35-min workflow timeout, Turns: 0	P1	NEW (this report)
5	AI Moderator	1 / 1	hide_comment passed REST numeric ID (4445729864) — GraphQL minimizeComment requires a global node ID	P1	NEW (this report)

Evidence

Cluster 1 — PR Sous Chef schema-field hallucination

Failed runs:

• §25832686137 — update_pull_request with pr_number: 32012 rejected: Target is "*" but no pull_request_number specified
• §25831659910 — add_comment with pr_number: "32015" and pr: "32012" both rejected: Target is "*" but no item_number/issue_number specified
• §25824436959 — correct field names, GitHub API error on PR fix: broaden ARC/DinD detection to match any tcp:// DOCKER_HOST #31996 update (likely transient API hiccup, distinct sub-cluster)

agent_output.json snippets from the rejected runs:

{"body":"Updating branch to resolve mergeability.","pr_number":32012,"update_branch":true,"type":"update_pull_request"}
{"body":"Quick nudge: `@copilot` ...","pr_number":"32015","type":"add_comment"}
{"body":"pr-sous-chef: quick nudge ...","pr":"32012","type":"add_comment"}

The safe-outputs validator rejects, fails the dispatch step, and the entire workflow run is marked failure.

Cluster 2 — Design Decision Gate (matches #31178)

§25831509157 — Claude hit error_max_turns at turn 21 after 9 Bash permission denials trying to read pre-fetched data with cat, jq, python3, cp against paths under /tmp/gh-aw/agent/. Same pattern as #31178; cost = $1.04 (~1.1M tokens), all wasted on denied tool calls before max-turns terminated the run.

See #31178 for the canonical remediation discussion. No new tracker created.

Cluster 3 — Daily Model Inventory Checker silent CLI crash

§25833891456 — Copilot CLI v1.0.43 spawned (pid 167), exited 1 second later with:

attempt 1: process closed exitCode=1 duration=1s stdout=0B stderr=0B hasOutput=false
attempt 1: no output produced — not retrying (possible causes: binary not found,
  permission denied, auth failure, or silent startup crash)

Failure happened immediately after chroot setup (Chroot mode enabled - dropping CAP_SYS_CHROOT and CAP_SYS_ADMIN) and [entrypoint] Transferred /host/tmp/gh-aw ownership to chroot user (1001:1001). There was also a warning right before the crash: [entrypoint][WARN] Failed to transfer /host/home/runner/work/_temp/gh-aw/safeoutputs ownership to chroot user.

Possibly related to (but distinct from) the chroot node-missing pattern in #31370 — #31370 is specific to aw-gpu-runner-T4; Daily Model Inventory Checker runs on the default runner. Recommend a separate investigation: capture full stderr from the spawned copilot process (the harness currently records 0 bytes), and add a pre-flight node --version and ldd /usr/local/bin/copilot inside the chroot to surface the missing dependency.

Cluster 4 — Agentic Workflow Audit Agent pip-install timeout

§25828013390 — 35-minute wall time, Turns: 0 reported, 13M effective tokens. The agent's very first tool call was:

Bash: pip install --quiet pandas matplotlib seaborn 2>&1 | tail -5  (timeout 180000ms)

The install ran for 180+ seconds in the background. Subsequent python3 -c "import pandas" raised ModuleNotFoundError: No module named 'pandas' (install never completed or installed into a path not on sys.path). The agent then looped on TaskOutput(block=true, timeout=180000) waits until the workflow-level timeout terminated the job.

Recommended fixes:

• Pre-install pandas, matplotlib, seaborn into the runner image (or pin them in requirements.txt).
• Update the prompt to instruct the agent to skip plotting/statistics and stick to pure-stdlib analysis when external libs aren't available.
• Tighten the agent's first-turn budget to fail fast if heavy installs are attempted.

Cluster 5 — AI Moderator `hide_comment` ID format

§25830438415 — Agent emitted {type: "hide_comment", comment_id: "4445729864"} (a REST API numeric comment ID). The handler called GraphQL minimizeComment, which responded:

Failed to hide comment: Request failed due to following response errors:
 - Could not resolve to a node with the global id of '4445729864'

GraphQL requires a global node ID (e.g., IC_kwDOPc...). The sister add_labels item in the same run succeeded.

Recommended fixes:

• Server-side: convert REST numeric IDs to GraphQL node IDs in the hide_comment safe-output handler before calling minimizeComment.
• Schema-side: update the hide_comment MCP tool description to specify the required ID format and provide an example.

Existing Issue Correlation

• [aw-failures] Claude workflows hit max-turns burning every turn on Bash permission denials (Step Name Alignment, Design Decision Gate) #31178 — Bash-permission max-turns loop: Design Decision Gate continues to recur (run 25831509157, $1.04 cost). Updated with a recurrence note in this investigation.
• [aw-failures] Daily Issues Report Generator: 9-day chronic node-missing-in-chroot failure on aw-gpu-runner-T4 #31370 — chroot node-missing on aw-gpu-runner-T4: not directly hit today. Cluster 3 here is on a different runner but may share a root cause; flagged for separate analysis rather than merged into [aw-failures] Daily Issues Report Generator: 9-day chronic node-missing-in-chroot failure on aw-gpu-runner-T4 #31370.
• [aw-failures] Smoke Gemini agent step fails on every PR — fetch failed to Gemini API (generateContentStream / generateJson) #31575, [aw-failures] [aw] Fix aw-portfolio-yield: placeholder npm package + empty OTel endpoint break compile and runtime #31456, [aw-failures] [Issue Triage Agent] safe_outputs add_labels fails with No issue/PR number available on schedule events #31080, [aw-failures] Multi-Device Docs Tester exhausts 100-LLM-invocation API cap during multi-device sweep #31651, [aw-failures] jsweep — Copilot CLI step times out at 20m while staging the cleanup PR #31652 — none recurred in this 6h window.
• [aw-failures] [Schema Consistency Checker] Recurring max-turns failures since 2026-05-03 #30960, [aw-failures] [aw] Failure Report 2026-05-10 (6h): Go Logger toolchain mismatch + Schema lock-file drift (recurs #31178, #31221) #31314, [aw-failures] [aw] Failure Report 2026-05-07: Session Resume Regression + Schema Checker Max-Turns Escalation #30795, [aw-failures] [aw] Failure Report 2026-05-11 (6h): aw-portfolio-yield workflow defect blocks compile + own runtime + recurrences [Content truncated due to length] #31455 — older parent failure reports; no fresh evidence of resolution or recurrence in this window. Not closed (insufficient confidence).

Proposed Fix Roadmap

Priority	Action	Status
P0	Fix PR Sous Chef Copilot field-name hallucination	Sub-issue created (see below)
P1	Investigate Copilot CLI silent crash in chroot (Daily Model Inventory Checker)	Pending — blocked by create_issue: max 2 budget; tracked here
P1	Pre-install / pin data-analysis libs OR rewrite prompt for Audit Agent	Pending — tracked here
P1	hide_comment handler accepts REST numeric IDs (convert server-side)	Pending — tracked here
P2	Continue work on #31178 (Bash permission-denial loop)	Recurrence note appended

Sub-Issues Created

• Sub-issue (P0): Fix PR Sous Chef Copilot field-name hallucination — linked via parent: aw_failrep5.

Confidence & Unknowns

• High confidence: Clusters 1, 2, 4, 5 root causes are well-supported by logs and agent_output.json content.
• Medium confidence: Cluster 3 (Daily Model Inventory Checker) — the Copilot CLI emitted zero output, so the precise crash reason is unknown; needs better diagnostic capture.
• The 6h window is short; PR Sous Chef's 37.5% rate may not be representative of the longer-term failure rate.

References:

• §25832686137
• §25831509157
• §25828013390

[github-actions]

🎉 Automatically closed by Sub-Issue Closer

All sub-issues have been completed. This parent issue is now closed automatically.

Sub-issues status: 1/1 closed (100%)

• [aw-failures] Fix PR Sous Chef: Copilot CLI emits wrong safe-output field names (pr_number vs pull_request_number, pr/pr_number vs item_number [Content truncated due to length] #32040 [CLOSED]

Generated by Sub-Issue Closer · ● 10.4M · ◷