Skip to content

[aw-failures] [aw] Failure Investigation Report — 2026-04-28 (6h window) #28869

Description

@github-actions

Executive Summary

This investigation covers 01:35–07:35 UTC on 2026-04-28. Out of 42 total runs in the window, 6 concluded as failures across 3 root-cause clusters. No previously-open issues appear fixed or stale.

Failure Clusters

Priority Cluster Workflow(s) Failures Root Cause Existing Issue
P0 A Smoke CI 3 safeoutputs: command not found (exitCode=127) in copilot chroot #28858
P1 B Daily Hippo Learn, Code Simplifier 2 Threat detection model not emitting THREAT_DETECTION_RESULT #28866
P1 C Contribution Check 1 safe_outputs job failure (root cause unclear) —

Also tracked: gpclean exit-code-22 failure in #28855 (outside 6h scope).

Evidence

Cluster A — Smoke CI: safeoutputs not on PATH in copilot chroot (3 runs)

All 3 runs fail identically on the first attempt with no partial execution; all retries exhaust:

[entrypoint][WARN] Failed to transfer /host/home/runner/work/_temp/gh-aw/safeoutputs ownership to chroot user
...
[copilot-harness] attempt 1: spawning: /tmp/gh-aw/engine-command.sh --add-dir /tmp/gh-aw/ ...
--add-dir: line 1: safeoutputs: command not found
[copilot-harness] attempt 1: process exit event exitCode=127
[copilot-harness] all 3 retries exhausted — giving up (exitCode=127)
  • Engine: GitHub Copilot CLI v1.0.36, firewall v0.25.28
  • Run timestamps: 05:21Z, 05:32Z, 05:56Z — all within 35 minutes
  • Deterministic failure: retrying never changes the outcome
Cluster B — Threat detection parse failure (2 runs)

Both workflows ran their agent successfully (18 turns, 543k and 874k tokens) but failed in the detection job:

📊 Detection log stats: 101 lines, 4885 bytes
📄 No lines containing THREAT_DETECTION_RESULT found in 101 lines
##[error]❌ Failed to parse detection result: No THREAT_DETECTION_RESULT found in detection log.
Expected: THREAT_DETECTION_RESULT:{"prompt_injection":bool,"secret_leak":bool,"malicious_patch":bool,"reasons":[...]}

Both runs share identical detection log stats (101 lines, 4885 bytes) — suggesting the same input content or a reproducible formatting failure. GH_AW_DETECTION_CONTINUE_ON_ERROR: true was set but the detection job conclusion is still failure, propagating to workflow failure.

Cluster C — Contribution Check: safe_outputs failure (1 run)

Run §25036239526 — 29 turns, 998k tokens. Agent ran to completion and produced a safe-outputs-items.zip artifact (364 bytes). The safe_outputs job is listed as failed. Tail logs show only successful artifact upload and git cleanup, so the error is in an earlier step. Output variable names (create_discussion_error_count, code_push_failure_count) suggest a failed GitHub write operation. Root cause requires deeper log inspection.

Existing Issue Correlation

Issue Status Action
#28858 Smoke CI failed Open Covers only first run; 2 additional runs confirmed identical failure
#28866 Detection Runs Open General tracking issue; updated with specific run evidence
#28855 gpclean failed Open Pre-6h; exit code 22; no new evidence
#28861 Workflows out of sync Open Lock file drift; unrelated to current failures
#28857 Q fix Open Pending manual PR; unrelated

No issues are fixed or stale based on current evidence.

Proposed Fix Roadmap

Priority Fix Issue
P0 Fix safeoutputs binary not on PATH in copilot chroot #28871
P1 Investigate detection model format compliance — check prompt truncation or output format reinforcement —
P1 Diagnose Contribution Check safe_outputs failure — inspect full step logs for the write operation error —

Sub-issues Created

References:

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions