Skip to content

[aw-failures] [aw] Copilot CLI exits 127 — node missing inside AWF chroot (chronic Daily News failure on main) #40074

Description

@github-actions

Problem statement

Fix the AWF chroot Node.js bootstrap: when chroot mode is enabled, the Copilot CLI engine cannot find node inside /host and the agent job exits 127 before a single turn runs. The entrypoint prints [entrypoint][ERROR] Copilot CLI requires Node.js, but 'node' is not available inside AWF chroot. and Command completed with exit code: 127. Distinct from #39946 (Copilot exits 1 with output and turns>0); here turns=0, tool_types=0, token_usage=0 — the engine never starts.

Affected workflows and run IDs

  1. Daily News§27751720901 (main, schedule, 2026-06-18 09:59 UTC) — engine copilot / claude-sonnet-4.6.
  2. Daily News§27760141601 (branch copilot/debug-daily-news-failure, workflow_dispatch) — same signature.

Chronic: Daily News has failed on main every scheduled run for 14+ consecutive days with no successful baseline — 27681588513, 27610721754, 27540688430, 27408472399, 27338873553, 27268191506, 27198178835, 27130454325, 27008097775, 26944476134, ... all failure.

Probable root cause

  1. The chroot entrypoint resolves node via GH_AW_NODE_BIN then command -v node; inside /host neither is populated, so the harness exits 127 with node runtime missing on this runner — check runtimes.node in workflow YAML. The host runner has Node, but the toolcache bin dirs are not on PATH inside the chroot for this workflow.
  2. audit-diff of two failing runs (§27751720901 vs §27681588513) shows zero drift: run1_token_usage=0, run2_token_usage=0, no firewall anomalies — confirming the engine never executes any LLM turn, consistent across the chronic streak.

Proposed remediation

  1. In chroot mode, ensure the host Node.js install (/opt/hostedtoolcache/node/**/bin, /home/runner/work/_tool/**, nvm path) is discovered and bind-mounted/added to PATH inside /host before launching copilot_harness.cjs, or set GH_AW_NODE_BIN to the resolved host node.
  2. Alternatively, pin runtimes.node in the Daily News workflow YAML so the chroot bootstrap installs/locates a known node path.
  3. Surface the exit-127 node missing entrypoint error as a classified failure class (engine_runtime_missing) so chronic streaks are flagged rather than reported as a bare exit 127.

Success criteria / verification

  1. A re-run of Daily News on main reaches at least one LLM turn (token_usage>0) and the agent job no longer exits 127.
  2. node is resolvable inside the AWF chroot for Copilot-engine workflows; the Copilot CLI requires Node.js ... not available inside AWF chroot entrypoint error no longer appears.
  3. Daily News produces at least one successful (or correctly-classified non-127) run on main within a subsequent 48h window.

Parent report: #39883. Analyzed runs: 27751720901, 27760141601. Comparator (drift): 27751720901 → 27681588513.
Related to #39883

Generated by 🔍 [aw] Failure Investigator (6h) ·

  • expires on Jun 25, 2026, 6:03 AM UTC-08:00

Resolved — Daily News agent step recovered.

As of §27799081331 (2026-06-19 01:06 UTC) the Daily News agent job runs to completion: audit reports 42 turns / 2,394,762 tokens and the agent job conclusion is success. audit-diff vs the prior failing run §27760141601 (the run cited in this issue) shows turns 0 → 42, token_usage 0 → 2.39M — the exit-127 "node missing inside AWF chroot / engine never starts" signature is gone, and this issue's success criteria (token_usage>0, ≥1 LLM turn, no exit 127) are met.

Daily News is still red overall, but for a new, unrelated cause: the push_repo_memory job rejects the orphan memory/daily-news branch's unsigned first commit under the verified signatures ruleset (GH013). That is tracked separately in #aw_memfix.

Closing as fixed. (Original chroot/node-bootstrap analysis preserved in this issue's edit history.)

Generated by 🔍 [aw] Failure Investigator (6h) · 205.4 AIC · ⌖ 12.6 AIC · ⊞ 4.9K ·

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions