Skip to content

[aw-failures] [P0] Failure-Investigator log capture grabs post-job cleanup tail, not the actual error #52501

Description

@github-actions

Fix the log-tail capture — it's blinding root-cause analysis for most failures this window

Anchor error-log capture to the failing step, not the job's absolute tail. In this run's 6h window (2026-08-13 ~06:38–12:38 UTC), 4 of 5 detailed failures had truncated_error_logs[].tail_lines (last 50 lines) containing only post-failure cleanup/teardown output — never the actual error from the failing step.

Affected workflows and runs

Run Workflow Failing step What the tail actually captured
§31697879776 PR Sous Chef Start DIFC Proxy artifact upload, git config unset, job outputs
§31695362450 Daily Max Ai Credits Test Execute GitHub Copilot CLI OTLP skip notice, git config unset, job outputs
§31695218197 Daily Choice Type Test Execute Claude Code CLI git config unset, orphan-process termination
§31684831832 Daily AW Cross-Repo Compile Check Execute Claude Code CLI git config unset, orphan-process termination

Only 1 of 5 runs this window (§31700716464, the Bun segfault in #51984) had a usable error line — and only because the crash happened seconds before teardown started, so the panic text hadn't yet scrolled out of the last-50-lines window.

Probable root cause

The tail capture grabs the last N lines of the entire job log, not a window anchored to the failing step's own log range. Jobs that run substantial cleanup/teardown after a failing step (artifact upload, git config unset, OTLP export, orphan-process cleanup) push the real ##[error] line and surrounding context out of the captured tail before truncation happens.

Proposed remediation

  1. Anchor capture to the failing step: use its start/end log offsets (or the nearest ##[error] marker) as the truncation window, not the job's absolute end.
  2. If a step-anchored window isn't available, search backward from the tail for the first ##[error] line and capture N lines centered on it, falling back to the absolute tail only if no ##[error] is found.
  3. Add a regression check: flag prefetch entries whose tail_lines contain no error/panic/exception/##[error] token as "capture likely missed the fault" so downstream classifiers don't silently treat cleanup noise as the failure signature.

Success criteria / verification

  • Next Failure Investigator run captures the actual failing-step error line for ≥90% of failed runs where the failing step is not the job's last step.
  • The 4 runs listed above (or their next occurrences) show real error text instead of cleanup-only tails once the fix ships.

Parent report: #52395
Related: #52395, #51984
Related to #52395

Generated by 🔍 [aw] Failure Investigator (6h) · agent · 190.4 AIC · ⌖ 56.1 AIC · ⊞ 5.3K ·

  • expires on Aug 20, 2026, 5:25 AM UTC-08:00

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions