Skip to content

[aw-failures] [aw-fix] P1: Agent Performance Analyzer — repo-memory exceeds 12 KB, agent idle-loops on push_repo_memory rejectio [Content truncated due to length] #48009

Description

@github-actions

Fix the repo-memory pruning path — a recoverable size-limit rejection is being turned into a hard run failure by an idle-loop → watchdog SIGTERM.

Executive summary

Do this: on a push_repo_memory size-exceeded rejection, prune/summarize the store and retry (or exit cleanly) instead of idle-probing the tool until the copilot-harness watchdog kills the process. This single defect converts an expected, recoverable condition into a full run failure.

Failure cluster

Cluster Sev Workflow Signature Representative run
B P1 Agent Performance Analyzer - Meta-Orchestrator push_repo_memory rejected: total memory 15 KB > 12 KB limit → agent idle → post-result watchdog SIGTERM §30158812091

Problem statement

The Execute GitHub Copilot CLI step in Agent Performance Analyzer - Meta-Orchestrator (.github/workflows/agent-performance-analyzer.lock.yml) fails the run. push_repo_memory is rejected because the repo-memory store exceeds its size cap; the agent then loops probing the tool, emits empty assistant messages, goes idle, and the harness watchdog terminates it.

Affected workflows and run IDs

  • Agent Performance Analyzer - Meta-Orchestrator — run §30158812091 (2026-07-25T12:56Z)

Probable root cause

Root cause is confirmed from the run logs:

  1. push_repo_memory returns: Total memory size (15 KB) exceeds the allowed limit of 12 KB (configured max-patch-size: 10 KB).
  2. ls/du of /tmp/gh-aw/repo-memory/default/ shows the metrics/ subdirectory alone at 44 KB on disk, plus shared-alerts.md (4.9 KB), workflow-health-latest.md (4.4 KB), agent-performance-latest.md (3.6 KB) — the store has grown past the cap.
  3. The agent does not prune; it re-probes push_repo_memory variants and produces empty assistant turns.
  4. [copilot-harness] attempt 1: post-result watchdog terminating idle process after 20244ms (SIGTERM)failureClass=permission_denied permissionDeniedCount=6AI credits budget exceeded — not retrying (non-retryable guard condition) → run exits code 1.

The real defect is the idle-loop → watchdog path: a size-exceeded rejection is an expected, recoverable outcome, but there is no prune-and-retry (or clean-exit) branch, so the agent stalls into a SIGTERM.

Proposed remediation

  1. Primary: In the workflow's memory-write logic, on a size-exceeded rejection, rotate/summarize metrics/ and truncate *-latest.md history to bring the store under the cap, then re-call push_repo_memory. If still over after one prune, exit the memory step cleanly (log a warning) rather than looping.
  2. Guardrail: Cap metrics/ growth at the source (rolling window / summarized aggregates) so the 44 KB accumulation cannot recur.
  3. Optional: If a larger store is genuinely required, raise max-patch-size for this workflow's repo-memory config — but pruning is preferred over unbounded growth.
  4. Harness hardening (track separately if needed): an idle agent after a tool rejection should surface the rejection as a terminal error, not stall until the watchdog fires and mislabels it permission_denied / AI credits budget exceeded.

Success criteria / verification

  • Agent Performance Analyzer completes without a push_repo_memory size-exceeded error, OR prunes and pushes successfully within the cap.
  • No post-result watchdog terminating idle process ... SIGTERM on this workflow.
  • Run conclusion success; no permission_denied / AI credits budget exceeded misclassification for this cause.
Other clusters observed this window (report-only, no issue)
  • Cluster A — startup_failure ×3 (Auto-Triage Issues, PR Triage Agent, Issue Monster), 12:29–12:33Z. Audit of §30158163884 shows the agent partially ran (5 turns, 11.4k tokens, error_count=0) yet concluded startup_failure; baseline cohort match to a success run 30148485921 with only posture_changed + run_unsuccessful deltas. Three concurrent scheduled workflows failing at startup with no per-step errors points to transient GitHub Actions platform flakiness, not a code defect — no actionable remediation. Watch for recurrence.
  • Cluster C — PR Sous Chef pre_activation failed, agent skipped (§30157572332). P2, benign activation-gate behavior; agent never ran.
Existing issue correlation

Reviewed 6 open agentic-workflows issues (#47525, #47753, #47659, #47135, #46647, #43499). None match the repo-memory size-limit signature or the startup_failure cluster. No fresh evidence that any existing issue is fixed/stale, so none were closed (gh read access unavailable this run; relied on prefetch state).

References:

Generated by 🔍 [aw] Failure Investigator (6h) · age00 · 129.7 AIC · ⌖ 34.3 AIC · ⊞ 5.2K ·

  • expires on Aug 1, 2026, 5:21 AM UTC-08:00

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions