[aw-failures] [aw] Phantom asset: upload_assets fails despite agent success (missing staged PNGs)

[github-actions]

Problem statement

Asset-producing agentic workflows complete the agent job successfully but then fail the downstream upload_assets (Push assets) job with ERR_SYSTEM: Asset file not found: /tmp/gh-aw/safeoutputs/assets/<file>.png. The agent emits upload_asset safe-output items that reference PNG files — including sha and byte size — that were never staged to the directory the upload pipeline reads. The run is reported as a failure even though the agent's analysis succeeded.

Affected workflows and run IDs

1. Daily Code Metrics and Trend Tracking Agent — §27713375907 (quality_score_breakdown.png)
2. Daily Security Observability Report — §27705239494
3. The Daily Repository Chronicle — §27704205632
4. Daily Agent of the Day Blog Writer — §27699677975

Probable root cause

1. Path mismatch: the agent references assets under .../.gh-aw-assets/<file> while the upload_assets job reads /tmp/gh-aw/safeoutputs/assets/. No safe-outputs-assets artifact is produced by the agent job (Artifact not found for name: safe-outputs-assets), so the staging directory does not exist at push time.
2. There is no validation that an upload_asset safe-output points to a real, staged file at emission time, so the agent can declare a phantom asset (with fabricated sha/size and markdown image links) that the pipeline then cannot find.

Proposed remediation

1. Stage asset files to /tmp/gh-aw/safeoutputs/assets/ (or align the upload pipeline to read the agent's actual output path) before the upload_assets job runs, and ensure the safe-outputs-assets artifact is uploaded by the agent job.
2. Validate file existence when an upload_asset safe-output is emitted: reject/skip items whose referenced file is missing instead of failing the whole job, and surface a clear agent-facing error.
3. Make upload_assets fail-soft when some declared assets are missing (warn + upload what exists) so a single missing image does not fail an otherwise successful run.

Success criteria / verification

1. A re-run of Daily Code Metrics produces the declared PNGs in /tmp/gh-aw/safeoutputs/assets/ and the upload_assets job succeeds.
2. Emitting an upload_asset for a non-existent file produces a clear validation error at agent time, not an ERR_SYSTEM in a later job.
3. No Asset file not found failures across the four affected workflows over a subsequent 24h window.

Parent report: see linked parent issue. Analyzed runs: 27713375907, 27705239494, 27704205632, 27699677975.
Related to #39883

Generated by 🔍 [aw] Failure Investigator (6h) · ◷

• expires on Jun 24, 2026, 11:47 AM UTC-08:00

---

6h-window recurrence update — 2026-06-18 01:38 UTC

1. Still active. Two more workflows hit the identical signature in the last 6h, both on main via schedule:
   • Agentic Workflow Audit Agent — §27721789885 — ERR_SYSTEM: Asset file not found: /tmp/gh-aw/safeoutputs/assets/wh_trends.png
   • Daily Project Performance Summary Generator (MCP Scripts) — §27720651183 — ERR_SYSTEM: Asset file not found: /tmp/gh-aw/safeoutputs/assets/activity_overview.png
2. Root cause confirmed from the Push assets logs: the agent writes the PNGs to /tmp/gh-aw/agent/ (the upload step's own file listing shows /tmp/gh-aw/agent/activity_overview.png and /tmp/gh-aw/agent/velocity_metrics.png present), while the upload pipeline reads /tmp/gh-aw/safeoutputs/assets/. The staging directory is never populated — a path mismatch, matching this issue's hypothesis.
3. Regression timing (audit-diff success→fail): Agentic Workflow Audit Agent succeeded through 2026-06-15 (§27579121502) and began failing 2026-06-16. The failing run added matplotlib charting (2.88M tokens / 43 turns vs a near-zero-asset baseline), i.e. it now emits PNG assets that never reach the staging dir.
4. An active fix branch copilot/aw-failures-fix-upload-assets exists (the Smoke Copilot workflow_dispatch runs this window target it). Keep this issue open until that fix lands and a clean 24h window is observed.

References: §27721789885 · §27720651183 · §27579121502

Generated by 🔍 [aw] Failure Investigator (6h) · ◷

---

---

6h-window recurrence update — 2026-06-18 08:26 UTC

1. Still active. Two more main/schedule workflows hit the identical upload_assets / Push assets signature in the last 6h while the agent job succeeded:
   • Documentation Noob Tester — §27738455642 — ERR_SYSTEM: Asset file not found for cli-commands.png, confusing-token-setup.png, home.png, quick-start.png, step3-no-monitoring.png.
   • Daily Firewall Logs Collector and Reporter — §27734553421 — same upload_assets / Push assets failure, agent job success.
2. Sharper root cause (path-prefix mismatch, not just dir mismatch). In §27738455642 the same file is present under one prefix and reported missing under another: /tmp/gh-aw/safeoutputs/assets/home.png and /tmp/gh-aw/safeoutputs/assets/cli.png exist on disk, but the upload step looks under /home/runner/work/_temp/gh-aw/safeoutputs/assets/ and fails with ERR_SYSTEM: Asset file not found: /home/runner/work/_temp/gh-aw/safeoutputs/assets/home.png. The staging dir and the read dir differ by the /tmp/gh-aw vs /home/runner/work/_temp/gh-aw prefix — the agent stages some assets to the former while the upload pipeline resolves the latter. This refines the original ".gh-aw-assets vs /tmp/gh-aw/safeoutputs/assets" hypothesis: at least one path now stages correctly to safeoutputs/assets but under the wrong base prefix.
3. Remediation rejig docs #1 (align staging path / read path) and Add workflow: githubnext/agentics/weekly-research #3 (fail-soft on missing assets, warn + upload what exists) both still apply and would have made both runs pass — §27738455642 had cli.png/home.png staged and only some assets missing, yet the whole job failed.
4. Keep open until copilot/aw-failures-fix-upload-assets lands and a clean 24h window is observed.

References: §27738455642 · §27734553421

Generated by 🔍 [aw] Failure Investigator (6h) · ◷

---

6h-window recurrence update — 2026-06-18 13:53 UTC

Still active and unfixed. Six asset-producing workflows hit the identical phantom-asset signature in the last 6h, all agent job success + upload_assets (Push assets) failure:

1. GitHub MCP Structural Analysis — §27760805416 — All 4 declared assets were missing; no assets published (toolset_sizes.png, usefulness_ratings.png, daily_trend.png, size_vs_usefulness.png).
2. Daily Agentic Workflow AIC Usage Audit — §27759742314.
3. Copilot PR Conversation NLP Analysis — §27756469234.
4. Copilot Agent Prompt Clustering Analysis — §27755055618.
5. daily-experiment-report — §27749537944.
6. Copilot Session Insights — §27746306280.

Confirmed signature: the Push assets step checks out the assets branch, finds the declared PNGs absent from /home/runner/work/_temp/gh-aw/safeoutputs/assets/, emits one ##[warning]ERR_SYSTEM: Asset file not found per file, then fails the job with ##[error]All N declared assets were missing; no assets published. Matches this issue's path-mismatch root cause exactly. No clean 24h window yet — keep open until the staging fix lands.

References: §27760805416 · §27759742314 · §27746306280

Generated by 🔍 [aw] Failure Investigator (6h) · ◷

[github-actions]

🍪 Issue Monster selected this for Copilot

I've identified this issue as a good candidate for automated resolution and requested assignment to the Copilot coding agent.

If assignment succeeds, the Copilot coding agent will analyze the issue and create a pull request with the fix.

Om nom nom! 🍪

🍪 Om nom nom by Issue Monster · ◷

[copilot-swe-agent]

The agent encountered an error and was unable to start working on this issue: Please try again later, or contact support if the problem persists. (Request id: 1F01:294C4:108268B:3CFF281:6A330268)