Skip to content

Preserve agent accounting in fallback artifacts - #61053

Merged
pelikhan merged 3 commits into
mainfrom
fix-60870-agent-aic-fallback
Sep 15, 2026
Merged

pelikhan merged 3 commits into
mainfrom
fix-60870-agent-aic-fallback

Conversation

@dsyme

@dsyme dsyme commented Sep 15, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • include agent execution evidence and engine usage summaries in agent-output-fallback
  • merge that fallback into conclusion processing so sampled no-inference evidence is staged in the primary usage artifact
  • preserve current and legacy firewall token-usage records when the large agent artifact is unavailable
  • verify a successful sampled agent with empty token usage accounts as zero instead of raising transient_error

The missing evals accounting reported in #60870 was fixed by #60869 and #60892. This PR addresses the distinct missing-agent-accounting failure found during that investigation and the successful sampled-run case documented on #61052.

Closes #61052

Copilot AI balanced review requested due to automatic review settings September 15, 2026 06:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

ARC/DinD fallback uploads still omit firewall accounting stored under the runner temporary directory.

Get a fresh assessment by requesting another Copilot review.

Review tier: Balanced (auto)
Findings: 1 Medium severity

Note

Copilot is running an experiment and ran this review at Balanced.

Open (1)
What changed in this PR

Preserves agent accounting when the primary agent artifact is unavailable.

Changes:

  • Adds execution, engine usage, and firewall records to the fallback artifact.
  • Extends compiler assertions for those paths.
  • Adds a patch changeset.
File Description
pkg/​workflow/​compiler_yaml_artifacts.go Adds accounting evidence to fallback uploads.
pkg/​workflow/​compiler_artifacts_test.go Verifies generated fallback paths.
.changeset/​patch-preserve-agent-aic-fallback.md Documents the fix.

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Comment thread pkg/workflow/compiler_yaml_artifacts.go Outdated
Comment on lines +79 to +81
constants.TmpGhAwDirSlash + "sandbox/firewall-audit-logs/api-proxy-logs/token-usage.jsonl",
constants.AWFProxyLogsDir.String() + "/api-proxy-logs/token-usage.jsonl",
constants.AWFAuditDir.String() + "/api-proxy-logs/token-usage.jsonl",
@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot Please take another pass on this PR and run the pr-finisher skill.

There is still an open Copilot review finding on the current work: preserve the fallback ARC/DinD artifact paths so runner temporary-directory firewall/accounting data is included when the primary agent artifact is unavailable.

Please either push the follow-up fix or summarize the remaining blocker for maintainers.

Recent review:

Run: https://github.com/github/gh-aw/actions/runs/34939252893

Generated by 👨‍🍳 PR Sous Chef · pi · gpt54 · 19.3 AIC · ⊞ 9K · ◷
Comment /souschef to run again

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot AI commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

@copilot Please take another pass on this PR and run the pr-finisher skill.

There is still an open Copilot review finding on the current work: preserve the fallback ARC/DinD artifact paths so runne...

Fixed in 615e5c5. The fallback now rewrites the complete path list after grader paths are appended, preserving ARC/DinD firewall and accounting data under ${{ runner.temp }}/gh-aw; focused regression coverage was added and workflow locks were regenerated.

@github-actions

Copy link
Copy Markdown
Contributor

Thanks @dsyme 👋 — this fix looks great! Preserving agent accounting in fallback artifacts when large agent artifacts are unavailable addresses a critical gap in observability for sampled agent runs. The clear description and focused scope make this easy to review.

The lock file updates are expected given the workflow infrastructure changes. This is ready for review.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • github.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "github.com"

See Network Configuration for more information.

Generated by ✅ Contribution Check · copilot · auto · 84.1 AIC · ⌖ 7.94 AIC · ⊞ 9.5K · ◷

@pelikhan
pelikhan merged commit 976492c into main Sep 15, 2026
42 checks passed
@pelikhan
pelikhan deleted the fix-60870-agent-aic-fallback branch September 15, 2026 22:01
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v0.89.17

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Preserve agent accounting evidence in fallback artifacts

5 participants