Skip to content

fix(release): a shared storage quota no longer takes down every nightly - #50

Merged
rynfar merged 1 commit into
pylonfrom
fix/release-artifact-quota-resilience
Aug 20, 2026
Merged

fix(release): a shared storage quota no longer takes down every nightly#50
rynfar merged 1 commit into
pylonfrom
fix/release-artifact-quota-resilience

Conversation

@rynfar

@rynfar rynfar commented Aug 18, 2026

Copy link
Copy Markdown
Collaborator

Every scheduled release since 2026-08-11 has failed with:

##[error]Failed to CreateArtifact: Artifact storage quota has been hit.

Actions storage is an org-wide quota billed by GB-hour. Per the billing API this org is at 1,488 GB-hours ≈ 2.04 GB-month against the 2.00 GB its plan includes, with net spend at $0.00 — so GitHub hard-blocks uploads rather than billing four cents of overage. Two release jobs (Resolve T3 Connect public config, Build WSL node-pty) treated that upload as fatal, and losing them skipped the entire build matrix, the npm publish, the GitHub release, and the Discord announce.

ci.yml already learned this lesson after PR #9 and made its upload continue-on-error. The release workflow never got the same treatment.

What changed

Tracing config degrades instead of failing. Connect is already optional here — the surrounding comments say the desktop apps build and auto-update without it. The upload and all three downloads are now best-effort, and the loader distinguishes a missing artifact from a Connect-less setup, emitting a ::warning:: on the former so a silently untraced production build doesn't look identical to a healthy one. It stays an artifact rather than becoming a job output because it carries T3CODE_RELAY_CLIENT_OTLP_TRACES_TOKEN, which the loader has to ::add-mask::.

A concurrency group, which the workflow had none of. When Blacksmith had no Ubuntu capacity from 08-03 to 08-06, 23 nightly runs stacked up instead of superseding each other and sat ~480 VM-hours waiting for runners that never arrived — each dying at GitHub's 24h queue cap with runner_name: "" and steps: 0.

timeout-minutes does not help here; it only counts execution time. Proof from this repo's own history: Mobile Native Static Analysis has timeout-minutes: 10 and still sat queued for 262 minutes on 08-12.

Nightlies now share one group. Tag releases and manual dispatches key off run_id so they stay unique and are never cancelled — losing a real release to a scheduling race is far worse than paying for a duplicate build.

Retention. resource-monitor-* drops from 7 days to 1, the only release artifact held above the minimum, and it's consumed within the same run.

Note

This reduces the blast radius; it does not restore nightlies on its own. The org's Actions spending limit still needs raising above $0 — that's a billing setting, not a code change.

Verification

actionlint reports 13 findings before and after, identical categories and counts — all pre-existing (custom Blacksmith runner labels, run_started_at, SC2129 style). No new findings introduced. Workflow YAML re-parsed and structurally checked; vp fmt clean.

Model: Claude Opus 5. Harness: Claude Code.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith with what you need. Autofix is disabled.

Every scheduled release since 2026-08-11 failed with "Failed to
CreateArtifact: Artifact storage quota has been hit". Actions storage is
an org-wide quota billed by GB-hour, and this org sits at 2.04 GB against
the 2.00 GB its plan includes, so all artifact uploads are blocked
account-wide. Two release jobs treated that upload as fatal, and losing
them skipped the whole build matrix, the npm publish, and the release
itself.

Connect was already designed to be optional here, so the tracing config
handoff now degrades the way the surrounding code says it should: the
upload and the three downloads are best-effort, and the loader tells a
missing artifact apart from a Connect-less setup, warning on the former so
an untraced production build does not look identical to a healthy one. The
config keeps travelling as an artifact rather than a job output because it
carries a token the loader has to mask.

Separately, the workflow had no concurrency group at all. When Blacksmith
had no Ubuntu capacity from 08-03 to 08-06, 23 nightly runs stacked up
instead of superseding each other and sat about 480 VM-hours waiting for
runners that never arrived, each dying at GitHub's 24h queue cap. Note
that timeout-minutes does not help here, since it only counts execution
time. Nightlies now share one group; tag releases and manual dispatches
key off run_id so they stay unique and are never cancelled.

Also drops resource-monitor artifacts from 7-day to 1-day retention, the
only release artifact held above the minimum, since it is consumed within
the same run.

Verified with actionlint: 13 findings before and after, all pre-existing
(custom Blacksmith runner labels, run_started_at, SC2129 style).

Model: Claude Opus 5. Harness: Claude Code.
@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:M labels Aug 18, 2026
@github-actions

Copy link
Copy Markdown

Thread transfer impact

⚠️ The latest CI run did not produce a thread transfer result for 30d499a.

This comment will update automatically after the next completed run.

@rynfar
rynfar merged commit 8dd0d6e into pylon Aug 20, 2026
11 checks passed
@rynfar
rynfar deleted the fix/release-artifact-quota-resilience branch August 20, 2026 18:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant