fix(release): a shared storage quota no longer takes down every nightly - #50
Merged
Merged
Conversation
Every scheduled release since 2026-08-11 failed with "Failed to CreateArtifact: Artifact storage quota has been hit". Actions storage is an org-wide quota billed by GB-hour, and this org sits at 2.04 GB against the 2.00 GB its plan includes, so all artifact uploads are blocked account-wide. Two release jobs treated that upload as fatal, and losing them skipped the whole build matrix, the npm publish, and the release itself. Connect was already designed to be optional here, so the tracing config handoff now degrades the way the surrounding code says it should: the upload and the three downloads are best-effort, and the loader tells a missing artifact apart from a Connect-less setup, warning on the former so an untraced production build does not look identical to a healthy one. The config keeps travelling as an artifact rather than a job output because it carries a token the loader has to mask. Separately, the workflow had no concurrency group at all. When Blacksmith had no Ubuntu capacity from 08-03 to 08-06, 23 nightly runs stacked up instead of superseding each other and sat about 480 VM-hours waiting for runners that never arrived, each dying at GitHub's 24h queue cap. Note that timeout-minutes does not help here, since it only counts execution time. Nightlies now share one group; tag releases and manual dispatches key off run_id so they stay unique and are never cancelled. Also drops resource-monitor artifacts from 7-day to 1-day retention, the only release artifact held above the minimum, since it is consumed within the same run. Verified with actionlint: 13 findings before and after, all pre-existing (custom Blacksmith runner labels, run_started_at, SC2129 style). Model: Claude Opus 5. Harness: Claude Code.
Thread transfer impact
This comment will update automatically after the next completed run. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every scheduled release since 2026-08-11 has failed with:
Actions storage is an org-wide quota billed by GB-hour. Per the billing API this org is at 1,488 GB-hours ≈ 2.04 GB-month against the 2.00 GB its plan includes, with net spend at
$0.00— so GitHub hard-blocks uploads rather than billing four cents of overage. Two release jobs (Resolve T3 Connect public config,Build WSL node-pty) treated that upload as fatal, and losing them skipped the entire build matrix, the npm publish, the GitHub release, and the Discord announce.ci.ymlalready learned this lesson after PR #9 and made its uploadcontinue-on-error. The release workflow never got the same treatment.What changed
Tracing config degrades instead of failing. Connect is already optional here — the surrounding comments say the desktop apps build and auto-update without it. The upload and all three downloads are now best-effort, and the loader distinguishes a missing artifact from a Connect-less setup, emitting a
::warning::on the former so a silently untraced production build doesn't look identical to a healthy one. It stays an artifact rather than becoming a job output because it carriesT3CODE_RELAY_CLIENT_OTLP_TRACES_TOKEN, which the loader has to::add-mask::.A concurrency group, which the workflow had none of. When Blacksmith had no Ubuntu capacity from 08-03 to 08-06, 23 nightly runs stacked up instead of superseding each other and sat ~480 VM-hours waiting for runners that never arrived — each dying at GitHub's 24h queue cap with
runner_name: ""andsteps: 0.timeout-minutesdoes not help here; it only counts execution time. Proof from this repo's own history:Mobile Native Static Analysishastimeout-minutes: 10and still sat queued for 262 minutes on 08-12.Nightlies now share one group. Tag releases and manual dispatches key off
run_idso they stay unique and are never cancelled — losing a real release to a scheduling race is far worse than paying for a duplicate build.Retention.
resource-monitor-*drops from 7 days to 1, the only release artifact held above the minimum, and it's consumed within the same run.Note
This reduces the blast radius; it does not restore nightlies on its own. The org's Actions spending limit still needs raising above
$0— that's a billing setting, not a code change.Verification
actionlintreports 13 findings before and after, identical categories and counts — all pre-existing (custom Blacksmith runner labels,run_started_at, SC2129 style). No new findings introduced. Workflow YAML re-parsed and structurally checked;vp fmtclean.Model: Claude Opus 5. Harness: Claude Code.
Need help on this PR? Tag
@codesmithwith what you need. Autofix is disabled.