fix(ci): stop the nightly release job failing on an immutable tag - #661
Conversation
GitHub immutable releases are enabled on this repository, so the tag of a published release can never be deleted or moved. The release job's delete-then-recreate flow therefore could not succeed once a release existed for the same date: the tag delete was refused with GH013 "Cannot delete this tag", the warning was swallowed, and the subsequent push died on "already exists", failing the job with every build green. Make tagging idempotent (reuse a tag that already points at this commit, try to move one that does not, tolerate a refusal) and only replace a release that is still mutable, skipping with a warning when it is immutable. Fixes the failure in run 30528566163.
📝 WalkthroughWalkthroughThe nightly workflow now handles existing nightly tags idempotently and avoids deleting immutable GitHub Releases. Mutable releases are deleted before replacement, while blocked tag moves and immutable releases allow the workflow to complete successfully. ChangesNightly release lifecycle
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant GitHubActions
participant GitRemote
participant GitHubRelease
participant DigitalOceanSpaces
GitHubActions->>GitRemote: Query nightly tag and current commit SHA
GitRemote-->>GitHubActions: Return tag target
GitHubActions->>GitRemote: Create or force-update nightly tag
GitHubActions->>GitHubRelease: Query release by nightly tag
GitHubRelease-->>GitHubActions: Return release id and immutable flag
GitHubActions->>GitHubRelease: Delete mutable release before recreation
GitHubActions->>DigitalOceanSpaces: Use Spaces for immutable-release artifacts
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
| RELEASE_ID=$(gh api "repos/${GITHUB_REPOSITORY}/releases/tags/${TAG}" -q '.id' 2>/dev/null || true) | ||
| if [ -n "$RELEASE_ID" ]; then | ||
| IMMUTABLE=$(gh api "repos/${GITHUB_REPOSITORY}/releases/tags/${TAG}" -q '.immutable' 2>/dev/null || true) | ||
| if [ "$IMMUTABLE" = "true" ]; then | ||
| echo "::warning::Release $TAG is already published and immutable, so it cannot be replaced. Skipping the GitHub release step; this run's artifacts are on DigitalOcean Spaces, which the release notes already name as the canonical download location." | ||
| exit 0 | ||
| fi | ||
| echo "Replacing the existing (mutable) release $TAG." | ||
| gh release delete "$TAG" --yes 2>/dev/null || true | ||
| fi |
There was a problem hiding this comment.
🔍 Immutability detection depends on the .immutable field being present and equal to "true"
The skip decision hinges entirely on gh api ... -q '.immutable' returning the literal string true. If the field is absent or the second gh api call fails transiently, IMMUTABLE is empty/null and the script falls through to gh release delete (silenced by || true) followed by gh release create, which is exactly the original failing sequence (::error:: already exists). A single API call reused for both .id and .immutable (e.g. gh api ... -q '.id + " " + (.immutable|tostring)') would remove the second round-trip and the possibility of the two reads disagreeing. Note also that on a 404 gh may print null for -q '.id', making RELEASE_ID non-empty; that path is harmless because IMMUTABLE is then also null and the delete-then-create behaves as before.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1da712bdb1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if git push --force origin "refs/tags/$TAG"; then | ||
| echo "Moved $TAG to $SHA." | ||
| else | ||
| echo "::warning::Could not move $TAG; it is protected by an immutable release. Leaving the published tag at $TAGGED." | ||
| fi |
There was a problem hiding this comment.
In the checked Tag commit for nightly step, this branch treats every nonzero result from git push --force as immutable-release protection, even though git push -h only defines --force as “force updates” and the command can also fail because of authentication, connectivity, hooks, or another tag rule. If such a failure occurs and the subsequent GitHub API calls succeed, the next step can create or replace a mutable release using the old remote tag while uploading artifacts and notes for ${{ github.sha }}, producing contradictory release provenance while the job reports success. Verify that the existing release is immutable before suppressing the push error; otherwise propagate the failure.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
.github/workflows/nightly.yml (1)
730-735: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win
TAGis recomputed independently of the tagging step'sDATE.This step derives
TAGvia a freshdate +'%Y-%m-%d'call (line 735) rather than reusing the value computed in "Tag commit for nightly" (lines 688-690). If the job happens to straddle a UTC midnight between the two steps, the release step would check/replace a different tag than the one just created/moved, breaking the very idempotency contract this PR establishes.♻️ Suggested fix: compute the date/tag once and share it
- name: Tag commit for nightly + id: tag run: | DATE=$(date +'%Y-%m-%d') TAG="nightly-${DATE}" + echo "tag=$TAG" >> "$GITHUB_OUTPUT" ... - name: Publish or update nightly release env: VERSION: ${{ needs.check-for-changes.outputs.version }} SHORT_SHA: ${{ needs.check-for-changes.outputs.short_sha }} + TAG: ${{ steps.tag.outputs.tag }} run: | - TAG="nightly-$(date +'%Y-%m-%d')"🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/nightly.yml around lines 730 - 735, Update the nightly workflow so the “Publish or update nightly release” step reuses the date or tag produced by the “Tag commit for nightly” step instead of invoking date again. Preserve the existing nightly tag format and ensure both tagging and release publishing reference the same shared value across a UTC midnight.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/nightly.yml:
- Around line 766-775: The release lookup logic around RELEASE_ID must
distinguish a missing release from authentication, rate-limit, transient, or
other API failures. Remove the unconditional suppression from the gh api calls,
preserve the lookup response for checking .id and .immutable, and fail the
workflow for any non-404 lookup error; retain the existing immutable skip and
mutable-release deletion behavior, while allowing only a confirmed “not found”
result to proceed to release creation.
- Around line 701-708: Update the REFS lookup in the nightly tag flow to
preserve and check the exit status of git ls-remote separately from its output.
Only create and push the tag when the query succeeds with no matching refs; when
the command fails due to communication, authentication, or another error, stop
the workflow and report the failure instead of treating it as a missing tag.
---
Nitpick comments:
In @.github/workflows/nightly.yml:
- Around line 730-735: Update the nightly workflow so the “Publish or update
nightly release” step reuses the date or tag produced by the “Tag commit for
nightly” step instead of invoking date again. Preserve the existing nightly tag
format and ensure both tagging and release publishing reference the same shared
value across a UTC midnight.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 99677f97-d89e-410b-ac30-4770838e3401
📒 Files selected for processing (1)
.github/workflows/nightly.yml
| REFS=$(git ls-remote --tags origin "refs/tags/$TAG" "refs/tags/$TAG^{}" || true) | ||
|
|
||
| if [ -z "$REFS" ]; then | ||
| echo "No existing remote tag $TAG; creating it at $SHA." | ||
| git tag -f -a "$TAG" -m "Nightly build $DATE" | ||
| git push origin "refs/tags/$TAG" | ||
| exit 0 | ||
| fi |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== workflow context =="
if [ -f .github/workflows/nightly.yml ]; then
nl -ba .github/workflows/nightly.yml | sed -n '670,735p'
else
echo "myfile not found"
fi
echo
echo "== git command semantics probe =="
python3 - <<'PY'
import subprocess, textwrap, tempfile, pathlib, os
def run(env=None, check=None, user_env=None):
cmd = ['git'] + list(env if env else [])
merged = os.environ.copy()
if user_env:
merged.update(user_env)
merged.pop('GIT_SSH_COMMAND', None)
merged.pop('GIT_ASKPASS', None)
res = subprocess.run(cmd, capture_output=True, text=True, check=False, env=merged)
return res.returncode, '\n'.join(res.stdout.splitlines()[:3]), res.stderr.splitlines()[:3]
print("local repo ls-remote exit:", "success" if run(['ls-remote', '--tags'], user_env={'GIT_SSH_COMMAND':'true'})[0]==0 else "fail")
print("repoless exact pattern exits non-zero?", end=" ")
rc = subprocess.run(['git','ls-remote','--tags','origin','refs/tags/nightly-2026-07-20'], text=True).returncode
print("non-zero" if rc else "zero")
print("repoless two patterns exits non-zero?", end=" ")
rc = subprocess.run(['git','ls-remote','--tags','origin','refs/tags/nightly-2026-07-20','refs/tags/nightly-2026-07-20^{}'], text=True).returncode
print("non-zero" if rc else "zero")
PYRepository: WebFirstLanguage/wfl
Length of output: 221
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== locate workflow =="
git ls-files .github/workflows/nightly.yml || true
if [ -f .github/workflows/nightly.yml ]; then
echo "== workflow lines 660-740 =="
awk 'NR>=660 && NR<=740 {printf "%6d\t%s\n", NR, $0}' .github/workflows/nightly.yml
fi
echo
echo "== git command semantics probe =="
python3 - <<'PY'
import subprocess, os
print("git_lsremote_no_repo_exact_exit_nonzero:", subprocess.run(["git","ls-remote","--tags","origin","refs/tags/DNE-EXACT"], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).returncode != 0)
print("git_lsremote_no_repo_two_patterns_exit_nonzero:", subprocess.run(["git","ls-remote","--tags","origin","refs/tags/DNE-EXACT","refs/tags/DNE-EXACT^{}"], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).returncode != 0)
PYRepository: WebFirstLanguage/wfl
Length of output: 4790
🌐 Web query:
Git documentation git ls-remote exit code remote errors pattern no match
💡 Result:
By default, the git ls-remote command returns an exit code of 0 if it successfully communicates with the remote repository, regardless of whether any references (refs) match your provided patterns [1][2]. To distinguish between successful communication and the absence of matching references, you must use the --exit-code flag [1][3]. When --exit-code is provided, the command will exit with status 2 if no matching references are found [1][4][2]. Regarding pattern matching, git ls-remote interprets patterns as glob patterns [2][5]. These patterns are matched against the tail of a reference name, meaning they either match the full name (e.g., refs/heads/foo) or match a component starting after a slash separator (e.g., bar matches refs/heads/bar but not refs/heads/foobar) [2][6]. If you provide multiple patterns, the command will display any reference that matches at least one of the specified patterns [2][6].
Citations:
- 1: https://git-scm.com/docs/git-ls-remote.html
- 2: https://git-scm.com/docs/git-ls-remote
- 3: https://public-inbox.org/git/20110512120536.GA22033@camk.edu.pl/T/
- 4: https://code.googlesource.com/git/+/HEAD/Documentation/git-ls-remote.adoc
- 5: https://www.kernel.org/pub/software/scm/git/docs/git-ls-remote.html
- 6: https://stackoverflow.com/questions/52662337/what-is-the-definition-of-pattern-for-git-ls-remote
Distinguish git ls-remote communication failures from a missing nightly tag.
At line 701, REFS=$(...) || true only turns real command failures into “no tag found,” which can lead to pushing a new tag instead of treating a query/auth/network failure as the real result.
Suggested fix
- REFS=$(git ls-remote --tags origin "refs/tags/$TAG" "refs/tags/$TAG^{}" || true)
+ if ! REFS=$(git ls-remote --exit-code --tags origin "refs/tags/$TAG"); then
+ echo "::error::No remote tag $TAG found; aborting rather than creating one."
+ exit 1
+ fi🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/nightly.yml around lines 701 - 708, Update the REFS lookup
in the nightly tag flow to preserve and check the exit status of git ls-remote
separately from its output. Only create and push the tag when the query succeeds
with no matching refs; when the command fails due to communication,
authentication, or another error, stop the workflow and report the failure
instead of treating it as a missing tag.
| RELEASE_ID=$(gh api "repos/${GITHUB_REPOSITORY}/releases/tags/${TAG}" -q '.id' 2>/dev/null || true) | ||
| if [ -n "$RELEASE_ID" ]; then | ||
| IMMUTABLE=$(gh api "repos/${GITHUB_REPOSITORY}/releases/tags/${TAG}" -q '.immutable' 2>/dev/null || true) | ||
| if [ "$IMMUTABLE" = "true" ]; then | ||
| echo "::warning::Release $TAG is already published and immutable, so it cannot be replaced. Skipping the GitHub release step; this run's artifacts are on DigitalOcean Spaces, which the release notes already name as the canonical download location." | ||
| exit 0 | ||
| fi | ||
| echo "Replacing the existing (mutable) release $TAG." | ||
| gh release delete "$TAG" --yes 2>/dev/null || true | ||
| fi |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "## files"
git ls-files | rg '(^|/)(nightly\.yml|.*workflow.*\.ya?ml)$' || true
echo
echo "## nightly workflow relevant section"
if [ -f .github/workflows/nightly.yml ]; then
wc -l .github/workflows/nightly.yml
sed -n '740,795p' .github/workflows/nightly.yml | cat -n
fi
echo
echo "## search gh release create/delete/api around workflows"
rg -n 'gh (api|release delete|release create)|REMOVE_GITHUB_RELEASE|DELETE.*RELEASE|TAG=' .github/workflows/nightly.yml || trueRepository: WebFirstLanguage/wfl
Length of output: 4326
Don’t swallow release-lookup failures before creating the release.
2>/dev/null || true on the gh api calls masks 401/403/ratelimit/transient errors the same way as “release does not exist,” leaving RELEASE_ID empty. If an existing release is actually present but lookup fails, the immutable/mutable checks are skipped and the later gh release create can fail with “release already exists.” Fail fast for lookup/delete failures that are not “not found,” and preserve the response only so the immutable field can be checked.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/nightly.yml around lines 766 - 775, The release lookup
logic around RELEASE_ID must distinguish a missing release from authentication,
rate-limit, transient, or other API failures. Remove the unconditional
suppression from the gh api calls, preserve the lookup response for checking .id
and .immutable, and fail the workflow for any non-404 lookup error; retain the
existing immutable skip and mutable-release deletion behavior, while allowing
only a confirmed “not found” result to proceed to release creation.
What was broken
Nightly Build run 30528566163 (manual dispatch, 2026-07-30 08:55 UTC, main
579eb806) failed. All three build jobs were green —Check for changes,Build WFL for Linux (static musl), andBuild WFL for Windows(25m35s, through MSI packaging and the installer smoke test). TheCreate or Update Nightly Releasejob died atTag commit for nightly:Publish or update nightly releasewas skipped, so a fully successful build published nothing.Root cause
This repository has GitHub immutable releases enabled —
GET /releases/tags/nightly-2026-07-30reports"immutable": true. Once a release is published, its tag is permanently protected against deletion and movement. The release job was built on a delete-then-recreate model that this makes impossible:Tag commit for nightlydeletes the remote tag, then recreates and pushes it. The delete is refused (GH013); the refusal is downgraded to a warning and execution continues; the push then fails hard with "already exists".Publish or update nightly releasedoes the same thing one level up —gh release delete "$TAG" --yes 2>/dev/null || truefollowed bygh release create "$TAG". An immutable release cannot be deleted either, so the delete is silently discarded and the create would fail because the release already exists. Despite the step's name it never updated anything; it only ever recreated. This is a latent second failure that today's run never reached because step 1 died first.So any nightly run that happens after a release already exists for the same calendar date fails. In practice that means every manual re-dispatch on a day the scheduled nightly already published. It has not bitten before now only by luck: the 2026-07-29 re-dispatch worked because that day's scheduled nightly failed before tagging (the musl portability gate, fixed in #657), so no tag existed. Today's scheduled run succeeded and published at 06:18, and the 08:55 dispatch hit the wall.
The fix
.github/workflows/nightly.ymlonly, two steps in thereleasejob. No Rust code, no other job touched.Tag commit for nightly— never deletes; is idempotent instead:git push --force, and if that is refused, warn and continue rather than fail.Annotated tags are resolved through the
^{}line ofgit ls-remoteso the comparison is against the tagged commit, with a fallback for a lightweight tag.Publish or update nightly release— checks before it replaces:immutable, warn and exit 0 rather than attempt an impossible delete-and-recreate;Asset collection and validation moved above that check so the "empty file" and "no artifacts" errors still fire. Release notes, flags, and asset handling are byte-identical.
The deliberate judgement call: on a same-day re-run against an already-published immutable release the job now succeeds with a warning instead of failing. The build genuinely is good, and
Publish artifacts to DigitalOcean Spaces— which runs earlier and is unaffected — has already refreshed the canonical CDN copies that the release notes point at. Only the case "release exists and is immutable" is tolerated; anything else still fails loudly.Verification
This is a pure CI-mechanics change with no Rust behaviour to test, so per
AGENTS.mdthe existing checks are the test. What was verified locally:mainconfirms step counts are unchanged in all four jobs (5 / 21 / 10 / 9), thecheck-for-changes,build, andbuild-linuxjobs are byte-identical, and triggers are unchanged. Only the two named steps differ.bash -nclean on both step scripts.git/ghcovering every branch:gh release create; mutable → deletes and recreates; no artifacts → still::error::and exit 1).git ls-remoteparsing was checked against the real repository, not just stubs:refs/tags/nightly-2026-07-30resolves to tag objectab3166a6and commit1bb02f2a, which is the commit the 05:57 scheduled nightly built.Not verified: an actual Actions run. That is what this PR's CI is for. The most useful post-merge check is to dispatch Nightly Build from
mainon a day the scheduled run has already published — the previously failing path — and confirm the release job reports the skip warning and goes green.Automated triage PR from the WFL repo warden — please review and merge; the warden does not merge its own PRs.
Summary by CodeRabbit