Skip to content

fix(ci): stop the nightly release job failing on an immutable tag #661

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
logbie merged 1 commit into from
Jul 30, 2026
Merged

fix(ci): stop the nightly release job failing on an immutable tag #661

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 50 additions & 22 deletions .github/workflows/nightly.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -687,40 +687,52 @@ jobs:
run: |
DATE=$(date +'%Y-%m-%d')
TAG="nightly-${DATE}"
SHA=$(git rev-parse HEAD)

# This repository has GitHub immutable releases enabled, so the tag of a
# published release is permanently protected: pushing a delete is refused
# with "GH013 ... Cannot delete this tag". The previous
# delete-then-recreate flow swallowed that refusal as a warning and then
# died on "tag already exists", failing the whole release job on any run
# that happened after a release already existed for the same date -- a
# manual re-dispatch on a day the scheduled nightly had already
# published, for instance -- even with every build job green.
# Tagging is idempotent now and never fails on an already-published tag.
REFS=$(git ls-remote --tags origin "refs/tags/$TAG" "refs/tags/$TAG^{}" || true)

if [ -z "$REFS" ]; then
echo "No existing remote tag $TAG; creating it at $SHA."
git tag -f -a "$TAG" -m "Nightly build $DATE"
git push origin "refs/tags/$TAG"
exit 0
fi
Comment on lines +701 to +708

@coderabbitai coderabbitai Bot Jul 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== workflow context =="
if [ -f .github/workflows/nightly.yml ]; then
  nl -ba .github/workflows/nightly.yml | sed -n '670,735p'
else
  echo "myfile not found"
fi

echo
echo "== git command semantics probe =="
python3 - <<'PY'
import subprocess, textwrap, tempfile, pathlib, os

def run(env=None, check=None, user_env=None):
    cmd = ['git'] + list(env if env else [])
    merged = os.environ.copy()
    if user_env:
        merged.update(user_env)
    merged.pop('GIT_SSH_COMMAND', None)
    merged.pop('GIT_ASKPASS', None)
    res = subprocess.run(cmd, capture_output=True, text=True, check=False, env=merged)
    return res.returncode, '\n'.join(res.stdout.splitlines()[:3]), res.stderr.splitlines()[:3]

print("local repo ls-remote exit:", "success" if run(['ls-remote', '--tags'], user_env={'GIT_SSH_COMMAND':'true'})[0]==0 else "fail")

print("repoless exact pattern exits non-zero?", end=" ")
rc = subprocess.run(['git','ls-remote','--tags','origin','refs/tags/nightly-2026-07-20'], text=True).returncode
print("non-zero" if rc else "zero")

print("repoless two patterns exits non-zero?", end=" ")
rc = subprocess.run(['git','ls-remote','--tags','origin','refs/tags/nightly-2026-07-20','refs/tags/nightly-2026-07-20^{}'], text=True).returncode
print("non-zero" if rc else "zero")

PY

Repository: WebFirstLanguage/wfl

Length of output: 221

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== locate workflow =="
git ls-files .github/workflows/nightly.yml || true

if [ -f .github/workflows/nightly.yml ]; then
  echo "== workflow lines 660-740 =="
  awk 'NR>=660 && NR<=740 {printf "%6d\t%s\n", NR, $0}' .github/workflows/nightly.yml
fi

echo
echo "== git command semantics probe =="
python3 - <<'PY'
import subprocess, os

print("git_lsremote_no_repo_exact_exit_nonzero:", subprocess.run(["git","ls-remote","--tags","origin","refs/tags/DNE-EXACT"], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).returncode != 0)
print("git_lsremote_no_repo_two_patterns_exit_nonzero:", subprocess.run(["git","ls-remote","--tags","origin","refs/tags/DNE-EXACT","refs/tags/DNE-EXACT^{}"], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).returncode != 0)
PY

Repository: WebFirstLanguage/wfl

Length of output: 4790

🌐 Web query:

Git documentation git ls-remote exit code remote errors pattern no match

💡 Result:

By default, the git ls-remote command returns an exit code of 0 if it successfully communicates with the remote repository, regardless of whether any references (refs) match your provided patterns [1][2]. To distinguish between successful communication and the absence of matching references, you must use the --exit-code flag [1][3]. When --exit-code is provided, the command will exit with status 2 if no matching references are found [1][4][2]. Regarding pattern matching, git ls-remote interprets patterns as glob patterns [2][5]. These patterns are matched against the tail of a reference name, meaning they either match the full name (e.g., refs/heads/foo) or match a component starting after a slash separator (e.g., bar matches refs/heads/bar but not refs/heads/foobar) [2][6]. If you provide multiple patterns, the command will display any reference that matches at least one of the specified patterns [2][6].

Citations:

Distinguish git ls-remote communication failures from a missing nightly tag.

At line 701, REFS=$(...) || true only turns real command failures into “no tag found,” which can lead to pushing a new tag instead of treating a query/auth/network failure as the real result.

Suggested fix 
-          REFS=$(git ls-remote --tags origin "refs/tags/$TAG" "refs/tags/$TAG^{}" || true)
+          if ! REFS=$(git ls-remote --exit-code --tags origin "refs/tags/$TAG"); then
+            echo "::error::No remote tag $TAG found; aborting rather than creating one."
+            exit 1
+          fi
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/nightly.yml around lines 701 - 708, Update the REFS lookup
in the nightly tag flow to preserve and check the exit status of git ls-remote
separately from its output. Only create and push the tag when the query succeeds
with no matching refs; when the command fails due to communication,
authentication, or another error, stop the workflow and report the failure
instead of treating it as a missing tag.


# Delete existing local tag if it exists
echo "Checking for existing local tag: $TAG"
if git rev-parse "$TAG" >/dev/null 2>&1; then
echo "Deleting existing local tag: $TAG"
git tag -d "$TAG"
else
echo "No existing local tag found"
# An annotated tag reports the tag object on the plain line and the
# commit it points at on the "^{}" line; prefer the latter.
TAGGED=$(printf '%s\n' "$REFS" | awk '$2 ~ /\^\{\}$/ { print $1 }' | head -n 1)
if [ -z "$TAGGED" ]; then
TAGGED=$(printf '%s\n' "$REFS" | awk 'NR == 1 { print $1 }')
fi

# Delete existing remote tag if it exists
echo "Checking for existing remote tag: $TAG"
if git ls-remote --tags origin | grep -q "refs/tags/$TAG$"; then
echo "Deleting existing remote tag: $TAG"
if ! git push origin ":refs/tags/$TAG" 2>&1; then
echo "::warning::Failed to delete remote tag $TAG, but continuing (may not exist or insufficient permissions)"
fi
else
echo "No existing remote tag found"
if [ "$TAGGED" = "$SHA" ]; then
echo "Tag $TAG already points at $SHA; nothing to do."
exit 0
fi

# Create new tag and push
echo "Creating new tag: $TAG"
git tag -a "$TAG" -m "Nightly build $DATE"
echo "Pushing new tag: $TAG"
git push origin "$TAG"
echo "::warning::Tag $TAG already exists at $TAGGED but this build is $SHA; attempting to move it."
git tag -f -a "$TAG" -m "Nightly build $DATE"
if git push --force origin "refs/tags/$TAG"; then
echo "Moved $TAG to $SHA."
else
echo "::warning::Could not move $TAG; it is protected by an immutable release. Leaving the published tag at $TAGGED."
fi
Comment on lines +724 to +728

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 30, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Fail when the tag push is rejected for other reasons

In the checked Tag commit for nightly step, this branch treats every nonzero result from git push --force as immutable-release protection, even though git push -h only defines --force as “force updates” and the command can also fail because of authentication, connectivity, hooks, or another tag rule. If such a failure occurs and the subsequent GitHub API calls succeed, the next step can create or replace a mutable release using the old remote tag while uploading artifacts and notes for ${{ github.sha }}, producing contradictory release provenance while the job reports success. Verify that the existing release is immutable before suppressing the push error; otherwise propagate the failure.

Useful? React with 👍 / 👎.


- name: Publish or update nightly release
env:
VERSION: ${{ needs.check-for-changes.outputs.version }}
SHORT_SHA: ${{ needs.check-for-changes.outputs.short_sha }}
run: |
TAG="nightly-$(date +'%Y-%m-%d')"
gh release delete "$TAG" --yes 2>/dev/null || true

# Collect all artifact files to upload with the release
# GitHub's immutable releases require assets to be attached during creation
Expand All @@ -746,6 +758,22 @@ jobs:
echo "Files to upload: $ASSET_FILES"
echo "Number of files: $(echo $ASSET_FILES | wc -w)"

# An immutable release cannot be deleted or edited once published, so the
# old unconditional "delete, then create" could not refresh one: the
# delete was silently discarded by "|| true" and the create then failed
# because the release already existed. Only replace a release that is
# still mutable.
RELEASE_ID=$(gh api "repos/${GITHUB_REPOSITORY}/releases/tags/${TAG}" -q '.id' 2>/dev/null || true)
if [ -n "$RELEASE_ID" ]; then
IMMUTABLE=$(gh api "repos/${GITHUB_REPOSITORY}/releases/tags/${TAG}" -q '.immutable' 2>/dev/null || true)
if [ "$IMMUTABLE" = "true" ]; then
echo "::warning::Release $TAG is already published and immutable, so it cannot be replaced. Skipping the GitHub release step; this run's artifacts are on DigitalOcean Spaces, which the release notes already name as the canonical download location."
exit 0
fi
echo "Replacing the existing (mutable) release $TAG."
gh release delete "$TAG" --yes 2>/dev/null || true
fi
Comment on lines +766 to +775

@devin-ai-integration devin-ai-integration Bot Jul 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Immutability detection depends on the .immutable field being present and equal to "true"

The skip decision hinges entirely on gh api ... -q '.immutable' returning the literal string true. If the field is absent or the second gh api call fails transiently, IMMUTABLE is empty/null and the script falls through to gh release delete (silenced by || true) followed by gh release create, which is exactly the original failing sequence (::error:: already exists). A single API call reused for both .id and .immutable (e.g. gh api ... -q '.id + " " + (.immutable|tostring)') would remove the second round-trip and the possibility of the two reads disagreeing. Note also that on a 404 gh may print null for -q '.id', making RELEASE_ID non-empty; that path is harmless because IMMUTABLE is then also null and the delete-then-create behaves as before.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +766 to +775

@coderabbitai coderabbitai Bot Jul 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "## files"
git ls-files | rg '(^|/)(nightly\.yml|.*workflow.*\.ya?ml)$' || true

echo
echo "## nightly workflow relevant section"
if [ -f .github/workflows/nightly.yml ]; then
  wc -l .github/workflows/nightly.yml
  sed -n '740,795p' .github/workflows/nightly.yml | cat -n
fi

echo
echo "## search gh release create/delete/api around workflows"
rg -n 'gh (api|release delete|release create)|REMOVE_GITHUB_RELEASE|DELETE.*RELEASE|TAG=' .github/workflows/nightly.yml || true

Repository: WebFirstLanguage/wfl

Length of output: 4326

Don’t swallow release-lookup failures before creating the release.

2>/dev/null || true on the gh api calls masks 401/403/ratelimit/transient errors the same way as “release does not exist,” leaving RELEASE_ID empty. If an existing release is actually present but lookup fails, the immutable/mutable checks are skipped and the later gh release create can fail with “release already exists.” Fail fast for lookup/delete failures that are not “not found,” and preserve the response only so the immutable field can be checked.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/nightly.yml around lines 766 - 775, The release lookup
logic around RELEASE_ID must distinguish a missing release from authentication,
rate-limit, transient, or other API failures. Remove the unconditional
suppression from the gh api calls, preserve the lookup response for checking .id
and .immutable, and fail the workflow for any non-404 lookup error; retain the
existing immutable skip and mutable-release deletion behavior, while allowing
only a confirmed “not found” result to proceed to release creation.


# Create release with assets attached in a single command
# This works with immutable releases since assets are part of the creation
gh release create "$TAG" \
Expand Down
Loading