docs: Guide release recovery through roll-forward #1821
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,163 +1,102 @@ | ||
| # Release recovery runbook | ||
|
|
||
| ## Purpose | ||
|
|
||
| Use this runbook when a native CLI release has a tag commit that differs from | ||
| the `sourceRepositoryDigest` in its asset attestations. Such a release cannot | ||
| be downloaded by the dispatcher. Recover by publishing the next valid release; | ||
| do not try to repair the broken release in place. | ||
|
|
||
| ## First response: roll forward | ||
|
|
||
| 1. Confirm the affected release is broken by comparing its tag commit with an | ||
| attached asset attestation. | ||
|
|
||
| ```sh | ||
| gh release download <release-tag> --repo <owner>/<repo> \ | ||
| --pattern <asset-name> --dir <download-directory> | ||
| gh api repos/<owner>/<repo>/commits/<release-tag> --jq .sha | ||
| gh attestation verify <download-directory>/<asset-name> --repo <owner>/<repo> --format json \ | ||
| | jq -r '.[].verificationResult.signature.certificate.sourceRepositoryDigest' | ||
| ``` | ||
|
|
||
| 2. Merge the pending release-please release PR for the affected branch. This | ||
| creates the next version and starts a normal head-based publish run. | ||
|
|
||
| 3. Verify the new runner release and the package release that pins it. The | ||
| runner tag commit, every asset's attestation digest, and the publish run's | ||
| `headSha` must match. | ||
|
|
||
| ```sh | ||
| gh run view <publish-run-id> --repo <owner>/<repo> --json headSha --jq .headSha | ||
| gh api repos/<owner>/<repo>/commits/<new-runner-tag> --jq .sha | ||
| gh attestation verify <new-asset> --repo <owner>/<repo> --format json \ | ||
| | jq -r '.[].verificationResult.signature.certificate.sourceRepositoryDigest' | ||
| ``` | ||
|
Comment on lines
+26
to
35
Contributor
There was a problem hiding this comment. 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win Verify the package pin and every published asset. This step claims to verify the package release and “every asset,” but the commands inspect only one 🤖 Prompt for AI Agents |
||
|
|
||
| The 2026-07-17 recovery followed this procedure: runner beta.48 was left | ||
| broken, the next release-please PR produced runner beta.49, and the following | ||
| package release pinned beta.49. The normal publish run, tag, and attestations | ||
| all resolved to the same commit. | ||
|
|
||
| ## Why a broken release cannot be revived | ||
|
|
||
| Do not create a new tag, draft, or publish run for the historical version. | ||
| Revival is structurally impossible for three independent reasons: | ||
|
|
||
| 1. Rerunning a workflow replays the workflow definition from the original | ||
| commit. If that workflow definition was itself broken, rerunning it repeats | ||
| the failure. | ||
| 2. A new workflow run attests its own `github.sha`, which is the current branch | ||
| head. It cannot create attestations for an earlier release commit. | ||
|
Comment on lines
+47
to
+51
Contributor
There was a problem hiding this comment. 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
# Show the relevant section of the runbook with line numbers.
sed -n '35,70p' docs/release-recovery-runbook.md | cat -n
# Search for any other references to github.sha / GITHUB_SHA in the repo.
rg -n "github\.sha|GITHUB_SHA|GITHUB_REF|rerun" docs .github . -g '!**/node_modules/**' || trueRepository: hatayama/unity-cli-loop Length of output: 10525 🌐 Web query:
💡 Result: When you rerun a GitHub Actions workflow or a specific job within a workflow, the system uses the exact same GITHUB_SHA and GITHUB_REF that were associated with the initial trigger of that workflow run [1][2]. These values are immutable for the duration of the workflow run, regardless of any subsequent changes to the branch or tag in the repository [1][3]. Key semantics to keep in mind regarding these environment variables: 1. Rerun Behavior: Because reruns utilize the original context, they do not "re-fetch" or re-evaluate the triggering commit or reference [1][4]. This ensures consistency, as the rerun operates on the exact codebase version that was initially intended to be tested [1][2]. 2. GITHUB_SHA Semantics: The value of GITHUB_SHA depends on the event that triggered the workflow [5][6]. - For push events, it is the SHA of the commit that triggered the push. - For pull_request events, it often refers to a temporary merge commit created by GitHub (e.g., refs/remotes/pull/:pr_number/merge), rather than the head commit of the source branch [6]. If you require the specific head SHA of a pull request, use github.event.pull_request.head.sha instead [6]. 3. Tag-Triggered Workflows: When a workflow is triggered by a tag (e.g., via a push event that includes a tag), GITHUB_REF will contain the reference to that tag (e.g., refs/tags/my-tag). During a rerun, this reference remains fixed to the specific tag that originally triggered the workflow [1][7]. In summary, rerunning a workflow is a "replay" of the original trigger context, ensuring that environment variables like GITHUB_SHA and GITHUB_REF remain consistent with the original run [1][4]. Citations:
Use the triggering commit/ref, not “current branch head”. Fresh workflow runs attest the commit that triggered the run, and reruns preserve the original 🧰 Tools🪛 LanguageTool[uncategorized] ~50-~50: The official name of this software platform is spelled with a capital “H”. (GITHUB) 🤖 Prompt for AI Agents |
||
| 3. The release resolver binds the historical version to its original release | ||
| commit. A later publish run therefore produces a tag-to-attestation digest | ||
| mismatch even when an owner pre-creates the tag or draft release. | ||
|
|
||
| For example, beta.48's tag pointed at `2d8d1b94` while its published asset | ||
| attestations carried `2c73c6ac`. Reusing that version would preserve the same | ||
| invariant violation. | ||
|
|
||
| ## When rerun recovery is valid | ||
|
|
||
| Rerun only when all of the following are true: | ||
|
|
||
| - The original failed run's `headSha` is the intended release commit. | ||
| - The workflow definition at that commit is known to be healthy. | ||
| - The release tag and draft release, if they exist, point at that same commit. | ||
|
|
||
| Check the original run before rerunning it: | ||
|
|
||
| ```sh | ||
| gh run view <run-id> --repo <owner>/<repo> --json headSha,workflowName,url | ||
| gh api repos/<owner>/<repo>/commits/<release-tag> --jq .sha | ||
| gh run view <run-id> --repo <owner>/<repo> --log-failed | ||
| ``` | ||
|
|
||
| If the run used an older workflow revision that lacks a required fix, or its | ||
| head SHA differs from the intended release commit, do not rerun it. Use the | ||
| roll-forward procedure instead. | ||
|
|
||
| When all conditions hold, rerun the failed jobs from the original run and | ||
| approve the `cli-release` environment if GitHub requests approval: | ||
|
|
||
| ```sh | ||
| gh run rerun <run-id> --repo <owner>/<repo> --failed | ||
| ``` | ||
|
Comment on lines
+60
to
+85
Contributor
There was a problem hiding this comment. 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win 🧩 Analysis chain🌐 Web query:
💡 Result: According to official GitHub Actions documentation, you can re-run a workflow run, all failed jobs, or specific individual jobs up to 30 days after the initial run [1][2]. It is important to note that the ability to re-run a job is dependent on the existence of the workflow's logs [3]. While the re-run window is capped at 30 days, the retention period for the logs themselves—which determines whether the run is still accessible for a re-run—can vary [4]. By default, logs and artifacts are retained for 90 days, but this can be configured by organizations to range from 1 to 400 days for private repositories, or 1 to 90 days for public repositories [4]. Even if log retention is extended beyond 30 days, the ability to initiate a re-run remains restricted to the 30-day window following the initial execution [1][2]. Additionally, there is a limit on the number of times a workflow can be re-run: a single workflow run can be re-run a maximum of 50 times, which includes both full re-runs and re-runs of specific subsets of jobs [1][5][6]. Citations:
Include the 30-day rerun limit. In 🧰 Tools🪛 LanguageTool[style] ~62-~62: Consider removing “of” to be more concise (ALL_OF_THE) 🤖 Prompt for AI Agents |
||
|
|
||
| ## Operational notes | ||
|
|
||
| - Do not dispatch `recovery-target` to publish a historical release. It is | ||
| limited to validation because a later run cannot produce valid attestations | ||
| for an earlier commit. | ||
| - Cancel an older run waiting for `cli-release` approval before retrying. The | ||
| workflow concurrency group does not cancel it automatically, so it can block | ||
| the newer run indefinitely. | ||
| - Delete a broken release only after the version bump is merged. Deleting it | ||
| first can make the resolver target the historical commit again and cause | ||
| later push builds to fail. | ||
| - Do not grant GitHub Actions tag-ruleset bypass permissions. That does not | ||
| resolve the attestation invariant and expands bot authority unnecessarily. | ||
| - A broken release may remain published while the roll-forward release is cut; | ||
| the dispatcher rejects it, so it is harmless once consumers use the newer | ||
| package pin. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Describe the dispatcher failure accurately.
The asset remains downloadable; the dispatcher rejects it during attestation verification. Replace “cannot be downloaded” with “cannot be verified or used by the dispatcher” to avoid sending operators toward the wrong diagnosis.
🤖 Prompt for AI Agents