diff --git a/docs/release-recovery-runbook.md b/docs/release-recovery-runbook.md index 6076a7990..fff224d81 100644 --- a/docs/release-recovery-runbook.md +++ b/docs/release-recovery-runbook.md @@ -1,163 +1,102 @@ # Release recovery runbook -## Root cause +## Purpose -In `recovery-target` mode, the release target is an earlier commit than the -workflow run's `GITHUB_SHA`. GitHub rejects the Actions `GITHUB_TOKEN` when it -tries to create a tag or release for that different commit, even when the token -has `contents: write`. This is neither a missing permission nor a tag ruleset -failure. +Use this runbook when a native CLI release has a tag commit that differs from +the `sourceRepositoryDigest` in its asset attestations. Such a release cannot +be downloaded by the dispatcher. Recover by publishing the next valid release; +do not try to repair the broken release in place. -The behavior was confirmed with the same token and `contents: write` -permission in reproduction runs [29459199798](https://github.com/hatayama/unity-cli-loop/actions/runs/29459199798) -and [29459266044](https://github.com/hatayama/unity-cli-loop/actions/runs/29459266044): +## First response: roll forward -- A tag pointing at the run head returned HTTP 201. -- A tag pointing at `2d8d1b9443843f45b48479134f846b6f540e928b`, an ancestor of - the run head, returned HTTP 403 `Resource not accessible by integration`. -- Creating a draft release directly at that past commit also returned HTTP 403. - -The precise condition is a ref target different from the run's `GITHUB_SHA`, -not merely an unreachable commit. Normal head-based releases are unaffected. - -## Publishing invariant: assets may only be published by the original run - -Attestation certificates bind `SourceRepositoryDigest` to the workflow run's -head commit (`GITHUB_SHA`), never to the commit the build checked out. The -dispatcher verifies that the release tag's commit equals the certificate -digest, so a release is only downloadable when the publishing run's head IS -the release commit. - -Therefore the only valid way to publish recovery assets is to **rerun the -original failed run whose head is the release commit** (step 5 below). Never -publish through a new `recovery-target` workflow-dispatch run on a later -branch head: its certificates would carry the later head's digest while the -tag points at the historical release commit, producing a release that -permanently fails attestation verification. The workflow now rejects this -combination in the publish job's "Verify release metadata" step; recovery -dispatch runs remain usable for build validation only. - -This is not theoretical: `uloop-project-runner-v3.0.0-beta.48` was published -on 2026-07-15 through a recovery dispatch run after the owner pre-created the -tag and draft release. The tag points at the historical release commit -(`2d8d1b94`) while the attestation certificates carry the dispatch run's head -digest (`2c73c6ac`), so every cold-cache download of beta.48 fails -verification on all platforms. - -## Recovery procedure - -Run these commands with an owner-authenticated `gh` session. Replace every -`` before running a command. - -1. Confirm the authenticated account and repository. - - ```sh - gh auth status - gh api user --jq .login - gh api repos// --jq .full_name - ``` - -2. Determine the release commit and release tag from the release PR and the - failed workflow run. +1. Confirm the affected release is broken by comparing its tag commit with an + attached asset attestation. ```sh - gh pr view --repo / --json mergeCommit --jq .mergeCommit.oid - gh run view --repo / --json headSha + gh release download --repo / \ + --pattern --dir + gh api repos///commits/ --jq .sha + gh attestation verify / --repo / --format json \ + | jq -r '.[].verificationResult.signature.certificate.sourceRepositoryDigest' ``` - The release commit must be the commit validated by the workflow. Do not use - the current branch head when recovering an older release. - -3. Create the release tag at the validated release commit. - - ```sh - gh api repos///git/refs \ - -f ref=refs/tags/ \ - -f sha= - ``` +2. Merge the pending release-please release PR for the affected branch. This + creates the next version and starts a normal head-based publish run. -4. Download the native release input artifact and create the draft release. - Creating only the tag is not sufficient: the Actions token also receives - HTTP 403 when it tries to create the draft release. +3. Verify the new runner release and the package release that pins it. The + runner tag commit, every asset's attestation digest, and the publish run's + `headSha` must match. ```sh - gh run download --repo / \ - -n native-cli-release-input -D - gh release create --repo / \ - --draft --title \ - --notes-file /release-input/release-notes.md \ - --target + gh run view --repo / --json headSha --jq .headSha + gh api repos///commits/ --jq .sha + gh attestation verify --repo / --format json \ + | jq -r '.[].verificationResult.signature.certificate.sourceRepositoryDigest' ``` - Add `--prerelease` when the release tag is a prerelease tag. - -5. Rerun the failed publish job of the original run whose head is the release - commit, and approve the release environment if prompted. Do not dispatch a - new `recovery-target` run to publish instead: a dispatch run on a later - head cannot produce valid attestations for the release commit, and the - workflow rejects it (see "Publishing invariant" above). - - ```sh - gh run rerun --repo / --failed - ``` - -6. If an older `dispatcher-publish` run is waiting for the `cli-release` - environment, cancel it before retrying the recovery. The workflow uses a - concurrency group without automatic cancellation, so an old approval-waiting - run can block the newer run indefinitely. - - ```sh - gh run list --repo / --workflow dispatcher-publish.yml \ - --branch --limit 20 - gh run cancel --repo / - ``` - -7. Recover the Unity package release only after the runner release has - completed and its assets are available. The package sync script creates a - release at the historical release commit and can hit the same 403; create - that release as the owner before rerunning the sync. - - ```sh - gh release create v --repo / \ - --title v \ - --notes-file \ - --target - ``` - - Add `--prerelease` when the package release is a prerelease. - - Run the package sync or the relevant post-publish job after the runner - release assets are complete. The sync reuses an existing correctly targeted - release. - -8. Complete the workflow-dispatch recovery manually. Push-only post-publish - steps are intentionally skipped for a `workflow_dispatch` run. - - ```sh - gh pr edit --repo / \ - --remove-label 'autorelease: pending' \ - --add-label 'autorelease: tagged' - gh workflow run release-please.yml --repo / --ref \ - -f branch= - ``` - -## Scope of the workflows - -`native-cli-publish.yml` is the workflow with `recovery-target` and now emits -this runbook path when tag or draft-release creation returns the observed 403. -The creation and reuse logic is unchanged. Its publish job refuses to publish -assets from a recovery dispatch run whose head differs from the release -commit, because such a run cannot produce matching attestations. - -`dispatcher-publish.yml` has no recovery-target input and requires its release -target to equal `GITHUB_SHA`, so it has no historical-commit creation path. -Its concurrency group still matters during recovery because an older run can -block a retry. - -`scripts/sync-release-please-package-releases.sh` can create package releases -at release-please commits from the repository history. This runbook's owner -pre-creation step covers that path; the runner release must be recovered first. - -Do not add GitHub Actions to the tag-ruleset bypass actors. That would grant -the bot tag deletion and force-update capabilities without addressing the -historical-commit restriction. +The 2026-07-17 recovery followed this procedure: runner beta.48 was left +broken, the next release-please PR produced runner beta.49, and the following +package release pinned beta.49. The normal publish run, tag, and attestations +all resolved to the same commit. + +## Why a broken release cannot be revived + +Do not create a new tag, draft, or publish run for the historical version. +Revival is structurally impossible for three independent reasons: + +1. Rerunning a workflow replays the workflow definition from the original + commit. If that workflow definition was itself broken, rerunning it repeats + the failure. +2. A new workflow run attests its own `github.sha`, which is the current branch + head. It cannot create attestations for an earlier release commit. +3. The release resolver binds the historical version to its original release + commit. A later publish run therefore produces a tag-to-attestation digest + mismatch even when an owner pre-creates the tag or draft release. + +For example, beta.48's tag pointed at `2d8d1b94` while its published asset +attestations carried `2c73c6ac`. Reusing that version would preserve the same +invariant violation. + +## When rerun recovery is valid + +Rerun only when all of the following are true: + +- The original failed run's `headSha` is the intended release commit. +- The workflow definition at that commit is known to be healthy. +- The release tag and draft release, if they exist, point at that same commit. + +Check the original run before rerunning it: + +```sh +gh run view --repo / --json headSha,workflowName,url +gh api repos///commits/ --jq .sha +gh run view --repo / --log-failed +``` + +If the run used an older workflow revision that lacks a required fix, or its +head SHA differs from the intended release commit, do not rerun it. Use the +roll-forward procedure instead. + +When all conditions hold, rerun the failed jobs from the original run and +approve the `cli-release` environment if GitHub requests approval: + +```sh +gh run rerun --repo / --failed +``` + +## Operational notes + +- Do not dispatch `recovery-target` to publish a historical release. It is + limited to validation because a later run cannot produce valid attestations + for an earlier commit. +- Cancel an older run waiting for `cli-release` approval before retrying. The + workflow concurrency group does not cancel it automatically, so it can block + the newer run indefinitely. +- Delete a broken release only after the version bump is merged. Deleting it + first can make the resolver target the historical commit again and cause + later push builds to fail. +- Do not grant GitHub Actions tag-ruleset bypass permissions. That does not + resolve the attestation invariant and expands bot authority unnecessarily. +- A broken release may remain published while the roll-forward release is cut; + the dispatcher rejects it, so it is harmless once consumers use the newer + package pin.