From fd43cd0febe2a7e8da216788b66865b403369301 Mon Sep 17 00:00:00 2001 From: hatayama Date: Fri, 17 Jul 2026 23:43:04 +0900 Subject: [PATCH 1/2] Document roll-forward release recovery Direct recovery toward a new valid release because an older broken release cannot gain attestations for its historical commit from a later workflow run. --- docs/release-recovery-runbook.md | 197 ++++++++++--------------------- 1 file changed, 60 insertions(+), 137 deletions(-) diff --git a/docs/release-recovery-runbook.md b/docs/release-recovery-runbook.md index 6076a7990..6396485a5 100644 --- a/docs/release-recovery-runbook.md +++ b/docs/release-recovery-runbook.md @@ -1,163 +1,86 @@ # Release recovery runbook -## Root cause +## Purpose -In `recovery-target` mode, the release target is an earlier commit than the -workflow run's `GITHUB_SHA`. GitHub rejects the Actions `GITHUB_TOKEN` when it -tries to create a tag or release for that different commit, even when the token -has `contents: write`. This is neither a missing permission nor a tag ruleset -failure. +Use this runbook when a native CLI release has a tag commit that differs from +the `sourceRepositoryDigest` in its asset attestations. Such a release cannot +be downloaded by the dispatcher. Recover by publishing the next valid release; +do not try to repair the broken release in place. -The behavior was confirmed with the same token and `contents: write` -permission in reproduction runs [29459199798](https://github.com/hatayama/unity-cli-loop/actions/runs/29459199798) -and [29459266044](https://github.com/hatayama/unity-cli-loop/actions/runs/29459266044): +## First response: roll forward -- A tag pointing at the run head returned HTTP 201. -- A tag pointing at `2d8d1b9443843f45b48479134f846b6f540e928b`, an ancestor of - the run head, returned HTTP 403 `Resource not accessible by integration`. -- Creating a draft release directly at that past commit also returned HTTP 403. - -The precise condition is a ref target different from the run's `GITHUB_SHA`, -not merely an unreachable commit. Normal head-based releases are unaffected. - -## Publishing invariant: assets may only be published by the original run - -Attestation certificates bind `SourceRepositoryDigest` to the workflow run's -head commit (`GITHUB_SHA`), never to the commit the build checked out. The -dispatcher verifies that the release tag's commit equals the certificate -digest, so a release is only downloadable when the publishing run's head IS -the release commit. - -Therefore the only valid way to publish recovery assets is to **rerun the -original failed run whose head is the release commit** (step 5 below). Never -publish through a new `recovery-target` workflow-dispatch run on a later -branch head: its certificates would carry the later head's digest while the -tag points at the historical release commit, producing a release that -permanently fails attestation verification. The workflow now rejects this -combination in the publish job's "Verify release metadata" step; recovery -dispatch runs remain usable for build validation only. - -This is not theoretical: `uloop-project-runner-v3.0.0-beta.48` was published -on 2026-07-15 through a recovery dispatch run after the owner pre-created the -tag and draft release. The tag points at the historical release commit -(`2d8d1b94`) while the attestation certificates carry the dispatch run's head -digest (`2c73c6ac`), so every cold-cache download of beta.48 fails -verification on all platforms. - -## Recovery procedure - -Run these commands with an owner-authenticated `gh` session. Replace every -`` before running a command. - -1. Confirm the authenticated account and repository. +1. Confirm the affected release is broken by comparing its tag commit with an + attached asset attestation. ```sh - gh auth status - gh api user --jq .login - gh api repos// --jq .full_name + gh api repos///commits/ --jq .sha + gh attestation verify --repo / --format json \ + | jq -r '.[].verificationResult.signature.certificate.sourceRepositoryDigest' ``` -2. Determine the release commit and release tag from the release PR and the - failed workflow run. - - ```sh - gh pr view --repo / --json mergeCommit --jq .mergeCommit.oid - gh run view --repo / --json headSha - ``` +2. Merge the pending release-please release PR for the affected branch. This + creates the next version and starts a normal head-based publish run. - The release commit must be the commit validated by the workflow. Do not use - the current branch head when recovering an older release. - -3. Create the release tag at the validated release commit. +3. Verify the new runner release and the package release that pins it. The + runner tag commit, every asset's attestation digest, and the publish run's + `headSha` must match. ```sh - gh api repos///git/refs \ - -f ref=refs/tags/ \ - -f sha= + gh run view --repo / --json headSha --jq .headSha + gh api repos///commits/ --jq .sha + gh attestation verify --repo / --format json \ + | jq -r '.[].verificationResult.signature.certificate.sourceRepositoryDigest' ``` -4. Download the native release input artifact and create the draft release. - Creating only the tag is not sufficient: the Actions token also receives - HTTP 403 when it tries to create the draft release. +The 2026-07-17 recovery followed this procedure: runner beta.48 was left +broken, the next release-please PR produced runner beta.49, and the following +package release pinned beta.49. The normal publish run, tag, and attestations +all resolved to the same commit. - ```sh - gh run download --repo / \ - -n native-cli-release-input -D - gh release create --repo / \ - --draft --title \ - --notes-file /release-input/release-notes.md \ - --target - ``` +## Why a broken release cannot be revived - Add `--prerelease` when the release tag is a prerelease tag. +Do not create a new tag, draft, or publish run for the historical version. +Revival is structurally impossible for three independent reasons: -5. Rerun the failed publish job of the original run whose head is the release - commit, and approve the release environment if prompted. Do not dispatch a - new `recovery-target` run to publish instead: a dispatch run on a later - head cannot produce valid attestations for the release commit, and the - workflow rejects it (see "Publishing invariant" above). +1. Rerunning a workflow replays the workflow definition from the original + commit. If that workflow definition was itself broken, rerunning it repeats + the failure. +2. A new workflow run attests its own `github.sha`, which is the current branch + head. It cannot create attestations for an earlier release commit. +3. The release resolver binds the historical version to its original release + commit. A later publish run therefore produces a tag-to-attestation digest + mismatch even when an owner pre-creates the tag or draft release. - ```sh - gh run rerun --repo / --failed - ``` +For example, beta.48's tag pointed at `2d8d1b94` while its published asset +attestations carried `2c73c6ac`. Reusing that version would preserve the same +invariant violation. -6. If an older `dispatcher-publish` run is waiting for the `cli-release` - environment, cancel it before retrying the recovery. The workflow uses a - concurrency group without automatic cancellation, so an old approval-waiting - run can block the newer run indefinitely. +## When rerun recovery is valid - ```sh - gh run list --repo / --workflow dispatcher-publish.yml \ - --branch --limit 20 - gh run cancel --repo / - ``` +Rerun only when all of the following are true: -7. Recover the Unity package release only after the runner release has - completed and its assets are available. The package sync script creates a - release at the historical release commit and can hit the same 403; create - that release as the owner before rerunning the sync. - - ```sh - gh release create v --repo / \ - --title v \ - --notes-file \ - --target - ``` - - Add `--prerelease` when the package release is a prerelease. - - Run the package sync or the relevant post-publish job after the runner - release assets are complete. The sync reuses an existing correctly targeted - release. - -8. Complete the workflow-dispatch recovery manually. Push-only post-publish - steps are intentionally skipped for a `workflow_dispatch` run. - - ```sh - gh pr edit --repo / \ - --remove-label 'autorelease: pending' \ - --add-label 'autorelease: tagged' - gh workflow run release-please.yml --repo / --ref \ - -f branch= - ``` +- The original failed run's `headSha` is the intended release commit. +- The workflow definition at that commit is known to be healthy. +- The release tag and draft release, if they exist, point at that same commit. -## Scope of the workflows +Check the original run before rerunning it: -`native-cli-publish.yml` is the workflow with `recovery-target` and now emits -this runbook path when tag or draft-release creation returns the observed 403. -The creation and reuse logic is unchanged. Its publish job refuses to publish -assets from a recovery dispatch run whose head differs from the release -commit, because such a run cannot produce matching attestations. +```sh +gh run view --repo / --json headSha,workflowName,url +gh api repos///commits/ --jq .sha +``` -`dispatcher-publish.yml` has no recovery-target input and requires its release -target to equal `GITHUB_SHA`, so it has no historical-commit creation path. -Its concurrency group still matters during recovery because an older run can -block a retry. +If the run used an older workflow revision that lacks a required fix, or its +head SHA differs from the intended release commit, do not rerun it. Use the +roll-forward procedure instead. -`scripts/sync-release-please-package-releases.sh` can create package releases -at release-please commits from the repository history. This runbook's owner -pre-creation step covers that path; the runner release must be recovered first. +## Operational notes -Do not add GitHub Actions to the tag-ruleset bypass actors. That would grant -the bot tag deletion and force-update capabilities without addressing the -historical-commit restriction. +- Do not dispatch `recovery-target` to publish a historical release. It is + limited to validation because a later run cannot produce valid attestations + for an earlier commit. +- Do not grant GitHub Actions tag-ruleset bypass permissions. That does not + resolve the attestation invariant and expands bot authority unnecessarily. +- A broken release may remain published while the roll-forward release is cut; + the dispatcher rejects it, so it is harmless once consumers use the newer + package pin. From ca1cb734db0a84f776109fc778f16db55eb0ca78 Mon Sep 17 00:00:00 2001 From: hatayama Date: Fri, 17 Jul 2026 23:49:03 +0900 Subject: [PATCH 2/2] Complete executable release recovery steps Include asset download, rerun, approval, and concurrency guidance so the recovery runbook can be followed without inferring missing operational commands. --- docs/release-recovery-runbook.md | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/docs/release-recovery-runbook.md b/docs/release-recovery-runbook.md index 6396485a5..fff224d81 100644 --- a/docs/release-recovery-runbook.md +++ b/docs/release-recovery-runbook.md @@ -13,8 +13,10 @@ do not try to repair the broken release in place. attached asset attestation. ```sh + gh release download --repo / \ + --pattern --dir gh api repos///commits/ --jq .sha - gh attestation verify --repo / --format json \ + gh attestation verify / --repo / --format json \ | jq -r '.[].verificationResult.signature.certificate.sourceRepositoryDigest' ``` @@ -68,17 +70,31 @@ Check the original run before rerunning it: ```sh gh run view --repo / --json headSha,workflowName,url gh api repos///commits/ --jq .sha +gh run view --repo / --log-failed ``` If the run used an older workflow revision that lacks a required fix, or its head SHA differs from the intended release commit, do not rerun it. Use the roll-forward procedure instead. +When all conditions hold, rerun the failed jobs from the original run and +approve the `cli-release` environment if GitHub requests approval: + +```sh +gh run rerun --repo / --failed +``` + ## Operational notes - Do not dispatch `recovery-target` to publish a historical release. It is limited to validation because a later run cannot produce valid attestations for an earlier commit. +- Cancel an older run waiting for `cli-release` approval before retrying. The + workflow concurrency group does not cancel it automatically, so it can block + the newer run indefinitely. +- Delete a broken release only after the version bump is merged. Deleting it + first can make the resolver target the historical commit again and cause + later push builds to fail. - Do not grant GitHub Actions tag-ruleset bypass permissions. That does not resolve the attestation invariant and expands bot authority unnecessarily. - A broken release may remain published while the roll-forward release is cut;