Skip to content

test(ci): self-test that --cli-install=binary generates and runs against a real release - #655

Merged
joshua-temple merged 1 commit into
mainfrom
test/binary-install-selftest
Jul 21, 2026
Merged

test(ci): self-test that --cli-install=binary generates and runs against a real release#655
joshua-temple merged 1 commit into
mainfrom
test/binary-install-selftest

Conversation

@joshua-temple

Copy link
Copy Markdown
Collaborator

Part of #654.

Problem

--cli-install=binary (#653) emits a self-contained CLI install whose live path (resolve tag, download, sha256 gate, keyless cosign verify with cosign fetched by pinned checksum) is not exercised by the PR gate or by any fleet lane. It needs to be proven that binary mode both generates as expected and runs as expected on real runners, without standing up a fleet member.

Fix

A standalone self-test workflow, cli-install-selftest.yaml, workflow_dispatch + weekly schedule only (never push/PR, so its network dependency cannot flake the merge gate). One matrix job across linux and macos, both arches, proves both halves so what is generated is provably what runs:

  1. Generate: generate-workflow --cli-install=binary on a minimal manifest, then assert the output carries no setup-cli action and no sigstore/cosign-installer, keeps the mandatory sha256sum -c gate, and pins the cosign identity/issuer. actionlint runs the generated file.
  2. Run: extract the emitted "Setup CLI" step's run body with yq and execute it verbatim against the published v1.0.0, then assert cascade version reports 1.0.0. On linux the log MUST show cosign signature on checksums.txt verified. (fail if it silently fell back). On macos, where binary mode has no pinned cosign, it asserts the documented sha256-only fallback fired, so the linux-only cosign behavior is a tested outcome rather than a silent gap.

permissions: contents: read; github.token only, no new secret.

Verification

  • actionlint (repo's pinned commit) on the new workflow: clean.
  • Local run on darwin/arm64: extraction produced the 254-line run body; executing it installed the real v1.0.0, the sha256 gate passed, cascade version printed 1.0.0, and both documented fallback warnings fired (no pinned cosign for darwin/arm64 -> sha256-only).
  • The linux keyless-cosign path proves out on first dispatch; I will dispatch and confirm after merge.

…unners

Add a self-contained workflow that both generates a binary-mode workflow and executes the emitted Setup CLI step against a published release, on a weekly schedule and on demand only (never on push/PR, so the live cosign path is not a merge gate). Asserts no third-party install action is emitted, the sha256 gate and pinned cosign identity are present, keyless cosign verification succeeds on linux, and the documented sha256-only fallback fires on macos.

Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
@joshua-temple
joshua-temple merged commit 78c186f into main Jul 21, 2026
17 checks passed
@joshua-temple
joshua-temple deleted the test/binary-install-selftest branch July 21, 2026 19:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant