Skip to content

ci: build Linux (static musl) in CI and publish artifacts to DO Spaces - #656

Merged
logbie merged 5 commits into
mainfrom
ci/blacksmith-linux-musl-spaces-publish
Jul 28, 2026
Merged

ci: build Linux (static musl) in CI and publish artifacts to DO Spaces#656
logbie merged 5 commits into
mainfrom
ci/blacksmith-linux-musl-spaces-publish

Conversation

@logbie

@logbie logbie commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

What this does

Three changes that only make sense together: CI now builds Linux, artifacts publish to DigitalOcean Spaces, and runners were right-sized per job.

1. Linux builds move into CI, statically linked against musl

New build-linux job produces wfl and wfl-lsp for x86_64-unknown-linux-musl. Until now the project shipped a Windows MSI nightly and no Linux artifact from CI at all - the tarballs that exist were built by hand on an out-of-band host against glibc.

That is issue #616: a binary linked against glibc 2.39 will not start on Debian 12 or Ubuntu 22.04. Building on Ubuntu 22.04 instead would only lower the floor; a static musl binary removes it, closing the class of bug rather than deferring it.

wfl-lsp is now in the Linux tarball. The MSI has always shipped both; the Linux tarball carried only wfl, making editor support Windows-only by accident of packaging. Layout is otherwise unchanged and remains a strict superset, so existing installers keep working.

2. Publishing to Spaces

scripts/publish_spaces.sh is the only writer to the bucket, so the layout lives in one place. Spaces is canonical; the GitHub Release is a mirror.

Three things it gets right that are easy to get wrong:

  • Objects upload public-read. The bucket's contents were private - the CDN returned AccessDenied for every key, so nothing could actually install from the published URL. The script now pulls every immutable object back through the CDN and compares its SHA-256 against the local artifact, so neither an ACL regression nor a corrupted upload can hide behind a green run. (Rolling keys get a status-code check only: at max-age=60 the CDN may still legitimately serve the previous publish, so a hash comparison there would be flaky by design.)
  • Rolling pointers get max-age=60, not the CDN's 1-hour default, so an installer can't fetch a stale "latest" for an hour after a successful publish.
  • AWS CLI v2.23+ sends CRC32 headers that Spaces rejects with a 400. AWS_REQUEST_CHECKSUM_CALCULATION=when_required handles it.
  • The CDN host is derived from the endpoint, not hardcoded, so a publish to a non-nyc3 region verifies the host it actually uploaded to. status.json is built with jq -n --arg rather than a here-doc, so a quote or newline in a version or branch name can't publish unparseable JSON.

Two failure orderings matter, and the obvious choice for each is the wrong one:

  • Inside the script, all immutable objects upload first; rolling pointers, SHA256SUMS and status.json are written only once every one of them succeeded. Writing each pointer next to its immutable upload reads better, but under set -e a later failure would leave wfl-latest-linux-x86_64.tar.gz on the new build while the Windows pointer, the checksums and status.json still described the previous one - a mixed release consumers could observe indefinitely.
  • Inside the release job, the publish runs before the nightly tag and Release. Publishing last looks safer, but the tag is the success marker check-for-changes reads: it resolves the newest nightly-* tag and sets should_build=false when it matches HEAD, so a publish failure behind an already-pushed tag would never be retried and the rolling downloads would stay stale until an unrelated commit landed. Failing before the tag is written keeps the run retryable.

3. Runner sizing

Cost is strictly linear in vCPU, so a bigger runner only saves money if wall-clock drops proportionally. Sized per job, not blanket.

Job Before After
nightly build (Windows) 4vcpu-windows-2025 8vcpu-windows-2025
nightly build-linux (new) 8vcpu-ubuntu-2404
fmt, auto-fmt, update-security-doc 4vcpu-ubuntu-2404 2vcpu-ubuntu-2404-arm
fuzz-check, both bump-version, config-lint 4vcpu-ubuntu-2404 4vcpu-ubuntu-2404-arm

The Windows change fixes a spiral, not just a slow job. It had died on its own timeout twice; a cancelled job never runs its cache-save step, so each failure left the next nightly equally cold and it timed out identically. Raising the timeout treated the symptom. Compiling 112 release-mode test binaries parallelizes well: ~70min at $0.016/min ($1.12) becomes ~30min at $0.032/min ($0.96) - cheaper and the loop breaks.

clippy-and-test, integration-tests, database-tests and run-wfl-programs deliberately stay on x64 even though nothing in them requires it. They gate the x86_64 binary we ship; moving them to ARM would mean testing an architecture we don't release. The ~38% saving isn't worth that gap.

Also on ci.yml: concurrency with cancel-in-progress for every ref except main (the post-merge bump-version job commits and then tags, and cancelling between those writes would leave main carrying an untagged version bump), CARGO_INCREMENTAL: 0, and retention-days: 7 on artifact uploads (the 90-day default retained ~90 nightlies of MSIs on separately-billed storage). .github/actionlint.yaml declares the Blacksmith runner labels so actionlint stops flagging every runs-on: in the repo as unknown.

Test evidence

Per the testing policy, the property claimed is portability, and it's verified at the real boundary rather than by proxy:

  • Both binaries must have no PT_INTERP program header (readelf) or the job fails - a binary naming no interpreter cannot ask a loader for libc. A musl target does not guarantee a static binary: one dynamic dep silently reintroduces the floor while CI still goes green. (Not file output: Rust's musl target emits a static-PIE, which file calls static-pie linked, so matching on statically linked fails a perfectly static binary.)
  • The gate runs after packaging and extracts the shipped tarball inside debian:12-slim - the exact distro where the glibc build failed in Publish a Linux nightly, and build it against an older glibc (or musl) — current Ubuntu 24.04 builds require glibc 2.38+ #616 - then runs both binaries and TestPrograms/basic_syntax_comprehensive.wfl from it. Testing the pre-package binaries would leave strip and the tar round-trip unverified on the only distro this gate exists for.
  • The publish script verifies public readability through the CDN and fails the job otherwise.

Red evidence: the current main artifact fails the Debian 12 gate by construction - that's the content of #616. This PR is the Green.

Risk

aws-lc-sys (via reqwest 0.13) compiles C and assembly and is the one dependency with a musl story to get wrong. There is no openssl-sys anywhere - TLS is rustls end to end - so the usual blocker doesn't apply, and musl-tools/cmake/clang are installed for it. If it fights, the clean escape hatch is pinning reqwest to rustls-tls-ring, which belongs in its own PR.

Risk class: R3 (backward compatibility - artifact layout and download URLs are user-facing).

Docs

Docs/reference/supported-platforms.md no longer claims static-musl Linux has "no CI lane" and is unverified; it stays Tier 2, since the new lane is post-merge only and does not run the integration or full TestPrograms suites, and the promotion policy in that document requires both. Docs/02-getting-started/installation.md gains a Linux x86_64 Tarball section covering the canonical CDN download, checksum verification against SHA256SUMS, and install steps for wfl and wfl-lsp.

Closes #616.


View with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is enabled.

Summary by CodeRabbit

  • New Features

    • Added Linux x86_64 nightlies as statically linked musl tarballs, including the language server.
    • Linux release downloads are now available through DigitalOcean Spaces, with GitHub Releases retained as a mirror.
    • Added versioned and “latest” artifact downloads, checksums, release status information, and post-upload availability checks.
  • Improvements

    • Reduced wasted CI work by canceling superseded runs.
    • Improved nightly Windows build reliability and artifact retention.
    • Optimized CI runner selection and build performance across workflows.
  • Documentation

    • Documented Linux publishing, validation, downloads, and CI improvements.

Adds a `build-linux` job to the nightly workflow producing `wfl` and
`wfl-lsp` for x86_64-unknown-linux-musl, statically linked, and publishes
all artifacts to the DigitalOcean Spaces bucket that backs
wfl.nyc3.cdn.digitaloceanspaces.com. Spaces becomes the canonical download
location; the GitHub Release remains as a mirror.

Closes the class of bug behind #616: the previous Linux tarballs were built
out-of-band against glibc 2.39 and would not start on Debian 12 or Ubuntu
22.04. A static musl binary has no libc floor. Two gates enforce it - `file`
must report "statically linked" for both binaries, and both are executed
inside debian:12-slim, the exact distro where the glibc build failed.

wfl-lsp is now included in the Linux tarball, matching the Windows MSI which
has always shipped both. The tarball layout is otherwise unchanged and
remains a superset of what is already published.

Runner right-sizing, per-job rather than blanket:
- nightly Windows build 4 -> 8 vCPU. It had died on its own timeout twice;
  a cancelled job never runs its cache-save step, so each failure left the
  next nightly equally cold and it timed out again. Doubling cores should
  more than halve wall-clock on 112 release-mode test binaries: ~$1.12
  becomes ~$0.96, and the spiral breaks.
- fmt, auto-fmt, update-security-doc -> 2vcpu ARM (no compile).
- fuzz-check, both bump-version jobs, config-lint -> 4vcpu ARM (these do
  compile, so core count is held while taking the 37.5% ARM discount).
- clippy-and-test, integration-tests, database-tests and run-wfl-programs
  deliberately stay on x64: they gate the x86_64 binary we ship.

Also adds concurrency cancel-in-progress and CARGO_INCREMENTAL=0 to ci.yml,
and retention-days: 7 to the artifact uploads (the 90-day default retained
~90 nightlies of MSIs on separately-billed storage).
@coderabbitai

coderabbitai Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Warning

Review limit reached

@logbie, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 14 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: c301a80d-463a-4241-9b04-0cbfb0286ebb

📥 Commits

Reviewing files that changed from the base of the PR and between 527f499 and 5af3f38.

📒 Files selected for processing (7)
  • .github/actionlint.yaml
  • .github/workflows/ci.yml
  • .github/workflows/nightly.yml
  • Dev diary/2026-07-28-linux-musl-build-and-spaces-publishing.md
  • Docs/02-getting-started/installation.md
  • Docs/reference/supported-platforms.md
  • scripts/publish_spaces.sh
📝 Walkthrough

Walkthrough

This PR adds a static musl Linux nightly build and release artifact, introduces DigitalOcean Spaces publishing, right-sizes workflow runners, and adds CI concurrency, compilation, and retention controls. Documentation records the build, publishing, and runner changes.

Changes

Linux nightly delivery

Layer / File(s) Summary
Static Linux build and release integration
.github/workflows/nightly.yml, Dev diary/...
The nightly workflow builds and validates static wfl and wfl-lsp musl binaries, packages them into a tarball, uploads the artifact, verifies it during release assembly, and lists it in GitHub release downloads.
Spaces artifact publishing
scripts/publish_spaces.sh, Dev diary/...
Artifacts are uploaded to immutable and rolling DigitalOcean Spaces paths with checksums, status metadata, cache policies, and CDN verification.
Runner sizing and CI hygiene
.github/workflows/*.yml, Dev diary/...
Selected jobs move to ARM runners, the Windows nightly build uses an 8-vCPU runner, CI cancels superseded runs, incremental compilation is disabled, and artifact retention is reduced.

Estimated code review effort: 4 (Complex) | ~45 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The PR satisfies #616 by publishing a Linux nightly with statically linked x86_64-unknown-linux-musl binaries and portability testing.
Out of Scope Changes check ✅ Passed The added workflow, publishing, and documentation changes all align with the stated Linux-nightly and Spaces publishing objectives.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly captures the main change: Linux musl builds in CI and publishing artifacts to DigitalOcean Spaces.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch ci/blacksmith-linux-musl-spaces-publish

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 527f499789

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/nightly.yml Outdated
for b in wfl wfl-lsp; do
BIN="target/$TARGET/release/$b"
file "$BIN"
if ! file "$BIN" | grep -q "statically linked"; then

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Accept static PIE output from file

In the new build-linux job, the Rust target specification enables both static CRT linking and position-independent executables, so normal x86_64-unknown-linux-musl binaries are static PIE executables. GNU file reports these as static-pie linked, which does not contain the searched text statically linked; consequently this assertion rejects the valid binaries and stops every nightly before the Debian test and packaging steps. Check the ELF interpreter/dynamic dependencies directly or accept the static-PIE result as well.

AGENTS.md reference: AGENTS.md:L138-L140

Useful? React with 👍 / 👎.

Comment thread .github/workflows/nightly.yml Outdated
Comment on lines +712 to +715
# Spaces is the canonical download location; the GitHub Release above is a
# mirror. This runs last so a failed upload cannot leave a half-written
# bucket sitting behind a release that claims the files are there.
- name: Publish artifacts to DigitalOcean Spaces

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Publish canonically before recording nightly success

If credentials, an upload, or CDN verification fails here, the preceding steps have already pushed the nightly tag and created the GitHub release. On the next scheduled run, check-for-changes resolves that release's tag to the current commit and sets should_build=false, so the failed canonical Spaces publication is not retried and its rolling downloads remain stale or partial until another code commit arrives. Publish before creating the success marker, or remove/avoid the tag and release when this step fails.

AGENTS.md reference: AGENTS.md:L134-L137

Useful? React with 👍 / 👎.

Comment thread scripts/publish_spaces.sh Outdated
Comment on lines +68 to +69
put "$TARBALL" "releases/$(basename "$TARBALL")" application/gzip "$IMMUTABLE"
put "$TARBALL" "releases/wfl-latest-linux-x86_64.tar.gz" application/gzip "$ROLLING"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Defer rolling-pointer updates until all uploads succeed

When any later aws call fails, set -e exits after this Linux rolling pointer has already been replaced, while the Windows pointer, SHA256SUMS, and status.json can still describe the previous publish. Consumers can then observe a mixed release indefinitely, contradicting the claimed failure safety. Upload every immutable artifact first and update rolling metadata/pointers only after all required uploads have succeeded.

AGENTS.md reference: AGENTS.md:L134-L140

Useful? React with 👍 / 👎.

Comment on lines +458 to +459
build-linux:
name: Build WFL for Linux (static musl)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Update the supported-platform documentation

This new lane builds musl binaries and executes them on Debian 12, but Docs/reference/supported-platforms.md still says Linux musl has “No CI lane” and is “unverified,” and the installation guide does not document the new canonical tarball. That leaves the repository's user-facing support boundary and installation instructions false immediately after this workflow lands; update those documents alongside the feature.

AGENTS.md reference: AGENTS.md:L20-L22

Useful? React with 👍 / 👎.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 potential issues.

Open in Devin Review

Comment thread scripts/publish_spaces.sh Outdated
# Prove the CDN actually serves what we just uploaded. The bucket was private
# until this pipeline existed, so a silent ACL regression would break every
# install while the workflow still reported success.
CDN="https://${BUCKET}.nyc3.cdn.digitaloceanspaces.com"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Post-publish CDN check always targets one fixed region, ignoring the configured endpoint

The final verification hard-codes the nyc3 region in the CDN hostname (scripts/publish_spaces.sh:116) even though the bucket endpoint and region are configurable, so pointing the script at any other region uploads to the right place but then checks the wrong host and fails the job.

Impact: A successful publish to a non-default region would be reported as a failure, blocking releases even though the files were uploaded correctly.

Region derivation mismatch between upload and verify

The script accepts SPACES_ENDPOINT (default https://nyc3.digitaloceanspaces.com) and AWS_DEFAULT_REGION (default nyc3) as configurable inputs (scripts/publish_spaces.sh:29, scripts/publish_spaces.sh:37), and all uploads go through $ENDPOINT. But the CDN readability check builds CDN="https://${BUCKET}.nyc3.cdn.digitaloceanspaces.com" (scripts/publish_spaces.sh:116) with nyc3 fixed. If the script is invoked against a different region, every uploaded object lives under that region's CDN host, but the loop at scripts/publish_spaces.sh:118-125 fetches from nyc3, gets a non-200, and exits 1 despite a correct publish. In the current workflow the region is always nyc3 so it does not trigger today, but it defeats the script's own configurability contract.

Prompt for agents
In scripts/publish_spaces.sh the CDN verification hardcodes the nyc3 region (CDN="https://${BUCKET}.nyc3.cdn.digitaloceanspaces.com") while SPACES_ENDPOINT and AWS_DEFAULT_REGION are configurable inputs. Derive the region from the configured endpoint/region (e.g. extract the region subdomain from SPACES_ENDPOINT or use AWS_DEFAULT_REGION) so the CDN host used for the post-publish readability check matches where the objects were actually uploaded. Otherwise a publish to any non-nyc3 region will upload correctly but fail the verification step.
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment thread .github/workflows/ci.yml Outdated
Comment on lines +12 to +14
concurrency:
group: ci-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 cancel-in-progress on main can interrupt the version-bump git writes

The new concurrency block in .github/workflows/ci.yml:12-14 applies cancel-in-progress: true to both push (to main) and pull_request events. For rapid successive pushes to main, a superseding push cancels the earlier run, which could interrupt the bump-version job (.github/workflows/ci.yml:555-658) mid-write (e.g., after the commit push but before tagging). This is largely benign because: (a) the version-bump commit is [skip ci] (confirmed in scripts/bump_version.py:372), so bump commits don't create new superseding runs; and (b) the final commit on main is never superseded, so its bump always eventually runs. Still, a genuine human double-push to main during CI could leave a superseded commit's tag unwritten. Worth a quick confirmation that this is acceptable for the post-merge tagging flow.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

The first verify run failed on a perfectly static binary. Rust's musl target
emits a static-PIE, which file(1) reports as "static-pie linked" rather than
"statically linked", so the original grep was a false negative.

Check the absence of a PT_INTERP program header instead. That is the actual
property being claimed - a binary naming no interpreter cannot ask a loader
for libc - and it does not depend on file(1)'s phrasing.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 7

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
.github/workflows/nightly.yml (1)

660-708: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Publish the canonical location before its mirror.

The GitHub Release—and its canonical-download claim—is created before the Spaces upload. If Spaces fails, users see a release pointing at unavailable or stale canonical artifacts. Publish to Spaces immediately after artifact verification, then create the GitHub mirror release.

Also applies to: 712-728

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/nightly.yml around lines 660 - 708, Reorder the nightly
publishing flow so the verified MSI, VSIX, and tarball artifacts are uploaded to
the canonical Spaces location before the “Publish or update nightly release”
step creates the GitHub release. Ensure the Spaces upload succeeds before
proceeding, while preserving the existing artifact validation and GitHub release
asset creation behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/ci.yml:
- Around line 9-14: Update the workflow-level concurrency configuration around
the ci-${{ github.workflow }}-${{ github.ref }} group so in-progress runs on
main are not cancelled before the “Tag the new version” step completes, while
retaining cancellation for non-main refs. Ensure the bump-version commit and tag
sequence remains protected from superseding CI runs.
- Around line 30-33: Declare both custom labels,
blacksmith-2vcpu-ubuntu-2404-arm and blacksmith-4vcpu-ubuntu-2404-arm, under
actionlint’s self-hosted-runner.labels configuration, or replace the affected
runner assignments with supported labels. Apply this for
.github/workflows/ci.yml ranges 30-33, 48-50, and 573-576;
.github/workflows/auto-fmt.yml 13-16; .github/workflows/update-security-doc.yml
11-14; .github/workflows/versioning.yml 8-11; and
.github/workflows/wfl-config-lint.yml 14-17, ensuring all custom labels resolve
during validation and execution.

In @.github/workflows/nightly.yml:
- Around line 539-566: Update the Debian 12 validation job after the “Package
tarball” step to extract the generated ${DIR}-${SHORT_SHA}.tar.gz, execute both
extracted wfl and wfl-lsp binaries, and run the established full TestPrograms/
compatibility suite against the extracted wfl binary. Ensure this gate validates
the packaged tarball contents rather than the pre-package target binaries, while
preserving the existing compatibility expectations.
- Line 133: Add the new runner labels used by the nightly workflow’s runs-on
entries to actionlint’s custom-label configuration, covering both occurrences of
blacksmith-8vcpu-windows-2025. Preserve the existing workflow runners and ensure
the workflow quality gate recognizes these labels.

In `@Dev` diary/2026-07-28-linux-musl-build-and-spaces-publishing.md:
- Around line 3-12: Update the release narrative around the “What changed”
section and the corresponding later section to classify this as R3 and record
auditable acceptance criteria mapped to specific tests, observed Red evidence,
executed test layers, and residual risk. Keep the existing Linux build, Spaces
publishing, and runner changes intact while adding the required verification
evidence for backward compatibility.

In `@scripts/publish_spaces.sh`:
- Around line 113-125: Update the CDN verification loop in
scripts/publish_spaces.sh to iterate over each newly published immutable
tarball/MSI artifact, download it from its CDN URL, and compare its SHA-256
against the corresponding local artifact before succeeding. Retain the existing
failure behavior and ensure checksum mismatches or download failures exit
nonzero; do not rely only on SHA256SUMS and status.json.
- Around line 100-110: Replace the heredoc-based status.json generation with
JSON serialization using an encoder such as jq -n --arg, covering COMMIT_SHA,
VERSION, PUBLISHED artifact names, BRANCH, and the timestamp so quotes and
control characters are escaped correctly. Ensure jq is available in the GitHub
Actions runner or image before using it, while preserving the existing field
names and values.

---

Outside diff comments:
In @.github/workflows/nightly.yml:
- Around line 660-708: Reorder the nightly publishing flow so the verified MSI,
VSIX, and tarball artifacts are uploaded to the canonical Spaces location before
the “Publish or update nightly release” step creates the GitHub release. Ensure
the Spaces upload succeeds before proceeding, while preserving the existing
artifact validation and GitHub release asset creation behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: f4bfd6cc-a608-4973-bacd-a41253081bc6

📥 Commits

Reviewing files that changed from the base of the PR and between 4f6cf7d and 527f499.

📒 Files selected for processing (8)
  • .github/workflows/auto-fmt.yml
  • .github/workflows/ci.yml
  • .github/workflows/nightly.yml
  • .github/workflows/update-security-doc.yml
  • .github/workflows/versioning.yml
  • .github/workflows/wfl-config-lint.yml
  • Dev diary/2026-07-28-linux-musl-build-and-spaces-publishing.md
  • scripts/publish_spaces.sh

Comment thread .github/workflows/ci.yml Outdated
Comment thread .github/workflows/ci.yml
# with cores, so doubling them should more than halve wall-clock: ~70min
# at $0.016/min = $1.12 becomes ~30min at $0.032/min = $0.96. Cheaper AND
# it breaks the timeout death spiral.
runs-on: blacksmith-8vcpu-windows-2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Declare the new Blacksmith labels to actionlint.

actionlint currently rejects both runner labels. Add them to its custom-label configuration (or use recognized labels) so the workflow quality gate passes.

Also applies to: 464-464

🧰 Tools
🪛 actionlint (1.7.12)

[error] 133-133: label "blacksmith-8vcpu-windows-2025" is unknown. available labels are "windows-latest", "windows-latest-8-cores", "windows-2025", "windows-2025-vs2026", "windows-2022", "windows-11-arm", "ubuntu-slim", "ubuntu-latest", "ubuntu-latest-4-cores", "ubuntu-latest-8-cores", "ubuntu-latest-16-cores", "ubuntu-24.04", "ubuntu-24.04-arm", "ubuntu-22.04", "ubuntu-22.04-arm", "macos-latest", "macos-latest-xlarge", "macos-latest-large", "macos-26-intel", "macos-26-xlarge", "macos-26-large", "macos-26", "macos-15-intel", "macos-15-xlarge", "macos-15-large", "macos-15", "macos-14-xlarge", "macos-14-large", "macos-14", "self-hosted", "x64", "arm", "arm64", "linux", "macos", "windows". if it is a custom label for self-hosted runner, set list of labels in actionlint.yaml config file

(runner-label)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/nightly.yml at line 133, Add the new runner labels used by
the nightly workflow’s runs-on entries to actionlint’s custom-label
configuration, covering both occurrences of blacksmith-8vcpu-windows-2025.
Preserve the existing workflow runners and ensure the workflow quality gate
recognizes these labels.

Source: Linters/SAST tools

Comment thread .github/workflows/nightly.yml
Comment thread Dev diary/2026-07-28-linux-musl-build-and-spaces-publishing.md
Comment thread scripts/publish_spaces.sh Outdated
Comment thread scripts/publish_spaces.sh
Addresses review feedback on the Linux musl build and Spaces publishing:

- publish_spaces.sh wrote each rolling pointer next to its immutable
  upload, so under `set -e` a later failure could leave "latest" pointing
  at the new build while SHA256SUMS, status.json, and the other pointer
  still described the previous one - a mixed release consumers could
  observe indefinitely. Uploads are now two-phase: immutable objects
  first, rolling pointers and metadata only after all of them succeed.
- The Spaces publish ran after the nightly tag and GitHub Release were
  created. Those are the success marker check-for-changes reads, so a
  publish failure behind an already-pushed tag would set
  should_build=false on the next scheduled run and never be retried. The
  canonical publish now runs before the tag is pushed.
- Docs ship with the feature: supported-platforms.md no longer claims
  static-musl Linux has no CI lane (it stays Tier 2, since the lane is
  post-merge only and skips the integration/TestPrograms suites), and
  installation.md documents the canonical Linux x86_64 tarball, its
  checksum verification, and install steps.
- Dev diary records the PT_INTERP assertion rationale and both failure
  orderings.

Co-authored-by: Codesmith <codesmith-bot@users.noreply.github.com>
@logbie

logbie commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator Author

Verification evidence

The musl build is real, and the staticness gate earned its keep

First verify run failed on a perfectly static binary — a false negative in my own check:

wfl: ELF 64-bit LSB pie executable, x86-64, static-pie linked, not stripped
::error::target/.../wfl is not statically linked - wfl#616 would reappear

Rust's musl target emits a static-PIE, which file(1) describes as static-pie linked, not statically linked. Grepping for the latter reports a good binary as broken. Fixed in 8d4f0f6 by asserting the actual property instead — the absence of a PT_INTERP program header. A binary that names no interpreter cannot ask a loader for libc at startup, and that test doesn't depend on file(1)'s phrasing.

Re-run: green end to end

OK   Build wfl and wfl-lsp
OK   Assert the binaries are statically linked
OK   Prove portability on Debian 12 (wfl#616 regression gate)
OK   Smoke test
OK   Package tarball

Staticness, both binaries:

wfl:     no PT_INTERP segment - statically linked
wfl-lsp: no PT_INTERP segment - statically linked

#616 regression gate — the real boundary, not a proxy. Both binaries executed inside debian:12-slim, the exact distro where the glibc build fails:

WebFirst Language (WFL) version 26.7.54
wfl-lsp 0.1.0

Smoke test ran TestPrograms/basic_syntax_comprehensive.wfl to completion. Tarball: 11.8 MB with both binaries.

aws-lc-sys was a non-issue

Flagged as the one dependency with a musl story to get wrong. The full musl build of both binaries completed in 2m22s with no intervention beyond musl-tools/musl-dev/cmake/clang. The rustls-tls-ring escape hatch is not needed.

Spaces publish path

Verified directly against the live bucket with a throwaway key (_cicheck/probe.txt, since removed):

  • aws s3 cp --acl public-read with the wfl-releases key: OK
  • Fetch back through the CDN: HTTP 200, correct body

That confirms the fix for the AccessDenied problem — the bucket's existing objects are private, so nothing could install from the published URL before this change.

One honest caveat: the probe ran under AWS CLI v1, which doesn't send the CRC32 integrity headers that Spaces rejects. So the AWS_REQUEST_CHECKSUM_CALCULATION=when_required setting is a documented precaution against v2 behaviour on the runner, not something this probe exercised. It's harmless if unnecessary, but it'll be genuinely proven only on the first real nightly.

Not verified yet

The full release job (tag → GitHub Release → publish_spaces.sh) has not been run. Dispatching the real nightly from this branch would have deleted and recreated the existing nightly-2026-07-28 release and tagged it against an unmerged branch, so I verified the Linux leg in isolation on a throwaway branch instead (now deleted). The release path gets its first real exercise on the nightly after merge — worth watching that run.

logbie and others added 2 commits July 28, 2026 17:52
Addresses a second round of review feedback:

- The CDN readability check hardcoded the nyc3 region while SPACES_ENDPOINT
  and AWS_DEFAULT_REGION are configurable, so any other region would
  upload correctly and then fail verification against a host the objects
  were never written to. The region is now derived from the endpoint.
- A 200 on SHA256SUMS proved a key exists, not that the artifact bytes an
  installer downloads are the ones we built. Each immutable object is now
  pulled back through the CDN and its SHA-256 compared against the local
  file. Rolling keys keep the status-code check only: they carry
  max-age=60, so a hash comparison there would be flaky by design.
- status.json was assembled with a here-doc, so a quote or newline in
  VERSION, BRANCH, or an artifact name published unparseable JSON. It is
  now serialized with `jq -n --arg`.
- The Debian 12 gate ran the pre-package binaries, leaving `strip` and the
  tar round-trip unverified on the only distro the gate exists for. It now
  runs after packaging and extracts the shipped tarball inside
  debian:12-slim, running both binaries and a TestPrograms program from it.
- ci.yml no longer cancels in-progress runs on main. The post-merge
  bump-version job commits and then tags; cancelling between those writes
  would leave main with an untagged version bump that no later run repairs.
- Added .github/actionlint.yaml declaring the Blacksmith runner labels, so
  actionlint stops reporting every runs-on in the repo as unknown.
- Dev diary records the R3 risk class, acceptance criteria mapped to gates,
  Red evidence, layers executed, and residual risk.

Co-authored-by: Codesmith <codesmith-bot@users.noreply.github.com>
@logbie
logbie merged commit e211a50 into main Jul 28, 2026
17 checks passed
@logbie
logbie deleted the ci/blacksmith-linux-musl-spaces-publish branch July 28, 2026 18:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Publish a Linux nightly, and build it against an older glibc (or musl) — current Ubuntu 24.04 builds require glibc 2.38+

1 participant