fix(docker): trixie base so arm64 native prebuilds load; execute bindings per-arch at build time - #384
Merged
Merged
Conversation
…ings per-arch at build time
Every arm64 image since v0.33.0 crash-loops at startup:
better-sqlite3 v13's bundled linux-arm64 prebuild requires glibc >= 2.38
and the bookworm base ships 2.36. amd64 was unaffected only because its
prebuild's glibc floor happens to fit — prod never saw it.
The deps stage's `npm rebuild better-sqlite3 onnxruntime-node` defense
stopped working silently: npm's allow-scripts gating skips the build,
prints a warning, and exits 0 (reproduced at the pinned digest), and v13
tarballs no longer produce a working binding from source anyway (compile
completes without linking; upstream moved to bundled prebuilds and set
gypfile:false in 13.0.2).
Fix:
- Base image node:24-slim (bookworm, glibc 2.36) -> node:24-trixie-slim
(glibc 2.41), digest-pinned as before.
- Replace the dead rebuild with an executed assertion: the deps stage
RUNs new Database(':memory:') and require('onnxruntime-node'). Release
builds are linux/amd64 + linux/arm64, so this executes on both arches
(arm64 under QEMU) and a glibc/arch mismatch fails the build instead
of shipping a crash-looping image.
- Drop the now-unused python3/make/g++ from the deps stage (the build
stage keeps its own).
Validated locally: arm64 local target builds and serves (healthz, MCP
initialize, index rebuild, embedding pass — both bindings exercised);
amd64 deps stage passes under emulation; remote target builds on trixie.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PR Code Suggestions ✨No code suggestions found for the PR. |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
aliasunder
added a commit
that referenced
this pull request
Jul 31, 2026
## Motivation Every arm64 image from v0.33.0 through v0.33.4 crash-looped at startup (better-sqlite3 v13's linux-arm64 prebuild requires glibc >= 2.38; the bookworm base had 2.36) and stayed invisible for five days across four releases: prod is amd64, and trivy-pr — the only PR-time image build — builds amd64 only, so nothing ever *executed* an arm64 artifact before release. PR #384 fixed the image and added the deps-stage assertion that executes both native bindings per-arch at build time. This PR makes executed-per-arch a PR gate. ## What the job does New `Arch Smoke` workflow, matrix over native runners (`ubuntu-latest` amd64, `ubuntu-24.04-arm` arm64 — no QEMU). Each leg: 1. **`docker build --target local`** — natively executes the Dockerfile deps-stage native-binding assertion for its arch (`new Database(':memory:')` + `require('onnxruntime-node')`). 2. **Boot smoke** — runs the image against a small inline fixture vault (frontmatter, tags, wikilink pair, task line — real FTS inserts) with `EMBEDDING_ENABLED=false` / `MEMORY_ENABLED=false` and a per-run generated token. `/healthz` 200 is a true proof of the incident class: `server.ts` opens the search DB, awaits the blocking FTS rebuild, and opens the OAuth DB before `app.listen`, exiting 1 on any failure. Then an authenticated MCP `initialize` round-trip is asserted (HTTP 2xx + `mcp-session-id` header + `serverInfo` in the body — a JSON-RPC error still returns 200, hence the body assertion). 3. **arm64 only**: `docker build --target remote` as a compile check — the first-ever arm64 build of the remote target at PR time. Build-only: booting remote needs Obsidian Sync credentials, and amd64 remote coverage is already trivy-pr's job. ## Design notes - **Check contexts are `arch-smoke (amd64)` / `arch-smoke (arm64)`** — explicit job `name` with a friendly arch key, so the ruleset contexts stay stable if runner labels ever change. Adding them to the branch ruleset is a follow-up settings change after this merges. - **Cache: explicit per-arch `scope=arch-smoke-<arch>`**, a deliberate divergence from the repo's bare `type=gha`: trivy-pr (two targets) plus both smoke legs writing `mode=max` into the one default scope would evict each other — worst for arm64, which shares no blobs with the amd64 writers. `cache-from` also lists the bare default scope so the amd64 leg piggybacks trivy-pr's layers (no-op on arm64). The `APT_UPGRADE_DATE` cache-bust step is reused verbatim from trivy.yml — a differing build-arg would change the apt-upgrade layer hash and break that sharing. - **No path filters** — a required check gated by `paths:` leaves docs-only PRs stuck on "Expected" forever; trivy-pr also runs unconditionally. - **`push: main` trigger** seeds the per-arch cache into the default-branch scope (readable by all PRs; PR-branch caches are not shared across PRs) and acts as a post-merge canary. - **Crash-loop fast-fail**: the healthz poll checks `docker inspect '{{.State.Running}}'` each iteration, so the incident's exact failure mode (startup `process.exit(1)`) fails in ~2s instead of burning the 60s poll window. Diagnostics step (`if: failure()`) dumps `docker ps -a`, container state JSON, and logs. - **`timeout-minutes: 20`** is a kill-switch cap (default is 360), not the expected runtime — expected ~3-5 min warm-cache, ~8-12 min cold, legs parallel. - **`fail-fast: false`** — arch failures are independent signals; one leg must not cancel the other. ## Verification - `actionlint`: zero findings on the new workflow; `npm run prettier:check` green. - **Full smoke executed locally on native arm64** (Apple Silicon, the incident-class arch): the exact script block passes under `bash -e` in 2.2s — healthz ready on attempt 2, MCP initialize round-trip clean, and the initialize curl exits naturally when the SSE response stream closes (no `--max-time` timeout). - **Failure path mutation-tested locally**: booting the container without `MCP_AUTH_TOKEN` (startup exit 1) trips the crash-detection branch on poll attempt 1 with the startup error visible in `docker logs`. - On this PR: both legs run for the first time in CI (new `pull_request` workflows run from the PR branch). A deliberate red push (probe pointed at a wrong port) will verify the gate bites and the diagnostics render, then be reverted. Also updates CONTRIBUTING.md's required-checks item to name the new checks. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Added automated smoke testing for native amd64 and arm64 Docker builds. * Tests now boot images, verify health and authenticated initialization, and collect diagnostics on failure. * **Documentation** * Updated pull request guidance to note that architecture smoke test failures block merging. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Every arm64 image since v0.33.0 (July 25) crash-loops at startup — all Apple Silicon local-mode users are affected:
better-sqlite3 v13 (Dependabot #369) bundles its prebuilds in the npm tarball, and the linux-arm64 prebuild requires glibc >= 2.38 while the bookworm base ships 2.36. amd64 was unaffected only because its prebuild's glibc floor happens to fit — which is why prod (Lightsail, amd64) never surfaced it and no CI check caught it: nothing ever executes the arm64 artifact.
Why the existing defense failed
The deps stage ran
npm ci --ignore-scripts && npm rebuild better-sqlite3 onnxruntime-nodeprecisely to compile bindings against the image's libc. Two independent failures made it dead weight:npm rebuild— it prints a warning, reports "rebuilt dependencies successfully", and exits 0 without building anything. Reproduced in a clean container at the exact pinned base digest..node(upstream moved to bundled prebuilds; 13.0.2 setsgypfile: false, making source builds a permanent no-op).Both shipped image arches contain no compiled binding at all — they run on bundled prebuilds, so amd64 is also one upstream prebuild-toolchain bump away from the same crash.
Fix
node:24-trixie-slim(glibc 2.41 ≥ 2.38), digest-pinned as before. Aligns with upstream's prebuild-only direction instead of fighting it.RUNsnew Database(':memory:')+require('onnxruntime-node'). Release builds arelinux/amd64,linux/arm64, so this executes on both architectures (arm64 under QEMU) — a glibc/arch mismatch now fails the build on the affected arch instead of shipping a crash-looping image. This makes per-arch validation a hard release gate.python3 make g++from the deps stage (the build stage keeps its own copy).Validation
.env— healthz ok, authenticated MCPinitializereturns serverInfo, index rebuild + embedding pass complete (both native bindings exercised for real).--platform linux/amd64emulation — assertion passes, proving the cross-arch gate works exactly as CI will run it.0.32.0(better-sqlite3 12.x) opens a database on arm64;0.33.2and:latestdo not. Note: barerequire('better-sqlite3')passes on broken images — v13 defers the native load — so the assertion opens a real database.🤖 Generated with Claude Code