Skip to content

docker: correct the sibling-clone cache claim, which cost an outage - #130

Merged
AdaWorldAPI merged 2 commits into
mainfrom
claude/q2-osm-map-reencoding-56p5e2
Aug 14, 2026
Merged

docker: correct the sibling-clone cache claim, which cost an outage#130
AdaWorldAPI merged 2 commits into
mainfrom
claude/q2-osm-map-reencoding-56p5e2

Conversation

@AdaWorldAPI

@AdaWorldAPI AdaWorldAPI commented Aug 14, 2026

Copy link
Copy Markdown
Owner

Merging this also unblocks the deploy — see the last section. Not draft for that reason.

The false claim

The comment above the sibling git clone layer said:

The COPY . /build/q2 above changes on every q2 commit, invalidating this RUN layer too, so each build re-clones fresh (no stale-cache problem the old pin was guarding against).

That is true per q2 commit and false per deploy. Docker busts a layer when an input changes, and the sibling repos are not inputs — nothing in the Dockerfile can observe that lance-graph's HEAD moved. A redeploy of the same q2 commit therefore reuses whatever clones the previous build happened to take.

What it cost, today

  1. 12:53 — q2 osm lance: degrade instead of panicking past Arrow's i32 array ceiling #129 merged → Railway starts building.
  2. 12:53 — OGAR fix(bd-3o8zmz46): evidence-based session expiry handling in hub-client quarto-dev/q2#268 merged in the same minute, adding osm_street_node to class_ids::ALL.
  3. That build's lance-graph clone carried a codebook mirror one concept short, so lance-graph-ogar's COUNT_FUSE panicked at const-eval:
    error[E0080]: ogar_codebook mirror drifted from ogar_vocab::class_ids::ALL
    
    The deploy died at compile — it never reached hydration, which is where we'd been looking.
  4. 13:01 — lance-graph #953 fixed main.
  5. The redeploy reproduced the identical failure, because this layer served a lance-graph that no longer had the bug anywhere except in Docker's cache.

Roughly two hours of the outage was spent diagnosing hydration and volume capacity for a build that had never produced a binary.

What this changes

Comment only — no build behaviour. It records the trap, the diagnostic (a sibling that is demonstrably green on main fails the build → suspect this layer first), and the two escapes: push any q2 commit, or redeploy with the cache disabled.

It also names the durable fix without making it: either the siblings become explicit SHA ARGs bumped deliberately (reproducible, staleness visible in the diff), or the cross-repo invariant stops being a hand-maintained mirror at all — the hotplug-enumeration direction, which deletes this failure class rather than detecting it faster. Both are architectural calls; a comment is not one, and this PR does not pretend otherwise.

Why merging it unblocks the deploy

The commit modifies a file under COPY . /build/q2, so it busts that layer and every layer after it — including the sibling clone. The next deploy clones fresh against the already-green lance-graph main.

If the build still fails after this, the cache is exonerated and the next candidate is the volume-capacity risk recorded in claude-notes/plans/2026-08-13-brandenburg-bake.md (4.13 GB of artifacts, no free-space preflight in osm_slab_hydrate) — and the deploy logs will name it directly.


Generated by Claude Code

Summary by CodeRabbit

  • Documentation
    • Clarified how sibling dependency caches refresh across commits and redeployments.
    • Documented potential stale-cache failure scenarios.
    • Added guidance on cache-busting and dependency-pinning options.
    • No changes were made to Docker build behavior.

The comment above the sibling `git clone` layer asserted that "each build
re-clones fresh (no stale-cache problem the old pin was guarding against)".
That is true per q2 COMMIT and false per DEPLOY, and the difference is a
production outage.

Docker busts a layer when an INPUT changes. The sibling repos are not inputs —
nothing in this file can observe that lance-graph's HEAD moved — so a redeploy
of the same q2 commit reuses whatever clones the last build happened to take.

Measured today. Merging q2 #129 and OGAR quarto-dev#268 in the same minute started a
build whose lance-graph clone carried a codebook mirror one concept short of
OGAR's `class_ids::ALL`; `lance-graph-ogar`'s COUNT_FUSE panicked at
const-eval (E0080) and the deploy died at COMPILE, never reaching hydration.
lance-graph #953 fixed main eight minutes later — and the redeploy reproduced
the identical failure, because this layer served a lance-graph that no longer
had the bug anywhere but in Docker's cache.

Records the trap, the diagnostic ("a sibling that is demonstrably green on main
fails the build -> suspect this layer first"), and the two escapes: any q2
commit busts COPY and therefore this, or redeploy with the cache disabled.

Names the durable fix without making it: either explicit SHA ARGs bumped
deliberately (reproducible, staleness visible in the diff) or removing the
hand-maintained mirror entirely via hotplug enumeration. Both are architectural
calls; a comment is not one, and this commit does not pretend otherwise.

This commit also busts the COPY layer, so the next deploy clones fresh siblings
and should build against the already-green lance-graph main.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NMeiLmtDKhomJNSo2ecbJw
@cursor

cursor Bot commented Aug 14, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_d1d1f1d7-5619-433e-b0e1-8032fc1de972)

@coderabbitai

coderabbitai Bot commented Aug 14, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@AdaWorldAPI, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 115 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 622075e4-233d-4645-9ceb-f6dc47da27eb

📥 Commits

Reviewing files that changed from the base of the PR and between 220d47b and 3c76143.

📒 Files selected for processing (1)
  • Dockerfile

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 02583f7e-aeb9-40cb-95d8-beda658f76eb

📥 Commits

Reviewing files that changed from the base of the PR and between 29c3126 and 220d47b.

📒 Files selected for processing (1)
  • Dockerfile

📝 Walkthrough

Walkthrough

The Dockerfile documentation now explains sibling dependency cache reuse across redeploys, a resulting compile failure, diagnostic steps, and cache-busting or dependency-pinning options. Docker build behavior is unchanged.

Changes

Docker cache documentation

Layer / File(s) Summary
Cache behavior and troubleshooting
Dockerfile
The comments document cache reuse, stale dependencies, diagnostic steps, and possible durable fixes. Docker instructions remain unchanged.

Estimated code review effort: 1 (Trivial) | ~3 minutes

Merge Risk: ⚪ Minimal · up to 220d4

This change only documents Docker cache behavior and does not alter build or runtime behavior; no actionable merge-blocking risk remains after normal checks and review.

Suggested reviewers: claude

Poem

I nibble through notes where the cache leaves a trace,
Fresh clones may sleep in the same build space.
The Docker steps stay steady and clear,
While stale dependencies now appear.
Hop, hop—documented and bright!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the Docker cache claim correction and its outage impact, which matches the main change.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

…oves

Upgrades the previous commit from documenting the trap to closing it.

Four `ADD`s of each sibling's commits atom feed run before the clone. Docker
re-fetches an ADD-from-URL every build and hashes the RESULT, so the clone
layer now busts exactly when a sibling moves and stays cached when none did —
restoring "latest of everything" to being true per DEPLOY rather than per q2
COMMIT, which is the property the old comment claimed and did not have.

THE ATOM FEED, NOT github.com/ghapi, and that choice is load-bearing.
Unauthenticated github.com/ghapi allows 60 requests/hour PER IP and CI builders
share egress IPs, so the API form would eventually 403 — and a failed ADD FAILS
THE BUILD. That swaps a stale cache for a brand-new outage cause, which is the
opposite of the intent. The atom feed is served by the web frontend, is not
under that quota, and its entries carry commit ids, so its content changes when
and only when HEAD does.

Verified rather than assumed: the feed returns 200 unauthenticated, and its
first entry is bfc4f6db700b17943967a86f196cbd040f57a333 — byte-identical to
lance-graph's current origin/main. (It 403s from the dev sandbox, which is that
environment's egress policy, not GitHub's; the same URLs the `git clone` lines
already fetch without credentials.)

The failure direction is also the safe one. If the feed ever changed without
HEAD moving, the cost is extra fresh clones — slower, still correct. The API
form's failure mode was a red build.

Still NOT the deeper fix: explicit SHA ARGs bumped deliberately, or removing
the hand-maintained mirror via hotplug enumeration so the fuse's whole failure
class disappears. Both are architectural calls and stay the operator's.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NMeiLmtDKhomJNSo2ecbJw

Copy link
Copy Markdown
Owner Author

⚠ The review above predates the commit that changes build behaviour

CodeRabbit reviewed 29c3126…220d47b and concluded "Docker build behavior is unchanged" / Merge Risk: Minimal. That was accurate for 220d47b, which was comment-only.

3c76143 landed after the review started and does change build behaviour — it adds four ADD instructions that fetch over the network on every build:

ADD https://github.com/AdaWorldAPI/lance-graph/commits/main.atom /tmp/rev/lance-graph.atom
ADD https://github.com/AdaWorldAPI/ndarray/commits/main.atom     /tmp/rev/ndarray.atom
ADD https://github.com/AdaWorldAPI/OGAR/commits/main.atom        /tmp/rev/OGAR.atom
ADD https://github.com/AdaWorldAPI/openstreetmap-website-rs/commits/main.atom /tmp/rev/osm-website.atom

So please don't merge this on the strength of "documentation only" — that describes the first commit, not the PR.

What the added lines do

Docker re-fetches an ADD-from-URL on every build and hashes the result, so the sibling-clone layer below now busts exactly when a sibling's main moves, and stays cached when none did. That restores "latest of everything" to being true per deploy rather than per q2 commit — the property the old comment claimed and did not have, and whose absence caused today's outage.

The two things worth reviewing

  1. Why the Atom feed and not github.com/ghapi. Unauthenticated github.com/ghapi allows 60 requests/hour per IP, CI builders share egress IPs, and a failed ADD fails the build. The API form would have traded a stale cache for a new outage cause. The Atom feed is served by the web frontend and is not under that quota.
  2. The failure direction is the safe one. If a feed ever changed without HEAD moving, the cost is extra fresh clones — slower, still correct. Nothing about this can turn a good build red, which was the design constraint.

Verified rather than assumed: the feed returns 200 unauthenticated and its first entry is bfc4f6db700b17943967a86f196cbd040f57a333, matching lance-graph's origin/main exactly. (It 403s from my sandbox — that is the sandbox's egress policy, not GitHub's; the git clone lines below already fetch the same repos without credentials.)

Still deliberately not done

SHA-pinned siblings, or removing the hand-maintained codebook mirror entirely via hotplug enumeration — which would delete this failure class rather than detect it faster. Both are architectural calls and stay the operator's; this PR is the narrow fix.


Generated by Claude Code

@AdaWorldAPI
AdaWorldAPI merged commit 835f7ea into main Aug 14, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants