fix(orchestrator): recover lost ancestor entries when persisting headers

[ValentaTomas]

Two complementary fixes for headers whose mapping references builds their Builds map lost. #3447 stopped load-time fabrication of zero entries for such gaps; this PR closes the two paths it left open.

Write path: stop persisting inherited gaps

A pause carries its source header's Builds map forward, so an entry lost once (e.g. the live header was peer-served/incomplete when an earlier pause persisted it) propagates to every descendant header, and each descendant diff pays a proactive header refresh per gap forever. Now, when appendAncestorBuilds resolves nothing locally and the entry is still missing, it recovers the entry from the referenced build's own stored header before the diff header is persisted — gaps heal on the next pause instead of being inherited, at the cost of one header GET per still-missing entry (normally zero). Builds with no stored header (legacy uncompressed) stay absent and keep resolving on the read path. The heal lives in appendAncestorBuilds rather than StoreHeader because that's where the ancestor barrier, storage handle, and file type already are.

Read path: recover stale zero entries

Before #3447, the fabricated zero entries were also serialized whenever a pause ran on a header loaded from storage — baked into descendant headers as real entries, on the wire indistinguishable from the legit V3 uncompressed sentinel. For a compressed ancestor the claimed suffix-less object does not exist, and createDiff failed the size lookup permanently; #3447 only helps when the entry is absent. Now a zero entry whose basic-name object is missing falls back to the build's own header (same recovery as the no-entry branch, telemetry cause zero_entry_miss). Legit V3 sentinels are unaffected — their object exists, so no extra roundtrip.

Stored zero entries are re-persisted by later pauses (indistinguishable from V3 sentinels without a per-pause probe), but reads now always recover; rewriting them is fleet hygiene, not orchestrator logic.

orchestrator.storage.diff.frame_table_refresh (cause=proactive) should trend down as affected lineages pause past the write-path fix; cause=zero_entry_miss counts the baked-zero recoveries.

Tests

• gap recovered from the build's stored header at upload; missing header file leaves the gap absent without failing the upload; present entries cause no storage round-trip
• stale zero entry: size-lookup 404 falls back to the build's own header and serves the compressed object; the healthy-sentinel latch test still passes unchanged
• negative controls: reverting either production change fails the corresponding test

[cursor]

PR Summary

Medium Risk
Changes sandbox header persistence and diff open logic on the pause and fault-read paths; behavior is guarded by tests and degrades to the prior gap/absent behavior when recovery fails.

Overview
Headers that reference ancestor builds without matching Builds entries used to keep that gap on every pause and could strand reads behind stale zero sentinels. On pause, when an ancestor isn’t available from the upload wait path and the outgoing header still lacks that build, the upload path now loads the ancestor’s stored header to fill Builds before persistence (best-effort on transient errors; legacy builds with no header stay absent). On read, when a zero sentinel points at a missing basic object or a peer transition during size lookup, createDiff refreshes from the build’s own header and opens the real compressed data instead of failing permanently.

^{Reviewed by Cursor Bugbot for commit a198284. Bugbot is set up for automated code reviews on this repo. Configure here.}

[codecov]

Codecov Report

❌ Patch coverage is 85.71429% with 7 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...ges/orchestrator/pkg/sandbox/build/storage_diff.go	75.86%	6 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Zero-entry recovery skips peer transition
  • Added PeerTransitionedError handling to the zero-entry recovery logic so it recovers via refreshHeader like the no-entry branch does.

Or push these changes by commenting:

@cursor push d39bdce632

Preview (d39bdce632)

diff --git a/packages/orchestrator/pkg/sandbox/build/storage_diff.go b/packages/orchestrator/pkg/sandbox/build/storage_diff.go
--- a/packages/orchestrator/pkg/sandbox/build/storage_diff.go
+++ b/packages/orchestrator/pkg/sandbox/build/storage_diff.go
@@ -209,7 +209,15 @@
 
 	if size == 0 {
 		size, err = upstream.Size(ctx)
-		if hasEntry && initialFT == storage.UncompressedFullFrameTable && errors.Is(err, storage.ErrObjectNotExist) {
+		var shouldRecover bool
+		if hasEntry && initialFT == storage.UncompressedFullFrameTable {
+			shouldRecover = errors.Is(err, storage.ErrObjectNotExist)
+			if !shouldRecover {
+				var transErr *storage.PeerTransitionedError
+				shouldRecover = errors.As(err, &transErr)
+			}
+		}
+		if shouldRecover {
 			// The zero entry claimed an uncompressed V3-era ancestor, but the
 			// suffix-less object does not exist: the entry is a gap persisted
 			// as zero by older releases. Resolve the build's own header like

_{You can send follow-ups to the cloud agent here.}

^{Reviewed by Cursor Bugbot for commit f26f8a6. Configure here.}