fix(orchestrator): pause upload retain retry

[jakubno]

Problem

A paused sandbox's snapshot could be partially uploaded and then permanently lost. The pause upload was fire-and-forget — a single attempt whose failure was only logged — so a transient source-side failure dropped the snapshot for good. The lost artifact is almost always the memfile (memory image) plus its header. Because every later build stores only a diff against its ancestors, a lost ancestor memfile is unrecoverable and cascades: descendant pauses can never finalize, and any resume crashes at runtime when it page-faults on the missing memory.

Where it was observed

Loki (grafanacloud-logs):

{service_name="orchestrator"} |= "error uploading snapshot files"

→ concentrated on a "hot host" per day

Concrete case — build 2210682a-30e2-4567-9091-53d57a625a1a (host orch-client-n2-2pkh, ~2026-05-29 18:13 UTC):

• Pause at 18:13:37; at 18:13:47.923:
  error uploading snapshot files → ... wait for ancestor d7b75d71…: memfile upload: context deadline exceeded
• The build's objects in storage were only metadata.json, snapfile, rootfs.ext4.zstd — memfile.zstd, memfile.header, rootfs.ext4.header were missing.
• Downstream, same sandbox at 18:15:58: UFFD serve data fetch error after retries → failed to get GCS object ("b52e4ea0-…/memfile") … object does not exist → sandbox killed.

Why it happened

1. Transient source-side failure — a memfile upload hit the 20‑min uploadTimeout, or GCS auth blipped (401/503 Metadata Server not available), or MemfileDiff.CachePath wasn't ready.
2. Fire-and-forget meant one shot — uploadSnapshotAsync ran Upload.Run once and only logged on failure; the local diff was then evicted, so the bytes were gone.
3. Failure propagated and cascaded — the failed build's future was stored with the error and broadcast (Redis); every descendant pause's appendAncestorBuilds → Uploads.Wait inherited it instantly, producing the deep wait for ancestor … chains. One sandbox auto-pausing every ~5 min re-failed identically 17×.
4. The loss surfaced at resume, not pause — pause returned success (upload is async); memory is paged lazily via UFFD from GCS, so the first fault on the missing …/memfile object killed the running sandbox.

What this PR does

• Retry until durable — replaces the single attempt with a bounded retry: fresh per-attempt timeout under a ~2h budget, exponential backoff, default-retryable error classification (only NoDiff / object-not-exist / parent-cancel stop). Recovers the transient GCS/auth/timeout roots, and because every pause retries, descendants' ancestor-waits resolve once the root finally lands — the cascade self-heals.
• General retry.Do helper (packages/shared/pkg/retry) — the loop runs under one budget context.WithTimeoutCause, so each attempt derives from it and is auto-capped to the remaining budget; the loop can't overrun TotalBudget. Distinguishes budget exhaustion from caller cancellation via context.Cause.
• Cross-orchestrator budget alignment — refreshHeaderBudget and futureTTL raised to ≥ the upload budget, so a dependent build's Wait doesn't give up (non-retryable object does not exist on poll-budget expiry) while the parent is still retrying.
• Success-only finalization — uploadedBuilds is marked only on actual success, so a failed build is never advertised as uploaded to peers; new orchestrator.snapshot.upload.failed counter surfaces give-ups.
• Graceful shutdown drains uploads — instead of cancelling on shutdown, Server.Close(ctx) waits for in-flight uploads (bounded by the close context, so a forced stop still exits fast), with periodic drain-progress logging.

[codecov]

Codecov Report

❌ Patch coverage is 41.91176% with 79 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
packages/orchestrator/pkg/server/main.go	0.00%	41 Missing ⚠️
packages/orchestrator/pkg/server/sandboxes.go	0.00%	25 Missing ⚠️
packages/orchestrator/pkg/server/upload_retry.go	60.00%	8 Missing ⚠️
packages/shared/pkg/retry/retry.go	93.75%	2 Missing and 1 partial ⚠️
packages/orchestrator/pkg/factories/run.go	0.00%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[gemini-code-assist]

Code Review

The background upload retry loop in uploadSnapshotAsync is detached from the request context using context.WithoutCancel(ctx), which prevents it from being cancelled when the request finishes. However, because it is completely detached, the retry loop will not abort when the server shuts down, causing the background goroutine to continue retrying for up to two hours. To prevent resource leaks during shutdown, the detached context should be linked to the server's shutdown channel s.done so that it is cancelled immediately when the server stops.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[cursor]

PR Summary

Medium Risk
Changes core snapshot durability and shutdown timing; mis-tuned waits or the unretryable sync checkpoint path can still lose snapshots or prolong node drains during storage outages.

Overview
Pause snapshot uploads now retry for up to two hours with per-attempt timeouts and exponential backoff via a shared retry.Do helper, and graceful shutdown waits on in-flight uploads instead of cancelling them on the old single 20-minute timeout. Upload coordination TTLs (futureTTL, refreshHeaderBudget, Redis peer keys) are stretched to match that window so child builds and cross-orchestrator waits do not give up while a parent is still retrying; peer “already uploaded” is only set after a successful land, with a metric when uploads fail permanently.

Sync checkpoint (when async P2P checkpoint is off) still runs one upload.Run under uploadTimeout only, so it does not get the new retry behavior. Graceful drain can block shutdown for as long as uploads keep retrying (up to the total budget) unless force-stop cancels the close context.

^{Reviewed by Cursor Bugbot for commit 186614e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 1c8dbc6. Configure here.}