fix(orchestrator): enforce body-before-header upload ordering in runV3

[AdaAibaby]

Problem

In runV3, the memfile body and header were uploaded by separate racing goroutines:

• Goroutine 1 wrote memfile.header as soon as DiffHeader resolved
• Goroutine 3 uploaded the memfile body independently

PollRemoteStorageForHeader treats a header appearing in storage as the durability
signal for the entire layer. When goroutine 1 won the race, child builds could start
deduplicating against a body that had not yet landed — causing data corruption or
"object does not exist" errors. The same race existed for rootfs body/header.

Root cause analysis

The original approach (if h == nil { return error }) was dead code: as reviewer
@jakubno correctly pointed out, pauseProcessMemory cannot produce (nil, nil) for
a full memory snapshot — ToDiffHeader always returns a non-nil header or an error.

The actual connected issue is the body-before-header race described above. A secondary
contributing factor was a SetOnce.WaitWithContext non-determinism bug (fixed in #3241,
now included via rebase): when both the result channel and the cancelled context fired
simultaneously, select could randomly pick ctx.Err() instead of the already-set
result, causing intermittent upload failures that left memfile.header unwritten on disk.

Fix

1. Body-before-header ordering (build_upload_v3.go): Merge the memfile body and
   header goroutines into one sequence: resolve header → upload body → write header.
   Same ordering applied to rootfs.

2. Explicit error for nil header (build_upload_v3.go): A nil DiffHeader for a
   non-filesystem snapshot is now an explicit error instead of a silent no-op.

3. Diagnostic logging (build_upload_v3.go, sandbox.go): Info-level logs at
   header resolution and upload boundaries for production observability.

4. Rebased on main: Includes the SetOnce.WaitWithContext fix (fix(shared): make SetOnce.WaitWithContext deterministic once value is set #3241) and the
   provisional header system (feat(orch): decouple warm resume from memfile dedup #3166).

Test plan

• Build a template in local storage mode; verify memfile.header exists after build
• Filesystem-only snapshots still work (FilesystemSnapshot == true skips memfile)
• P2P/cloud mode: child builds no longer start before parent body is fully uploaded

Closes #3226
/cc @jakubno @dobrac @ValentaTomas @arkamar @tvi Looking forward to your code review.

[gemini-code-assist]

Code Review

This pull request adds validation checks in both build_upload_v3.go and build_upload_v4.go to return an error if a memfile diff header resolves to nil for non-filesystem-only snapshots, and updates a log level from Debug to Warn in storage.go for legacy headerless fallbacks. The reviewer noted a potential nil pointer dereference panic in other goroutines that wait on these headers and call u.layerSizeMetadata(h) without checking if h is nil, suggesting that similar nil checks should be added there.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[jakubno]

please fix lint issues

[AdaAibaby]

please fix lint issues

done.

[jakubno]

Checking the code it seems full-snapshot code cannot produce (nil, nil)
pauseProcessMemory resolves DiffHeader as either:
non-nil header, nil error
nil header, non-nil error

This doesn't fix the connected issue

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f557af2271

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b21997a995

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[AdaAibaby]

Checking the code it seems full-snapshot code cannot produce (nil, nil) pauseProcessMemory resolves DiffHeader as either: non-nil header, nil error nil header, non-nil error

This doesn't fix the connected issue

Agreed — (nil, nil) should be analytically unreachable for full-memory snapshots,
so the original if h == nil { return nil } was dead code on that path.

Updated in 8a4cf57: the nil case now returns an explicit error for non-filesystem
snapshots instead of silently succeeding, so if this ever does trigger the build
fails loudly with the build_id rather than leaving a header-less template.

Also added Info-level logs at both ends of the resolution chain
(pauseProcessMemory's setHeader call and runV3's WaitWithContext return) to make
the actual runtime values observable on the next reproduction — that should tell
us whether metaOut never resolved, ToDiffHeader returned unexpectedly, or
something else entirely.