fix(orchestrator): atomically replace metadata

[jakubno]

Prevent prefetch updates from exposing truncated metadata to concurrent readers.

[linear-code]

EN-1879

[cursor]

PR Summary

Medium Risk
Changes how local snapshot metadata is written while uploads and harvest run concurrently; behavior is more correct but errors and remote follow-up now depend on the new committed-vs-failed distinction.

Overview
Prefetch and other metadata updates no longer truncate the on-disk file in place. Updates go through a temp file and rename, and template cache writes call that path instead of a plain overwrite. If the new bytes are visible but syncing the parent directory fails, the error is surfaced as committed-but-not-fully-durable; the pause prefetch harvest treats that as a successful local update and can still push enriched metadata remotely.

^{Reviewed by Cursor Bugbot for commit 6e37fb9. Bugbot is set up for automated code reviews on this repo. Configure here.}

[codecov]

❌ 5 Tests Failed:

Tests completed	Failed	Passed	Skipped
3439	5	3434	9

View the top 3 failed test(s) by shortest run time

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_1_lists_root_directory

Stack Traces | 0.01s run time

=== RUN   TestListDir/depth_1_lists_root_directory
=== PAUSE TestListDir/depth_1_lists_root_directory
=== CONT  TestListDir/depth_1_lists_root_directory
    filesystem_test.go:96: 
        	Error Trace:	.../tests/envd/filesystem_test.go:96
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_1_lists_root_directory
--- FAIL: TestListDir/depth_1_lists_root_directory (0.01s)

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)

Stack Traces | 0.01s run time

=== RUN   TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
=== PAUSE TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
=== CONT  TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
    filesystem_test.go:96: 
        	Error Trace:	.../tests/envd/filesystem_test.go:96
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
--- FAIL: TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory) (0.01s)

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_0_lists_only_root_directory

Stack Traces | 0.02s run time

=== RUN   TestListDir/depth_0_lists_only_root_directory
=== PAUSE TestListDir/depth_0_lists_only_root_directory
=== CONT  TestListDir/depth_0_lists_only_root_directory
    filesystem_test.go:96: 
        	Error Trace:	.../tests/envd/filesystem_test.go:96
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_0_lists_only_root_directory
--- FAIL: TestListDir/depth_0_lists_only_root_directory (0.02s)

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir

Stack Traces | 0.88s run time

=== RUN   TestListDir
=== PAUSE TestListDir
=== CONT  TestListDir
--- FAIL: TestListDir (0.88s)

View the full list of 1 ❄️ flaky test(s)

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_3_lists_all_directories_and_files

Flake rate in main: 30.77% (Passed 9 times, Failed 4 times)

Stack Traces | 0.01s run time

=== RUN   TestListDir/depth_3_lists_all_directories_and_files
=== PAUSE TestListDir/depth_3_lists_all_directories_and_files
=== CONT  TestListDir/depth_3_lists_all_directories_and_files
    filesystem_test.go:96: 
        	Error Trace:	.../tests/envd/filesystem_test.go:96
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_3_lists_all_directories_and_files
--- FAIL: TestListDir/depth_3_lists_all_directories_and_files (0.01s)

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

[gemini-code-assist]

Code Review

This pull request introduces atomic file replacement capabilities to ensure metadata updates are written safely without exposing partial contents. It adds WriteToFileFromReaderAtomically in ioutils which writes to a temporary file and syncs the parent directory, and integrates this into the template metadata handling via a new ReplaceFile method. Additionally, the prefetch harvester is updated to handle cases where the local atomic write is committed but directory durability cannot be fully confirmed (ErrReplaceCommitted), ensuring remote metadata updates still proceed. There are no review comments, so I have no feedback to provide.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[claude]

No bugs found, but I'd still like a human to look at this before merging — it changes on-disk write semantics for the orchestrator's template metadata file (a new atomic-write-via-rename primitive) and adjusts the prefetch harvester's error-handling control flow around a partial-durability signal (ErrReplaceCommitted), both on a production-critical sandbox pause/resume path.

Extended reasoning...

Overview

The PR replaces plain truncate-and-write metadata persistence with an atomic replace (temp file + fsync + rename + parent-directory fsync) in packages/shared/pkg/ioutils/write_to_file.go, wires it up via a new Template.ReplaceFile in template_metadata.go, and switches storageTemplate.UpdateMetadata to use it. The prefetch harvester (prefetch_harvest.go) is updated to distinguish a genuine local-write failure from the new "replace committed, but parent-directory durability unconfirmed" case (ErrReplaceCommitted), continuing on to the remote metadata re-upload in the latter case while still surfacing the error for logging.

Security risks

None identified — this is local filesystem I/O with no external input, no auth/permission changes, and no new attack surface.

Level of scrutiny

This warrants a closer look than a typical mechanical PR: it touches the orchestrator's sandbox pause/resume metadata path (production-critical), introduces a new atomic-write primitive with several interacting cleanup/rollback branches (temp-file removal, close-then-rename ordering, directory fsync), and changes control flow in the harvester based on error identity (errors.Is against a wrapped sentinel). I traced through the atomic-write function's error/defer paths (Copy failure, Chmod failure, Sync failure, post-close failure, post-rename directory-sync failure) and the harvester's branching on ErrReplaceCommitted and did not find a defect, but the number of edge cases and the fact that it changes durability semantics on a path that affects sandbox resume correctness makes this a good candidate for a second pair of eyes rather than an auto-approval.

Other factors

Test coverage is solid — new tests cover the atomic-write happy path (including that partial writes aren't visible to concurrent readers), read-failure cleanup, and missing-file/non-regular-file preconditions, plus a new harvester test for the ErrReplaceCommitted pass-through. No unresolved reviewer comments are present in the timeline yet (only bot summaries).