fix(orchestrator): order envd.service after systemd-tmpfiles-setup

[dobrac]

What

Add systemd-tmpfiles-setup.service to envd.service's After= ordering.

Why

Template builds hot-update envd via updateEnvdInSandbox, which stages the new binary at /tmp/envd_updated, then chmod/mvs it into place. In production this fails with:

error building step N: update envd: failed to replace envd binary: exit status 1
[update-envd-replace] [stderr]: chmod: cannot access '/tmp/envd_updated': No such file or directory
[update-envd-replace] [stderr]: mv: cannot stat '/tmp/envd_updated': No such file or directory

(and earlier as error writing file metadata: xattr.list /tmp/envd_updated: no such file or directory)

Root cause

On our Ubuntu/Debian base images /tmp is not a tmpfs mount — it's a plain 1777 directory on the rootfs (tmp.mount ships only as a disabled template in /usr/share/systemd/, and fstab is unconfigured). At boot, systemd-tmpfiles-setup.service runs systemd-tmpfiles --create --remove --boot, and /usr/lib/tmpfiles.d/tmp.conf contains D /tmp 1777 root root 30d — the D type with --remove wipes /tmp's contents at boot.

#3020 made envd start very early (DefaultDependencies=no), and #3043 ordered it only After=local-fs.target. But systemd-tmpfiles-setup.service is also After=local-fs.target, so the two were unordered relative to each other. When envd answered the build's upload before the wipe ran, the staged file was deleted underneath it.

Ordering envd after systemd-tmpfiles-setup.service ensures the boot-time /tmp wipe has completed before envd answers requests, closing the race. The early-start cold-boot win from #3020 is preserved (systemd-tmpfiles-setup.service runs Before=sysinit.target, so the added wait is minimal).

Deterministic reproducer

docker run --rm ubuntu:24.04 bash -c '
  export DEBIAN_FRONTEND=noninteractive
  apt-get -qq update >/dev/null 2>&1
  apt-get -qq install -y --no-install-recommends systemd >/dev/null 2>&1
  echo data > /tmp/envd_updated
  systemd-tmpfiles --create --remove --boot --exclude-prefix=/dev
  chmod +x /tmp/envd_updated   # -> No such file or directory
  mv -f /tmp/envd_updated /usr/bin/envd
'

Notes

• Changing an embedded rootfs file shifts rootfs.FilesHash(), so base layers rebuild and the fix propagates to newly built templates.
• local-fs.target is kept in After= (still relevant if a base image ever enables a real tmp.mount, which is ordered before local-fs.target).

[cursor]

PR Summary

Medium Risk
Changes sandbox boot ordering for envd, which affects template builds and early-boot behavior, but the scope is a single systemd unit and a targeted race fix.

Overview
Fixes a boot race where template builds could fail during envd hot-update because the staged /tmp/envd_updated binary was removed before chmod/mv.

The embedded envd.service unit now starts after systemd-tmpfiles-setup.service, not only after local-fs.target. On Ubuntu/Debian base images, tmpfiles’ D /tmp wipe runs at boot and was unordered vs early-start envd, so uploads could land in /tmp and then disappear. Comments in the template document that behavior.

rootfs_test.go adds a regression check that generated envd.service includes systemd-tmpfiles-setup.service in its After= line. Changing the template shifts rootfs.FilesHash(), so new base layers pick up the fix.

^{Reviewed by Cursor Bugbot for commit 824d528. Bugbot is set up for automated code reviews on this repo. Configure here.}

[gemini-code-assist]

Code Review

This pull request updates the envd systemd service configuration to order it after systemd-tmpfiles-setup.service to prevent a race condition with /tmp cleanup. There is no feedback to provide as no issues were identified.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[codecov]

❌ 6 Tests Failed:

Tests completed	Failed	Passed	Skipped
3091	6	3085	8

View the top 3 failed test(s) by shortest run time

github.com/e2b-dev/infra/tests/integration/internal/tests/orchestrator::TestSandboxFilesystemPauseResumeIntegrity

Stack Traces | 0s run time

=== RUN   TestSandboxFilesystemPauseResumeIntegrity
=== PAUSE TestSandboxFilesystemPauseResumeIntegrity
=== CONT  TestSandboxFilesystemPauseResumeIntegrity
--- FAIL: TestSandboxFilesystemPauseResumeIntegrity (0.00s)
Executing command bash in sandbox i5wfz6qr8u3aycvype101 (user: root)

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_1_lists_root_directory

Stack Traces | 0.01s run time

=== RUN   TestListDir/depth_1_lists_root_directory
=== PAUSE TestListDir/depth_1_lists_root_directory
=== CONT  TestListDir/depth_1_lists_root_directory
    filesystem_test.go:96: 
        	Error Trace:	.../tests/envd/filesystem_test.go:96
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_1_lists_root_directory
--- FAIL: TestListDir/depth_1_lists_root_directory (0.01s)

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)

Stack Traces | 0.01s run time

=== RUN   TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
=== PAUSE TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
=== CONT  TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
    filesystem_test.go:96: 
        	Error Trace:	.../tests/envd/filesystem_test.go:96
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
--- FAIL: TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory) (0.01s)

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_3_lists_all_directories_and_files

Stack Traces | 0.01s run time

=== RUN   TestListDir/depth_3_lists_all_directories_and_files
=== PAUSE TestListDir/depth_3_lists_all_directories_and_files
=== CONT  TestListDir/depth_3_lists_all_directories_and_files
    filesystem_test.go:96: 
        	Error Trace:	.../tests/envd/filesystem_test.go:96
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_3_lists_all_directories_and_files
--- FAIL: TestListDir/depth_3_lists_all_directories_and_files (0.01s)

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_0_lists_only_root_directory

Stack Traces | 0.02s run time

=== RUN   TestListDir/depth_0_lists_only_root_directory
=== PAUSE TestListDir/depth_0_lists_only_root_directory
=== CONT  TestListDir/depth_0_lists_only_root_directory
    filesystem_test.go:96: 
        	Error Trace:	.../tests/envd/filesystem_test.go:96
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_0_lists_only_root_directory
--- FAIL: TestListDir/depth_0_lists_only_root_directory (0.02s)

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir

Stack Traces | 0.85s run time

=== RUN   TestListDir
=== PAUSE TestListDir
=== CONT  TestListDir
--- FAIL: TestListDir (0.85s)

github.com/e2b-dev/infra/tests/integration/internal/tests/proxies::TestMaskRequestHostAPIParameter

Stack Traces | 5.15s run time

=== RUN   TestMaskRequestHostAPIParameter
=== PAUSE TestMaskRequestHostAPIParameter
=== CONT  TestMaskRequestHostAPIParameter
    mask_request_host_test.go:44: Command [python3] output: event:{start:{pid:1107}}
Executing command ls in sandbox iy0n9nwvmh339ya3713ig
    mask_request_host_test.go:68: Command [cat] output: event:{start:{pid:1109}}
    mask_request_host_test.go:68: Command [cat] output: event:{data:{stderr:"cat: /tmp/nc_output.txt: No such file or directory\n"}}
    mask_request_host_test.go:68: Command [cat] output: event:{end:{exit_code:1 exited:true status:"exit status 1" error:"exit status 1"}}
    mask_request_host_test.go:69: 
        	Error Trace:	.../tests/proxies/mask_request_host_test.go:69
        	Error:      	Received unexpected error:
        	            	command cat in sandbox insm2ql0tf5iwpcgvumup failed with exit code 1
        	Test:       	TestMaskRequestHostAPIParameter
--- FAIL: TestMaskRequestHostAPIParameter (5.15s)

github.com/e2b-dev/infra/tests/integration/internal/tests/orchestrator::TestSandboxFilesystemPauseResumeIntegrity/zeroed_ranges_and_truncate_survive_pause

Stack Traces | 9.55s run time

=== RUN   TestSandboxFilesystemPauseResumeIntegrity/zeroed_ranges_and_truncate_survive_pause
=== PAUSE TestSandboxFilesystemPauseResumeIntegrity/zeroed_ranges_and_truncate_survive_pause
=== CONT  TestSandboxFilesystemPauseResumeIntegrity/zeroed_ranges_and_truncate_survive_pause
    filesystem_pause_resume_integrity_test.go:55: Command [bash] output: event:{start:{pid:1116}}
Executing command bash in sandbox iyrjw2jw2f5d70mn3p0q2 (user: root)
    filesystem_pause_resume_integrity_test.go:55: Command [bash] output: event:{end:{exited:true status:"exit status 0"}}
    filesystem_pause_resume_integrity_test.go:55: Command [bash] completed successfully in sandbox i8w5nznaogtd7mb2gk4n6
Executing command bash in sandbox i8w5nznaogtd7mb2gk4n6 (user: root)
    filesystem_pause_resume_integrity_test.go:55: Command [bash] output: event:{start:{pid:1121}}
    filesystem_pause_resume_integrity_test.go:55: Command [bash] output: event:{data:{stdout:"a1032207670ae79148e0d88a8da11beb70f200ac23dbcb9a443e2e1d71a06e8e\n"}}
    filesystem_pause_resume_integrity_test.go:55: Command [bash] output: event:{end:{exited:true status:"exit status 0"}}
    filesystem_pause_resume_integrity_test.go:55: Command [bash] completed successfully in sandbox i8w5nznaogtd7mb2gk4n6
Executing command bash in sandbox il2ne02eokf285fihuvh9 (user: root)
    filesystem_pause_resume_integrity_test.go:55: Command [bash] output: event:{start:{pid:1127}}
    filesystem_pause_resume_integrity_test.go:55: Command [bash] output: event:{data:{stdout:"12582912\n"}}
    filesystem_pause_resume_integrity_test.go:55: Command [bash] output: event:{end:{exited:true status:"exit status 0"}}
    filesystem_pause_resume_integrity_test.go:55: Command [bash] completed successfully in sandbox i8w5nznaogtd7mb2gk4n6
    filesystem_pause_resume_integrity_test.go:55: Command [bash] output: event:{start:{pid:1131}}
    filesystem_pause_resume_integrity_test.go:55: Command [bash] output: event:{data:{stdout:"a1032207670ae79148e0d88a8da11beb70f200ac23dbcb9a443e2e1d71a06e8e\n"}}
    filesystem_pause_resume_integrity_test.go:55: Command [bash] output: event:{end:{exited:true status:"exit status 0"}}
    filesystem_pause_resume_integrity_test.go:55: Command [bash] completed successfully in sandbox i8w5nznaogtd7mb2gk4n6
Executing command bash in sandbox i8w5nznaogtd7mb2gk4n6 (user: root)
    filesystem_pause_resume_integrity_test.go:55: Command [bash] output: event:{start:{pid:1137}}
    filesystem_pause_resume_integrity_test.go:55: 
        	Error Trace:	.../tests/orchestrator/filesystem_pause_resume_integrity_test.go:155
        	            				.../tests/orchestrator/filesystem_pause_resume_integrity_test.go:202
        	            				.../tests/orchestrator/filesystem_pause_resume_integrity_test.go:55
        	Error:      	Received unexpected error:
        	            	failed to execute command bash in sandbox i8w5nznaogtd7mb2gk4n6: invalid_argument: protocol error: incomplete envelope: unexpected EOF
        	Test:       	TestSandboxFilesystemPauseResumeIntegrity/zeroed_ranges_and_truncate_survive_pause
--- FAIL: TestSandboxFilesystemPauseResumeIntegrity/zeroed_ranges_and_truncate_survive_pause (9.55s)

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}