fix(orchestrator): drop stale pre-init logs by rguliyev · Pull Request #3297 · e2b-dev/runtime

rguliyev · 2026-07-16T04:17:31Z

A sandbox resumed from a memory snapshot boots with its clock still set to the snapshot's time. envd's log exporter is restored along with the rest of the VM, so it can send startup logs stamped with that old time before /init corrects the clock.

Each sandbox now records a fresh host timestamp when its lifecycle is created, before it's reachable. The /logs handler rejects any envd log timestamped more than a minute before that anchor. envd's exporter doesn't retry on a rejected request, so those stale records are simply dropped instead of being forwarded to the collector. Logs with a reasonable timestamp go through unchanged.

gemini-code-assist

Code Review

This pull request introduces a mechanism to correct stale log timestamps emitted by a sandbox guest before its clock is synchronized upon resume or creation. It adds a LifecycleStartedAt field to the Sandbox struct, initialized during sandbox creation and resumption, and uses it to clamp stale log timestamps to the host's start time while preserving the original timestamp. The review feedback suggests ensuring timezone consistency by explicitly converting and storing these timestamps in UTC, preventing issues with local timezone offsets in downstream log aggregation systems.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

gemini-code-assist · 2026-07-16T04:18:36Z

+		LifecycleID:        lifecycleID,
+		LifecycleStartedAt: time.Now(),


It is safer to initialize LifecycleStartedAt using time.Now().UTC() to ensure that the anchor timestamp is always stored in UTC, avoiding any local timezone dependencies or discrepancies when comparing or formatting it later.

Suggested change

LifecycleID: lifecycleID,

LifecycleStartedAt: time.Now(),

LifecycleID: lifecycleID,

LifecycleStartedAt: time.Now().UTC(),

Good catch, fixed - forced UTC on both the write side (time.Now().UTC() in CreateSandbox/ResumeSandbox) and the format call in correctStaleTimestamp.

codecov · 2026-07-16T05:11:18Z

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
3346	1	3345	7

View the top 3 failed test(s) by shortest run time

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_0_lists_only_root_directory

Stack Traces | 0.01s run time

=== RUN   TestListDir/depth_0_lists_only_root_directory
=== PAUSE TestListDir/depth_0_lists_only_root_directory
=== CONT  TestListDir/depth_0_lists_only_root_directory
    filesystem_test.go:96: 
        	Error Trace:	.../tests/envd/filesystem_test.go:96
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_0_lists_only_root_directory
--- FAIL: TestListDir/depth_0_lists_only_root_directory (0.01s)

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)

Stack Traces | 0.01s run time

=== RUN   TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
=== PAUSE TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
=== CONT  TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
    filesystem_test.go:96: 
        	Error Trace:	.../tests/envd/filesystem_test.go:96
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
--- FAIL: TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory) (0.01s)

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_1_lists_root_directory

Stack Traces | 0.02s run time

=== RUN   TestListDir/depth_1_lists_root_directory
=== PAUSE TestListDir/depth_1_lists_root_directory
=== CONT  TestListDir/depth_1_lists_root_directory
    filesystem_test.go:96: 
        	Error Trace:	.../tests/envd/filesystem_test.go:96
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_1_lists_root_directory
--- FAIL: TestListDir/depth_1_lists_root_directory (0.02s)

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir

Stack Traces | 1.33s run time

=== RUN   TestListDir
=== PAUSE TestListDir
=== CONT  TestListDir
--- FAIL: TestListDir (1.33s)
Executing command cat in sandbox ip6yqx9gthbyizovx0dyb (user: root)

github.com/e2b-dev/infra/tests/integration/internal/tests/api/templates::TestUpdateTemplateWithInvalidAPIKey

Stack Traces | 168s run time

=== RUN   TestUpdateTemplateWithInvalidAPIKey
=== PAUSE TestUpdateTemplateWithInvalidAPIKey
=== CONT  TestUpdateTemplateWithInvalidAPIKey
    template_update_test.go:107: Build failed: {<nil> An internal error occurred. Please try again or contact support with the build ID. <nil>}
--- FAIL: TestUpdateTemplateWithInvalidAPIKey (168.27s)

View the full list of 1 ❄️ flaky test(s)

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_3_lists_all_directories_and_files

Flake rate in main: 30.77% (Passed 9 times, Failed 4 times)

Stack Traces | 0.01s run time

=== RUN   TestListDir/depth_3_lists_all_directories_and_files
=== PAUSE TestListDir/depth_3_lists_all_directories_and_files
=== CONT  TestListDir/depth_3_lists_all_directories_and_files
    filesystem_test.go:96: 
        	Error Trace:	.../tests/envd/filesystem_test.go:96
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_3_lists_all_directories_and_files
--- FAIL: TestListDir/depth_3_lists_all_directories_and_files (0.01s)

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9cdcd7489b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

matthewlouisbrockman

good enough for now - can we yeet some of the excessive verboseness from the comments please?

one improvement might want to consider later: comparison's happening forever. not sure if there's a way to only do these checks while we're resuming the sandbox? also i think we're dropping logs from before we fix clock while resuming but tradeoff seems fine.

A sandbox resumed from a memory snapshot boots with its clock still set to the snapshot's time, so envd's log exporter (also restored from the snapshot) can emit startup logs stamped with that old time before /init corrects the clock. Record a fresh per-lifecycle timestamp on the host before the sandbox is reachable, and reject any log timestamped more than a minute before it. envd's exporter doesn't retry on rejection, so these stale records are simply dropped instead of forwarded.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9ceb20d55b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-07-16T23:49:17Z

 		return
 	}

+	if hasStaleLogTimestamp(payload, sbx.LifecycleStartedAt) {


Preserve queued pre-pause logs

When the same sandbox is paused with envd log entries still buffered in HTTPExporter.logs, that queue is captured in the memory snapshot and is sent after resume with its original pre-pause timestamp. If the pause lasted more than a minute, this new check returns 400 for those valid same-sandbox entries, and the exporter drops non-2xx sends, so logs produced immediately before pause disappear; the filter needs to distinguish restored pre-init records from buffered pre-pause records.

Useful? React with 👍 / 👎.

🤖 I have created a release *beep* *boop* --- ## 0.0.1 (2026-07-30) ### Features * **api:** add sandbox IAM workload token configuration ([13ddb3d](13ddb3d)) * **api:** add sandbox workload identity permission ([#3319](#3319)) ([13ddb3d](13ddb3d)) * **api:** SOCKS5 egress proxy on sandbox network config (BYOP) ([#2642](#2642)) ([1fc3820](1fc3820)) * **cfg:** add DISABLE_STARTUP_RECLAIM flag ([#3081](#3081)) ([7677ca6](7677ca6)) * **clickhouse:** implement multi-cluster fan-out for events and stats ([#2925](#2925)) ([39594c6](39594c6)) * dynamic sandbox log routing and ClickHouse-backed log reads ([#3236](#3236)) ([1b19a3b](1b19a3b)) * **envd:** give envd realtime IO priority, reset for user processes ([#2681](#2681)) ([f4bd1b2](f4bd1b2)) * **envd:** split collapse stats into real migrations vs already-huge ([#3021](#3021)) ([0d77614](0d77614)) * **envd:** support user-defined file metadata via xattrs ([#2732](#2732)) ([da8fbe4](da8fbe4)) * **featureflags:** support per-service context providers ([#3100](#3100)) ([65297c1](65297c1)) * freeze user cgroup across pause/resume to keep envd /init responsive ([#2688](#2688)) ([eceb741](eceb741)) * **metrics:** break down pause-snapshot latency by step ([#3426](#3426)) ([657559e](657559e)) * **metrics:** label pause telemetry by fs_only ([#3425](#3425)) ([411b63e](411b63e)) * **observability:** add kill_reason to sandbox.lifecycle.killed ([#2833](#2833)) ([e45418f](e45418f)) * **observability:** include kill_reason in kill-path structured logs ([#2846](#2846)) ([33c49f7](33c49f7)) * **orch:** add envd-version to LaunchDarkly sandbox context ([#3051](#3051)) ([37d3b92](37d3b92)) * **orch:** add less, nftables, iputils-ping, and jq to base provisioning ([#2736](#2736)) ([a1e010e](a1e010e)) * **orch:** collapse envd's heap into 2 MiB hugepages before pause to cut cold-resume faults ([#2997](#2997)) ([6677f73](6677f73)) * **orch:** debug a sandbox guest kernel with resume-build -gdb ([#3040](#3040)) ([37bb0dc](37bb0dc)) * **orch:** decouple warm resume from memfile dedup ([#3166](#3166)) ([77f25a0](77f25a0)) * **orch:** distro-aware template base-image provisioning ([#3411](#3411)) ([f8c7b5b](f8c7b5b)) * **orchestrator/cgroup:** list and destroy leaked sandbox cgroups ([#3086](#3086)) ([bce1d84](bce1d84)) * **orchestrator/nbd:** inspect and disconnect connected devices ([#3087](#3087)) ([4d47148](4d47148)) * **orchestrator/network:** list slot namespaces ([#3089](#3089)) ([c23dbc7](c23dbc7)) * **orchestrator/network:** list slot namespaces ([#3090](#3090)) ([fbfce25](fbfce25)) * **orchestrator:** add -force-reboot to resume-build to cold-boot memory-snaphsot builds ([#3208](#3208)) ([cf8f15b](cf8f15b)) * **orchestrator:** add allocated resource metrics for sandboxes ([#2943](#2943)) ([95cb6d3](95cb6d3)) * **orchestrator:** add dummy orchestrator binary for local API dev ([#2744](#2744)) ([ab56e25](ab56e25)) * **orchestrator:** add NetworkAssignHook for sandbox lifecycle extensions ([#3290](#3290)) ([3261963](3261963)) * **orchestrator:** add soft-delete marker label to the check metric ([#3144](#3144)) ([1ce64f8](1ce64f8)) * **orchestrator:** add v4HeaderForUncompressed FF bit ([#2669](#2669)) ([1f459ee](1f459ee)) * **orchestrator:** always include execution metrics in sandbox webhook events ([#2852](#2852)) ([440edfe](440edfe)) * **orchestrator:** classify envd-init by exit type ([#3139](#3139)) ([1e39a4f](1e39a4f)) * **orchestrator:** graceful sandbox drain on shutdown ([#3069](#3069)) ([6ce68e3](6ce68e3)) * **orchestrator:** graceful template-build drain on shutdown ([#3079](#3079)) ([1b3001c](1b3001c)) * **orchestrator:** improved read-path telemetry ([#3063](#3063)) ([bc3fe84](bc3fe84)) * **orchestrator:** LD-gated ClickHouse write fan-out feature flag ([#3152](#3152)) ([f046fcf](f046fcf)) * **orchestrator:** make build-reserved-disk-space-mb default 256MB ([#3065](#3065)) ([d473f98](d473f98)) * **orchestrator:** record upload compression metrics ([#2761](#2761)) ([9092e35](9092e35)) * **orchestrator:** report hugepage metrics to API ([#3182](#3182)) ([7735bae](7735bae)) * **orchestrator:** run startup reclaim on boot ([#3123](#3123)) ([79b838e](79b838e)) * **orchestrator:** single-instance flock on startup ([#3143](#3143)) ([1320d6e](1320d6e)) * **orchestrator:** soft-delete consumer enforcement for storage index ([#3034](#3034)) ([fbfc918](fbfc918)) * **orchestrator:** tag envd-init meters with start_type ([#3125](#3125)) ([4466b48](4466b48)) * **orchestrator:** track and report last status change timestamp ([#2980](#2980)) ([f79be77](f79be77)) * **orchestrator:** track sandbox lifecycles ([#2998](#2998)) ([057f20c](057f20c)) * **orchestrator:** write layer sizes (logical/mapped/diff) to object metadata ([#3122](#3122)) ([11869c0](11869c0)) * **orch:** harvest resume-prefetch trace on pause ([#3067](#3067)) ([97bd4a5](97bd4a5)) * **orch:** last-cycle memory prefetch on resume ([#3258](#3258)) ([b22e820](b22e820)) * **orch:** make resume-build -gdb work on real nodes + add copy-build -gdb ([#3108](#3108)) ([c684bd2](c684bd2)) * **orch:** opt-in DSCP marker for sandbox egress (SANDBOX_EGRESS_DSCP) ([#3039](#3039)) ([a98cf2c](a98cf2c)) * **orch:** per-start UFFD startup working-set metric ([#2960](#2960)) ([dc386b2](dc386b2)) * **orch:** premade NixOS base-image support ([#3412](#3412)) ([776ba39](776ba39)) * **orch:** record envd init duration histogram on failure with success attribute ([#2749](#2749)) ([afa7458](afa7458)) * **orch:** snapshot fragmentation metrics ([#2931](#2931)) ([842b007](842b007)) * per-team events TTL limit (tier + addons) ([#3181](#3181)) ([f76b2cb](f76b2cb)) * **shared:** add OTEL instrumentation to AWS S3 storage client ([#3172](#3172)) ([25b0fd1](25b0fd1)) * **storage:** per-role storage URLs, env-free storage library ([#3246](#3246)) ([fcbe909](fcbe909)) * **storage:** stamp provenance custom metadata on uploaded objects (incl. headers) ([#3033](#3033)) ([ba8604e](ba8604e)) * **storage:** write-through compressed templates to NFS on upload ([#2827](#2827)) ([57503c1](57503c1)) ### Bug Fixes * added api and orch ([#3454](#3454)) ([fda5e45](fda5e45)) * **block:** rephrase misleading error message in pwritevAll ([#2816](#2816)) ([1555f1b](1555f1b)) * **cache:** use 512-byte units for stat.Blocks in FileSize ([#2949](#2949)) ([0f632a9](0f632a9)) * **clean-nfs-cache:** exclude zombies from delete_age ([#3191](#3191)) ([3fa2aeb](3fa2aeb)) * **compression:** correctness findings from compression audit ([#2803](#2803)) ([d21a6a9](d21a6a9)) * **copy-build:** resolve compression suffix for build data files ([#2859](#2859)) ([8966f7e](8966f7e)) * correct 3 CVES ([#3218](#3218)) ([076823b](076823b)) * **envd:** stop freezing socat cgroup across pause/resume ([#2923](#2923)) ([8b6f2b9](8b6f2b9)) * **inspect-build:** adapt validate to new Chunker upstream API ([#2989](#2989)) ([2e0d3da](2e0d3da)) * **nbd:** adjust status poll sleep from 100ns to 100µs ([02bf51b](02bf51b)) * **nbd:** change NBD status poll sleep from 100ns to 100µs to avoid useless busy spinning ([#2884](#2884)) ([02bf51b](02bf51b)) * **nfsproxy:** deflake TestRoundTrip EADDRINUSE ([#2987](#2987)) ([55f4d18](55f4d18)) * **orch:** denormalize upload metric file type ([#2865](#2865)) ([b1646ca](b1646ca)) * **orch:** disable the chronyd seccomp filter on Alpine when using PHC ([#3453](#3453)) ([e58af28](e58af28)) * **orchestrator:** anchor rsync CWD to root in template file copy ([#2835](#2835)) ([7160db9](7160db9)) * **orchestrator:** atomically replace metadata ([#3321](#3321)) ([0c4ad6b](0c4ad6b)) * **orchestrator:** avoid serializing upload headers twice ([#2762](#2762)) ([9b7b149](9b7b149)) * **orchestrator:** chunk readiness bug in P2P->compressed ([#3185](#3185)) ([74a6e5b](74a6e5b)) * **orchestrator:** deschedule flaky eviction-loop race in TestDiffSto… ([#3173](#3173)) ([88ff17c](88ff17c)) * **orchestrator:** discard poisoned nftables conn on firewall errors ([#3008](#3008)) ([03f10e0](03f10e0)) * **orchestrator:** drop stale pre-init logs ([#3297](#3297)) ([8ec4be5](8ec4be5)) * **orchestrator:** emit compression ratios as fractions, not BP ([#2772](#2772)) ([866f4c1](866f4c1)) * **orchestrator:** export dirty-page stall counter from process start ([#2992](#2992)) ([badc8ad](badc8ad)) * **orchestrator:** harden Firecracker process shutdown ([#2996](#2996)) ([df662e7](df662e7)) * **orchestrator:** harden shutdown network cleanup ([#3000](#3000)) ([de2f391](de2f391)) * **orchestrator:** implement Docker COPY merge semantics in template builds ([#3283](#3283)) ([9174104](9174104)) * **orchestrator:** keep dedup empty-pages telemetry scan-only ([#2991](#2991)) ([35d0832](35d0832)) * **orchestrator:** let build-cache threshold flag raise above its fal… ([#3175](#3175)) ([06393c3](06393c3)) * **orchestrator:** log missing egress proxy in startup reclaim instead of defaulting silently ([#3116](#3116)) ([6ca3163](6ca3163)) * **orchestrator:** make copy-build handle filesystem-only snapshots ([#3299](#3299)) ([62add04](62add04)) * **orchestrator:** measure ext4 free space from block groups ([#3282](#3282)) ([f18f05f](f18f05f)) * **orchestrator:** normalize upload metric file labels ([#2767](#2767)) ([6dec8b3](6dec8b3)) * **orchestrator:** order egress config/firewall updates to close BYOP enable race ([#3313](#3313)) ([7faa59e](7faa59e)) * **orchestrator:** order envd.service after local-fs.target ([#3043](#3043)) ([ea2663e](ea2663e)) * **orchestrator:** order envd.service after systemd-tmpfiles-setup ([#3130](#3130)) ([9481811](9481811)) * **orchestrator:** pause upload retain retry ([#2993](#2993)) ([4f81799](4f81799)) * **orchestrator:** pin tap device host-side MAC address ([#3271](#3271)) ([3c786ba](3c786ba)) * **orchestrator:** pin UFFD copy source buffers ([#2745](#2745)) ([837fa91](837fa91)) * **orchestrator:** preserve full ENV value across stdout chunks ([#2740](#2740)) ([4822e6d](4822e6d)) * **orchestrator:** read V3 ancestors as uncompressed instead of failing ([#2994](#2994)) ([c479dd3](c479dd3)) * **orchestrator:** reject standby while draining ([#3325](#3325)) ([475a7ee](475a7ee)) * **orchestrator:** report real V4 header compression ratio ([#2771](#2771)) ([ecd344e](ecd344e)) * **orchestrator:** resolve remaining P2P/compression/V5 issues ([#3015](#3015)) ([1e4379e](1e4379e)) * **orchestrator:** sanitize OCI pull errors ([#3096](#3096)) ([a3af6c0](a3af6c0)) * **orchestrator:** scope rootfs hash to provision default ([#3129](#3129)) ([475f955](475f955)) * **orchestrator:** stop Checks health-loop leaking ([#2739](#2739)) ([17e6e60](17e6e60)) * **orchestrator:** survive SIGBUS from failing disks under mmap'd caches ([#3385](#3385)) ([728bba3](728bba3)) * **orchestrator:** tolerate missing header for legacy templates ([#3026](#3026)) ([8a44bfe](8a44bfe)) * **orch:** fall back to ID_LIKE with a warning instead of rejecting ([#3459](#3459)) ([7167818](7167818)) * **orch:** prevent NBD dispatch read-loop stall on WRITE_ZEROES (behind flag) ([#3048](#3048)) ([efd3d4d](efd3d4d)) * **orch:** split scheduling base build id per artifact ([#2920](#2920)) ([3e35a2a](3e35a2a)) * **orch:** validate copy-build -gdb buckets before the snapshot copy ([#3446](#3446)) ([586ad74](586ad74)) * **shared:** never report a failed envd command stream as success ([#3281](#3281)) ([69c06b6](69c06b6)) * **storage:** compression upload & cache correctness fixes ([#3231](#3231)) ([980748f](980748f)) * **storage:** don't assume V4+ ancestor gaps are uncompressed ([#3447](#3447)) ([bfdbb24](bfdbb24)) * **uffd:** dedupe deferred page faults ([#2864](#2864)) ([9680a41](9680a41)) * WrapContextAsUserError should not misclassify internal timeouts as user cancellations ([#3155](#3155)) ([8f83959](8f83959)) ### Performance Improvements * **build:** cache resolved Diff per BuildId within File.ReadAt ([#2838](#2838)) ([53de07f](53de07f)) * **build:** parallelize fragmented backing reads ([#2872](#2872)) ([c7655a7](c7655a7)) * **clean-nfs-cache:** restore dirfd-relative statx ([#2766](#2766)) ([6bdbedb](6bdbedb)) * **header:** add V5 columnar varint header format ([#2847](#2847)) ([9dd931b](9dd931b)) * **header:** pack cached Header.Mapping into a compact form ([#2844](#2844)) ([7f0b13c](7f0b13c)) * **orchestrator:** add memfile dedup density threshold ([#2862](#2862)) ([7ccfa02](7ccfa02)) * **orchestrator:** avoid V3-ancestor header refresh ([#2999](#2999)) ([cb6aa0b](cb6aa0b)) * **orch:** metrics for dirty page throttling ([#2858](#2858)) ([d2aa554](d2aa554)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: e2b-release-please[bot] <298072688+e2b-release-please[bot]@users.noreply.github.com> Co-authored-by: Charlie Wyse <charlie.wyse@e2b.dev>

🤖 I have created a release *beep* *boop* --- ## 0.0.1 (2026-07-30) ### Features * **api:** add sandbox IAM workload token configuration ([13ddb3d](13ddb3d)) * **api:** add sandbox workload identity permission ([#3319](#3319)) ([13ddb3d](13ddb3d)) * **api:** SOCKS5 egress proxy on sandbox network config (BYOP) ([#2642](#2642)) ([1fc3820](1fc3820)) * **cfg:** add DISABLE_STARTUP_RECLAIM flag ([#3081](#3081)) ([7677ca6](7677ca6)) * **clickhouse:** implement multi-cluster fan-out for events and stats ([#2925](#2925)) ([39594c6](39594c6)) * dynamic sandbox log routing and ClickHouse-backed log reads ([#3236](#3236)) ([1b19a3b](1b19a3b)) * **envd:** give envd realtime IO priority, reset for user processes ([#2681](#2681)) ([f4bd1b2](f4bd1b2)) * **envd:** split collapse stats into real migrations vs already-huge ([#3021](#3021)) ([0d77614](0d77614)) * **envd:** support user-defined file metadata via xattrs ([#2732](#2732)) ([da8fbe4](da8fbe4)) * **featureflags:** support per-service context providers ([#3100](#3100)) ([65297c1](65297c1)) * freeze user cgroup across pause/resume to keep envd /init responsive ([#2688](#2688)) ([eceb741](eceb741)) * **metrics:** break down pause-snapshot latency by step ([#3426](#3426)) ([f551118](f551118)) * **metrics:** label pause telemetry by fs_only ([#3425](#3425)) ([4be33ba](4be33ba)) * **observability:** add kill_reason to sandbox.lifecycle.killed ([#2833](#2833)) ([e45418f](e45418f)) * **observability:** include kill_reason in kill-path structured logs ([#2846](#2846)) ([33c49f7](33c49f7)) * **orch:** add envd-version to LaunchDarkly sandbox context ([#3051](#3051)) ([37d3b92](37d3b92)) * **orch:** add less, nftables, iputils-ping, and jq to base provisioning ([#2736](#2736)) ([a1e010e](a1e010e)) * **orch:** collapse envd's heap into 2 MiB hugepages before pause to cut cold-resume faults ([#2997](#2997)) ([6677f73](6677f73)) * **orch:** debug a sandbox guest kernel with resume-build -gdb ([#3040](#3040)) ([37bb0dc](37bb0dc)) * **orch:** decouple warm resume from memfile dedup ([#3166](#3166)) ([77f25a0](77f25a0)) * **orch:** distro-aware template base-image provisioning ([#3411](#3411)) ([1abece1](1abece1)) * **orchestrator/cgroup:** list and destroy leaked sandbox cgroups ([#3086](#3086)) ([bce1d84](bce1d84)) * **orchestrator/nbd:** inspect and disconnect connected devices ([#3087](#3087)) ([4d47148](4d47148)) * **orchestrator/network:** list slot namespaces ([#3089](#3089)) ([c23dbc7](c23dbc7)) * **orchestrator/network:** list slot namespaces ([#3090](#3090)) ([fbfce25](fbfce25)) * **orchestrator:** add -force-reboot to resume-build to cold-boot memory-snaphsot builds ([#3208](#3208)) ([cf8f15b](cf8f15b)) * **orchestrator:** add allocated resource metrics for sandboxes ([#2943](#2943)) ([95cb6d3](95cb6d3)) * **orchestrator:** add dummy orchestrator binary for local API dev ([#2744](#2744)) ([ab56e25](ab56e25)) * **orchestrator:** add NetworkAssignHook for sandbox lifecycle extensions ([#3290](#3290)) ([3261963](3261963)) * **orchestrator:** add soft-delete marker label to the check metric ([#3144](#3144)) ([1ce64f8](1ce64f8)) * **orchestrator:** add v4HeaderForUncompressed FF bit ([#2669](#2669)) ([1f459ee](1f459ee)) * **orchestrator:** always include execution metrics in sandbox webhook events ([#2852](#2852)) ([440edfe](440edfe)) * **orchestrator:** classify envd-init by exit type ([#3139](#3139)) ([1e39a4f](1e39a4f)) * **orchestrator:** graceful sandbox drain on shutdown ([#3069](#3069)) ([6ce68e3](6ce68e3)) * **orchestrator:** graceful template-build drain on shutdown ([#3079](#3079)) ([1b3001c](1b3001c)) * **orchestrator:** improved read-path telemetry ([#3063](#3063)) ([bc3fe84](bc3fe84)) * **orchestrator:** LD-gated ClickHouse write fan-out feature flag ([#3152](#3152)) ([f046fcf](f046fcf)) * **orchestrator:** make build-reserved-disk-space-mb default 256MB ([#3065](#3065)) ([d473f98](d473f98)) * **orchestrator:** record upload compression metrics ([#2761](#2761)) ([9092e35](9092e35)) * **orchestrator:** report hugepage metrics to API ([#3182](#3182)) ([7735bae](7735bae)) * **orchestrator:** run startup reclaim on boot ([#3123](#3123)) ([79b838e](79b838e)) * **orchestrator:** single-instance flock on startup ([#3143](#3143)) ([1320d6e](1320d6e)) * **orchestrator:** soft-delete consumer enforcement for storage index ([#3034](#3034)) ([fbfc918](fbfc918)) * **orchestrator:** tag envd-init meters with start_type ([#3125](#3125)) ([4466b48](4466b48)) * **orchestrator:** track and report last status change timestamp ([#2980](#2980)) ([f79be77](f79be77)) * **orchestrator:** track sandbox lifecycles ([#2998](#2998)) ([057f20c](057f20c)) * **orchestrator:** write layer sizes (logical/mapped/diff) to object metadata ([#3122](#3122)) ([11869c0](11869c0)) * **orch:** harvest resume-prefetch trace on pause ([#3067](#3067)) ([97bd4a5](97bd4a5)) * **orch:** last-cycle memory prefetch on resume ([#3258](#3258)) ([ea94196](ea94196)) * **orch:** make resume-build -gdb work on real nodes + add copy-build -gdb ([#3108](#3108)) ([5385594](5385594)) * **orch:** opt-in DSCP marker for sandbox egress (SANDBOX_EGRESS_DSCP) ([#3039](#3039)) ([a98cf2c](a98cf2c)) * **orch:** per-start UFFD startup working-set metric ([#2960](#2960)) ([dc386b2](dc386b2)) * **orch:** premade NixOS base-image support ([#3412](#3412)) ([4bd42d2](4bd42d2)) * **orch:** record envd init duration histogram on failure with success attribute ([#2749](#2749)) ([afa7458](afa7458)) * **orch:** snapshot fragmentation metrics ([#2931](#2931)) ([842b007](842b007)) * per-team events TTL limit (tier + addons) ([#3181](#3181)) ([f76b2cb](f76b2cb)) * **shared:** add OTEL instrumentation to AWS S3 storage client ([#3172](#3172)) ([25b0fd1](25b0fd1)) * **storage:** per-role storage URLs, env-free storage library ([#3246](#3246)) ([fcbe909](fcbe909)) * **storage:** stamp provenance custom metadata on uploaded objects (incl. headers) ([#3033](#3033)) ([ba8604e](ba8604e)) * **storage:** write-through compressed templates to NFS on upload ([#2827](#2827)) ([57503c1](57503c1)) ### Bug Fixes * added api and orch ([#3454](#3454)) ([d56e0a8](d56e0a8)) * **block:** rephrase misleading error message in pwritevAll ([#2816](#2816)) ([1555f1b](1555f1b)) * **cache:** use 512-byte units for stat.Blocks in FileSize ([#2949](#2949)) ([0f632a9](0f632a9)) * **clean-nfs-cache:** exclude zombies from delete_age ([#3191](#3191)) ([3fa2aeb](3fa2aeb)) * **compression:** correctness findings from compression audit ([#2803](#2803)) ([d21a6a9](d21a6a9)) * **copy-build:** resolve compression suffix for build data files ([#2859](#2859)) ([8966f7e](8966f7e)) * correct 3 CVES ([#3218](#3218)) ([076823b](076823b)) * **envd:** stop freezing socat cgroup across pause/resume ([#2923](#2923)) ([8b6f2b9](8b6f2b9)) * **inspect-build:** adapt validate to new Chunker upstream API ([#2989](#2989)) ([2e0d3da](2e0d3da)) * **nbd:** adjust status poll sleep from 100ns to 100µs ([02bf51b](02bf51b)) * **nbd:** change NBD status poll sleep from 100ns to 100µs to avoid useless busy spinning ([#2884](#2884)) ([02bf51b](02bf51b)) * **nfsproxy:** deflake TestRoundTrip EADDRINUSE ([#2987](#2987)) ([55f4d18](55f4d18)) * **orch:** denormalize upload metric file type ([#2865](#2865)) ([b1646ca](b1646ca)) * **orch:** disable the chronyd seccomp filter on Alpine when using PHC ([#3453](#3453)) ([dfa9764](dfa9764)) * **orchestrator:** anchor rsync CWD to root in template file copy ([#2835](#2835)) ([7160db9](7160db9)) * **orchestrator:** atomically replace metadata ([#3321](#3321)) ([0c4ad6b](0c4ad6b)) * **orchestrator:** avoid serializing upload headers twice ([#2762](#2762)) ([9b7b149](9b7b149)) * **orchestrator:** chunk readiness bug in P2P->compressed ([#3185](#3185)) ([74a6e5b](74a6e5b)) * **orchestrator:** deschedule flaky eviction-loop race in TestDiffSto… ([#3173](#3173)) ([88ff17c](88ff17c)) * **orchestrator:** discard poisoned nftables conn on firewall errors ([#3008](#3008)) ([03f10e0](03f10e0)) * **orchestrator:** drop stale pre-init logs ([#3297](#3297)) ([8ec4be5](8ec4be5)) * **orchestrator:** emit compression ratios as fractions, not BP ([#2772](#2772)) ([866f4c1](866f4c1)) * **orchestrator:** export dirty-page stall counter from process start ([#2992](#2992)) ([badc8ad](badc8ad)) * **orchestrator:** harden Firecracker process shutdown ([#2996](#2996)) ([df662e7](df662e7)) * **orchestrator:** harden shutdown network cleanup ([#3000](#3000)) ([de2f391](de2f391)) * **orchestrator:** implement Docker COPY merge semantics in template builds ([#3283](#3283)) ([9174104](9174104)) * **orchestrator:** keep dedup empty-pages telemetry scan-only ([#2991](#2991)) ([35d0832](35d0832)) * **orchestrator:** let build-cache threshold flag raise above its fal… ([#3175](#3175)) ([06393c3](06393c3)) * **orchestrator:** log missing egress proxy in startup reclaim instead of defaulting silently ([#3116](#3116)) ([6ca3163](6ca3163)) * **orchestrator:** make copy-build handle filesystem-only snapshots ([#3299](#3299)) ([62add04](62add04)) * **orchestrator:** measure ext4 free space from block groups ([#3282](#3282)) ([f18f05f](f18f05f)) * **orchestrator:** normalize upload metric file labels ([#2767](#2767)) ([6dec8b3](6dec8b3)) * **orchestrator:** order egress config/firewall updates to close BYOP enable race ([#3313](#3313)) ([7faa59e](7faa59e)) * **orchestrator:** order envd.service after local-fs.target ([#3043](#3043)) ([ea2663e](ea2663e)) * **orchestrator:** order envd.service after systemd-tmpfiles-setup ([#3130](#3130)) ([9481811](9481811)) * **orchestrator:** pause upload retain retry ([#2993](#2993)) ([4f81799](4f81799)) * **orchestrator:** pin tap device host-side MAC address ([#3271](#3271)) ([3c786ba](3c786ba)) * **orchestrator:** pin UFFD copy source buffers ([#2745](#2745)) ([837fa91](837fa91)) * **orchestrator:** preserve full ENV value across stdout chunks ([#2740](#2740)) ([4822e6d](4822e6d)) * **orchestrator:** read V3 ancestors as uncompressed instead of failing ([#2994](#2994)) ([c479dd3](c479dd3)) * **orchestrator:** reject standby while draining ([#3325](#3325)) ([475a7ee](475a7ee)) * **orchestrator:** report real V4 header compression ratio ([#2771](#2771)) ([ecd344e](ecd344e)) * **orchestrator:** resolve remaining P2P/compression/V5 issues ([#3015](#3015)) ([1e4379e](1e4379e)) * **orchestrator:** sanitize OCI pull errors ([#3096](#3096)) ([a3af6c0](a3af6c0)) * **orchestrator:** scope rootfs hash to provision default ([#3129](#3129)) ([475f955](475f955)) * **orchestrator:** stop Checks health-loop leaking ([#2739](#2739)) ([17e6e60](17e6e60)) * **orchestrator:** survive SIGBUS from failing disks under mmap'd caches ([#3385](#3385)) ([8694d08](8694d08)) * **orchestrator:** tolerate missing header for legacy templates ([#3026](#3026)) ([8a44bfe](8a44bfe)) * **orch:** fall back to ID_LIKE with a warning instead of rejecting ([#3459](#3459)) ([73399b3](73399b3)) * **orch:** prevent NBD dispatch read-loop stall on WRITE_ZEROES (behind flag) ([#3048](#3048)) ([efd3d4d](efd3d4d)) * **orch:** split scheduling base build id per artifact ([#2920](#2920)) ([3e35a2a](3e35a2a)) * **orch:** validate copy-build -gdb buckets before the snapshot copy ([#3446](#3446)) ([9be382f](9be382f)) * **shared:** never report a failed envd command stream as success ([#3281](#3281)) ([69c06b6](69c06b6)) * **storage:** compression upload & cache correctness fixes ([#3231](#3231)) ([980748f](980748f)) * **storage:** don't assume V4+ ancestor gaps are uncompressed ([#3447](#3447)) ([f828d12](f828d12)) * **uffd:** dedupe deferred page faults ([#2864](#2864)) ([9680a41](9680a41)) * WrapContextAsUserError should not misclassify internal timeouts as user cancellations ([#3155](#3155)) ([8f83959](8f83959)) ### Performance Improvements * **build:** cache resolved Diff per BuildId within File.ReadAt ([#2838](#2838)) ([53de07f](53de07f)) * **build:** parallelize fragmented backing reads ([#2872](#2872)) ([c7655a7](c7655a7)) * **clean-nfs-cache:** restore dirfd-relative statx ([#2766](#2766)) ([6bdbedb](6bdbedb)) * **header:** add V5 columnar varint header format ([#2847](#2847)) ([9dd931b](9dd931b)) * **header:** pack cached Header.Mapping into a compact form ([#2844](#2844)) ([7f0b13c](7f0b13c)) * **orchestrator:** add memfile dedup density threshold ([#2862](#2862)) ([7ccfa02](7ccfa02)) * **orchestrator:** avoid V3-ancestor header refresh ([#2999](#2999)) ([cb6aa0b](cb6aa0b)) * **orch:** metrics for dirty page throttling ([#2858](#2858)) ([d2aa554](d2aa554)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: e2b-release-please[bot] <298072688+e2b-release-please[bot]@users.noreply.github.com> Co-authored-by: Charlie Wyse <charlie.wyse@e2b.dev>

rguliyev requested review from ValentaTomas, dobrac and jakubno as code owners July 16, 2026 04:17

cla-bot Bot added the cla-signed label Jul 16, 2026

rguliyev requested a review from matthewlouisbrockman July 16, 2026 04:17

gemini-code-assist Bot reviewed Jul 16, 2026

View reviewed changes

rguliyev requested a review from wj-e2b July 16, 2026 04:19

rguliyev force-pushed the fix/envd-pre-init-log-timestamp branch from 9f99e1b to 308c608 Compare July 16, 2026 04:23

claude Bot reviewed Jul 16, 2026

View reviewed changes

Comment thread packages/orchestrator/pkg/hyperloopserver/handlers/logs.go Outdated

tvi approved these changes Jul 16, 2026

View reviewed changes

rguliyev force-pushed the fix/envd-pre-init-log-timestamp branch 2 times, most recently from 734eb85 to 9cdcd74 Compare July 16, 2026 18:57

chatgpt-codex-connector Bot reviewed Jul 16, 2026

View reviewed changes

Comment thread packages/orchestrator/pkg/hyperloopserver/handlers/logs.go Outdated

rguliyev force-pushed the fix/envd-pre-init-log-timestamp branch from 9cdcd74 to 9a50790 Compare July 16, 2026 19:12

rguliyev changed the title ~~fix(orchestrator): preserve stale pre-init envd logs' true timing~~ fix(orchestrator): drop stale pre-init logs Jul 16, 2026

matthewlouisbrockman approved these changes Jul 16, 2026

View reviewed changes

rguliyev force-pushed the fix/envd-pre-init-log-timestamp branch from 9a50790 to b9d2947 Compare July 16, 2026 23:43

Merge branch 'main' into fix/envd-pre-init-log-timestamp

9ceb20d

chatgpt-codex-connector Bot reviewed Jul 16, 2026

View reviewed changes

rguliyev enabled auto-merge (squash) July 16, 2026 23:51

rguliyev merged commit 8ec4be5 into main Jul 17, 2026
40 checks passed

rguliyev deleted the fix/envd-pre-init-log-timestamp branch July 17, 2026 00:06

e2b-release-please Bot mentioned this pull request Jul 30, 2026

chore(main): release orchestrator 0.0.1 #3470

Merged

e2b-release-please Bot mentioned this pull request Aug 3, 2026

chore(main): release orchestrator 0.1.0 #3506

Closed

Navigation Menu

fix(orchestrator): drop stale pre-init logs - #3297

fix(orchestrator): drop stale pre-init logs#3297
rguliyev merged 2 commits into
mainfrom
fix/envd-pre-init-log-timestamp

rguliyev commented Jul 16, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

gemini-code-assist Bot Jul 16, 2026

Uh oh!

rguliyev Jul 16, 2026

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Jul 16, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

matthewlouisbrockman left a comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Jul 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

rguliyev commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

gemini-code-assist Bot Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

rguliyev Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 1 Tests Failed:

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

matthewlouisbrockman left a comment

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rguliyev commented Jul 16, 2026 •

edited

Loading

codecov Bot commented Jul 16, 2026 •

edited

Loading