feat(clickhouse): implement multi-cluster fan-out for events and stats by rguliyev · Pull Request #2925 · e2b-dev/runtime

rguliyev · 2026-06-04T02:02:43Z

Adds support for CLICKHOUSE_CONNECTION_STRINGS to allow concurrent writing to multiple clusters. Includes deduplication logic, isolated batchers for additional endpoints, and resource cleanup fixes.

linear-code · 2026-06-04T02:02:47Z

ENG-4250

gemini-code-assist

Code Review

The sync.WaitGroup type does not have a Go method in packages/clickhouse/pkg/hoststats/hoststats.go and packages/orchestrator/pkg/events/events.go, which will cause compilation errors. Additionally, calling Close on driver.Batch in packages/clickhouse/pkg/events/delivery.go and packages/clickhouse/pkg/hoststats/delivery.go will result in compilation errors as the driver.Batch interface in clickhouse-go/v2 does not expose a Close method.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

codecov · 2026-06-04T02:04:24Z

❌ 4 Tests Failed:

Tests completed	Failed	Passed	Skipped
2746	4	2742	7

View the full list of 4 ❄️ flaky test(s)

github.com/e2b-dev/infra/tests/integration/internal/tests/orchestrator::TestSandboxMemoryIntegrity

Flake rate in main: 54.90% (Passed 884 times, Failed 1076 times)

Stack Traces | 73.4s run time

=== RUN   TestSandboxMemoryIntegrity
=== PAUSE TestSandboxMemoryIntegrity
=== CONT  TestSandboxMemoryIntegrity
    sandbox_memory_integrity_test.go:27: Build completed successfully
--- FAIL: TestSandboxMemoryIntegrity (73.43s)

github.com/e2b-dev/infra/tests/integration/internal/tests/orchestrator::TestSandboxMemoryIntegrity/tmpfs_hash

Flake rate in main: 54.97% (Passed 874 times, Failed 1067 times)

Stack Traces | 202s run time

=== RUN   TestSandboxMemoryIntegrity/tmpfs_hash
=== PAUSE TestSandboxMemoryIntegrity/tmpfs_hash
=== CONT  TestSandboxMemoryIntegrity/tmpfs_hash
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{start:{pid:1276}}
Executing command bash in sandbox ittycfh3xys0t7a145y2v (user: root)
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stdout:"Total memory: 985 MB\nUsed memory before tmpfs mount: 190 MB\nFree memory before tmpfs mount: 793 MB\nMemory to use in integrity test (60% of free, min 64MB): 475 MB\n"}}
Executing command bash in sandbox ittycfh3xys0t7a145y2v (user: root)
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"475+0 records in\n475+0 records out\n498073600 bytes (498 MB, 475 MiB) copied, 2.18746 s, 228 MB/s\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\t"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"C"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"o"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"m"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"m"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"a"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"d"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"b"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"e"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"i"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"g"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"t"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"i"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"m"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"e"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"d"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:":"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\""}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"dd"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"if=/dev/urandom"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"of=/mnt/testfile"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"bs=1M"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"count=475"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\""}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\t"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"U"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"s"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"e"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"r"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"t"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"i"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"m"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"e"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"("}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"s"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"e"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"c"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"o"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"d"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"s"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:")"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:":"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"0.00"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\t"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"S"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"y"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"s"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"t"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"e"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"m"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"time (seconds): 2.12\n\tPercent of C"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"PU this job got: 97%\n\tElapsed (wall clock) time (h:mm:ss or m:ss): "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"0:02.19\n\tAverage shared text size (kbytes)"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:": 0\n\tAverage unshared data size (kbytes): 0\n\tAverage stack size (kbytes): 0\n\tAverage total size (kbytes): 0\n\tMaximum resident set size (kbytes): 2680\n\tAverage resident set size (kbytes): 0\n\tMajor (requiring I/O) page faults: 2\n\tMinor (reclaiming a frame) page faults: 345\n\tVoluntary context switches: 3\n\tInvoluntary context switches: 58\n\tSwaps: 0\n\tFile system inputs: 176\n\tFile system outputs: 0\n\tSocket messages sent: 0\n\tSocket messages received: 0\n\tSignals delivered: 0\n\tPage size (bytes): 4096\n\tExit status: 0\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stdout:"Used memory after tmpfs mount and file fill: 675 MB\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{end:{exited:true status:"exit status 0"}}
    sandbox_memory_integrity_test.go:70: Command [bash] completed successfully in sandbox i68dpq0vpdl3vfgovyyrd
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{start:{pid:1293}}
Executing command bash in sandbox ib9ja84o3160tb3yn3p58 (user: root)
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{data:{stdout:"e7acc342d4d90a7c0966e12eb31bb0ef238c64339a7e2a794c88b821463d64f0\n"}}
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{end:{exited:true status:"exit status 0"}}
    sandbox_memory_integrity_test.go:80: Command [bash] completed successfully in sandbox i68dpq0vpdl3vfgovyyrd
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{start:{pid:1296}}
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
Executing command bash in sandbox i68dpq0vpdl3vfgovyyrd (user: root)
    sandbox_memory_integrity_test.go:110: 
        	Error Trace:	.../tests/orchestrator/sandbox_memory_integrity_test.go:81
        	            				.../hostedtoolcache/go/1.26.3.../src/runtime/asm_amd64.s:1771
        	Error:      	Received unexpected error:
        	            	failed to execute command bash in sandbox i68dpq0vpdl3vfgovyyrd: unavailable: HTTP status 502 Bad Gateway
    sandbox_memory_integrity_test.go:110: 
        	Error Trace:	.../tests/orchestrator/sandbox_memory_integrity_test.go:78
        	            				.../tests/orchestrator/sandbox_memory_integrity_test.go:110
        	Error:      	Condition never satisfied
        	Test:       	TestSandboxMemoryIntegrity/tmpfs_hash
--- FAIL: TestSandboxMemoryIntegrity/tmpfs_hash (201.52s)

github.com/e2b-dev/infra/tests/integration/internal/tests/proxies::TestEnvdAccessTokenAutoResumeViaProxy

Flake rate in main: 40.01% (Passed 880 times, Failed 587 times)

Stack Traces | 10.7s run time

=== RUN   TestEnvdAccessTokenAutoResumeViaProxy
=== PAUSE TestEnvdAccessTokenAutoResumeViaProxy
=== CONT  TestEnvdAccessTokenAutoResumeViaProxy
    traffic_access_token_test.go:357: 
        	Error Trace:	.../tests/proxies/traffic_access_token_test.go:357
        	Error:      	Received unexpected error:
        	            	Get "http://localhost:3002/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
        	Test:       	TestEnvdAccessTokenAutoResumeViaProxy
--- FAIL: TestEnvdAccessTokenAutoResumeViaProxy (10.68s)

github.com/e2b-dev/infra/tests/integration/internal/tests/proxies::TestSandboxAutoResumeViaProxy

Flake rate in main: 40.47% (Passed 878 times, Failed 597 times)

Stack Traces | 25.5s run time

=== RUN   TestSandboxAutoResumeViaProxy
=== PAUSE TestSandboxAutoResumeViaProxy
=== CONT  TestSandboxAutoResumeViaProxy
    auto_resume_test.go:116: 
        	Error Trace:	.../tests/proxies/auto_resume_test.go:116
        	Error:      	Received unexpected error:
        	            	Get "http://localhost:3002": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
        	Test:       	TestSandboxAutoResumeViaProxy
--- FAIL: TestSandboxAutoResumeViaProxy (25.49s)

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bbbd9468aa

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 711b5938a0

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Adds support for CLICKHOUSE_CONNECTION_STRINGS to allow concurrent writing to multiple clusters. Includes deduplication logic, isolated batchers for additional endpoints, and resource cleanup fixes.

e2b-dev#2925) Adds support for CLICKHOUSE_CONNECTION_STRINGS to allow concurrent writing to multiple clusters. Includes deduplication logic, isolated batchers for additional endpoints, and resource cleanup fixes.

🤖 I have created a release *beep* *boop* --- ## 1.0.0 (2026-07-07) ### Features * Adding client-proxy and clickhouse to e2b-artifacts ([#3210](#3210)) ([5686d88](5686d88)) * **api:** LD-gated ClickHouse read switcher ([#3061](#3061)) ([29e74ca](29e74ca)) * **clickhouse:** implement multi-cluster fan-out for events and stats ([#2925](#2925)) ([39594c6](39594c6)) * **migrations:** add webhook deliveries table to ClickHouse ([#2741](#2741)) ([f55a5bd](f55a5bd)) * **orchestrator:** LD-gated ClickHouse write fan-out feature flag ([#3152](#3152)) ([f046fcf](f046fcf)) * **orch:** re-enable memory.peak per-FD reset (cgroups v2) ([#2430](#2430)) ([5d9ac5f](5d9ac5f)) * per-team events TTL limit (tier + addons) ([#3181](#3181)) ([f76b2cb](f76b2cb)) ### Bug Fixes * push clickhouse-migrator image to both latest and commit SHA tags ([#2954](#2954)) ([3b780d5](3b780d5)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: e2b-release-please[bot] <298072688+e2b-release-please[bot]@users.noreply.github.com>

🤖 I have created a release *beep* *boop* --- ## 0.0.1 (2026-07-10) ### Features * Adding client-proxy and clickhouse to e2b-artifacts ([#3210](#3210)) ([5686d88](5686d88)) * **api:** LD-gated ClickHouse read switcher ([#3061](#3061)) ([29e74ca](29e74ca)) * **clickhouse:** implement multi-cluster fan-out for events and stats ([#2925](#2925)) ([39594c6](39594c6)) * **migrations:** add webhook deliveries table to ClickHouse ([#2741](#2741)) ([f55a5bd](f55a5bd)) * **orchestrator:** LD-gated ClickHouse write fan-out feature flag ([#3152](#3152)) ([f046fcf](f046fcf)) * per-team events TTL limit (tier + addons) ([#3181](#3181)) ([f76b2cb](f76b2cb)) ### Bug Fixes * correct 3 CVES ([#3218](#3218)) ([076823b](076823b)) * push clickhouse-migrator image to both latest and commit SHA tags ([#2954](#2954)) ([3b780d5](3b780d5)) * reset artifacts ([#3259](#3259)) ([93f7eb5](93f7eb5)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: e2b-release-please[bot] <298072688+e2b-release-please[bot]@users.noreply.github.com> Co-authored-by: Charlie Wyse <charlie.wyse@e2b.dev>

🤖 I have created a release *beep* *boop* --- ## 0.0.1 (2026-07-30) ### Features * **api:** add sandbox IAM workload token configuration ([13ddb3d](13ddb3d)) * **api:** add sandbox workload identity permission ([#3319](#3319)) ([13ddb3d](13ddb3d)) * **api:** SOCKS5 egress proxy on sandbox network config (BYOP) ([#2642](#2642)) ([1fc3820](1fc3820)) * **cfg:** add DISABLE_STARTUP_RECLAIM flag ([#3081](#3081)) ([7677ca6](7677ca6)) * **clickhouse:** implement multi-cluster fan-out for events and stats ([#2925](#2925)) ([39594c6](39594c6)) * dynamic sandbox log routing and ClickHouse-backed log reads ([#3236](#3236)) ([1b19a3b](1b19a3b)) * **envd:** give envd realtime IO priority, reset for user processes ([#2681](#2681)) ([f4bd1b2](f4bd1b2)) * **envd:** split collapse stats into real migrations vs already-huge ([#3021](#3021)) ([0d77614](0d77614)) * **envd:** support user-defined file metadata via xattrs ([#2732](#2732)) ([da8fbe4](da8fbe4)) * **featureflags:** support per-service context providers ([#3100](#3100)) ([65297c1](65297c1)) * freeze user cgroup across pause/resume to keep envd /init responsive ([#2688](#2688)) ([eceb741](eceb741)) * **metrics:** break down pause-snapshot latency by step ([#3426](#3426)) ([657559e](657559e)) * **metrics:** label pause telemetry by fs_only ([#3425](#3425)) ([411b63e](411b63e)) * **observability:** add kill_reason to sandbox.lifecycle.killed ([#2833](#2833)) ([e45418f](e45418f)) * **observability:** include kill_reason in kill-path structured logs ([#2846](#2846)) ([33c49f7](33c49f7)) * **orch:** add envd-version to LaunchDarkly sandbox context ([#3051](#3051)) ([37d3b92](37d3b92)) * **orch:** add less, nftables, iputils-ping, and jq to base provisioning ([#2736](#2736)) ([a1e010e](a1e010e)) * **orch:** collapse envd's heap into 2 MiB hugepages before pause to cut cold-resume faults ([#2997](#2997)) ([6677f73](6677f73)) * **orch:** debug a sandbox guest kernel with resume-build -gdb ([#3040](#3040)) ([37bb0dc](37bb0dc)) * **orch:** decouple warm resume from memfile dedup ([#3166](#3166)) ([77f25a0](77f25a0)) * **orch:** distro-aware template base-image provisioning ([#3411](#3411)) ([f8c7b5b](f8c7b5b)) * **orchestrator/cgroup:** list and destroy leaked sandbox cgroups ([#3086](#3086)) ([bce1d84](bce1d84)) * **orchestrator/nbd:** inspect and disconnect connected devices ([#3087](#3087)) ([4d47148](4d47148)) * **orchestrator/network:** list slot namespaces ([#3089](#3089)) ([c23dbc7](c23dbc7)) * **orchestrator/network:** list slot namespaces ([#3090](#3090)) ([fbfce25](fbfce25)) * **orchestrator:** add -force-reboot to resume-build to cold-boot memory-snaphsot builds ([#3208](#3208)) ([cf8f15b](cf8f15b)) * **orchestrator:** add allocated resource metrics for sandboxes ([#2943](#2943)) ([95cb6d3](95cb6d3)) * **orchestrator:** add dummy orchestrator binary for local API dev ([#2744](#2744)) ([ab56e25](ab56e25)) * **orchestrator:** add NetworkAssignHook for sandbox lifecycle extensions ([#3290](#3290)) ([3261963](3261963)) * **orchestrator:** add soft-delete marker label to the check metric ([#3144](#3144)) ([1ce64f8](1ce64f8)) * **orchestrator:** add v4HeaderForUncompressed FF bit ([#2669](#2669)) ([1f459ee](1f459ee)) * **orchestrator:** always include execution metrics in sandbox webhook events ([#2852](#2852)) ([440edfe](440edfe)) * **orchestrator:** classify envd-init by exit type ([#3139](#3139)) ([1e39a4f](1e39a4f)) * **orchestrator:** graceful sandbox drain on shutdown ([#3069](#3069)) ([6ce68e3](6ce68e3)) * **orchestrator:** graceful template-build drain on shutdown ([#3079](#3079)) ([1b3001c](1b3001c)) * **orchestrator:** improved read-path telemetry ([#3063](#3063)) ([bc3fe84](bc3fe84)) * **orchestrator:** LD-gated ClickHouse write fan-out feature flag ([#3152](#3152)) ([f046fcf](f046fcf)) * **orchestrator:** make build-reserved-disk-space-mb default 256MB ([#3065](#3065)) ([d473f98](d473f98)) * **orchestrator:** record upload compression metrics ([#2761](#2761)) ([9092e35](9092e35)) * **orchestrator:** report hugepage metrics to API ([#3182](#3182)) ([7735bae](7735bae)) * **orchestrator:** run startup reclaim on boot ([#3123](#3123)) ([79b838e](79b838e)) * **orchestrator:** single-instance flock on startup ([#3143](#3143)) ([1320d6e](1320d6e)) * **orchestrator:** soft-delete consumer enforcement for storage index ([#3034](#3034)) ([fbfc918](fbfc918)) * **orchestrator:** tag envd-init meters with start_type ([#3125](#3125)) ([4466b48](4466b48)) * **orchestrator:** track and report last status change timestamp ([#2980](#2980)) ([f79be77](f79be77)) * **orchestrator:** track sandbox lifecycles ([#2998](#2998)) ([057f20c](057f20c)) * **orchestrator:** write layer sizes (logical/mapped/diff) to object metadata ([#3122](#3122)) ([11869c0](11869c0)) * **orch:** harvest resume-prefetch trace on pause ([#3067](#3067)) ([97bd4a5](97bd4a5)) * **orch:** last-cycle memory prefetch on resume ([#3258](#3258)) ([b22e820](b22e820)) * **orch:** make resume-build -gdb work on real nodes + add copy-build -gdb ([#3108](#3108)) ([c684bd2](c684bd2)) * **orch:** opt-in DSCP marker for sandbox egress (SANDBOX_EGRESS_DSCP) ([#3039](#3039)) ([a98cf2c](a98cf2c)) * **orch:** per-start UFFD startup working-set metric ([#2960](#2960)) ([dc386b2](dc386b2)) * **orch:** premade NixOS base-image support ([#3412](#3412)) ([776ba39](776ba39)) * **orch:** record envd init duration histogram on failure with success attribute ([#2749](#2749)) ([afa7458](afa7458)) * **orch:** snapshot fragmentation metrics ([#2931](#2931)) ([842b007](842b007)) * per-team events TTL limit (tier + addons) ([#3181](#3181)) ([f76b2cb](f76b2cb)) * **shared:** add OTEL instrumentation to AWS S3 storage client ([#3172](#3172)) ([25b0fd1](25b0fd1)) * **storage:** per-role storage URLs, env-free storage library ([#3246](#3246)) ([fcbe909](fcbe909)) * **storage:** stamp provenance custom metadata on uploaded objects (incl. headers) ([#3033](#3033)) ([ba8604e](ba8604e)) * **storage:** write-through compressed templates to NFS on upload ([#2827](#2827)) ([57503c1](57503c1)) ### Bug Fixes * added api and orch ([#3454](#3454)) ([fda5e45](fda5e45)) * **block:** rephrase misleading error message in pwritevAll ([#2816](#2816)) ([1555f1b](1555f1b)) * **cache:** use 512-byte units for stat.Blocks in FileSize ([#2949](#2949)) ([0f632a9](0f632a9)) * **clean-nfs-cache:** exclude zombies from delete_age ([#3191](#3191)) ([3fa2aeb](3fa2aeb)) * **compression:** correctness findings from compression audit ([#2803](#2803)) ([d21a6a9](d21a6a9)) * **copy-build:** resolve compression suffix for build data files ([#2859](#2859)) ([8966f7e](8966f7e)) * correct 3 CVES ([#3218](#3218)) ([076823b](076823b)) * **envd:** stop freezing socat cgroup across pause/resume ([#2923](#2923)) ([8b6f2b9](8b6f2b9)) * **inspect-build:** adapt validate to new Chunker upstream API ([#2989](#2989)) ([2e0d3da](2e0d3da)) * **nbd:** adjust status poll sleep from 100ns to 100µs ([02bf51b](02bf51b)) * **nbd:** change NBD status poll sleep from 100ns to 100µs to avoid useless busy spinning ([#2884](#2884)) ([02bf51b](02bf51b)) * **nfsproxy:** deflake TestRoundTrip EADDRINUSE ([#2987](#2987)) ([55f4d18](55f4d18)) * **orch:** denormalize upload metric file type ([#2865](#2865)) ([b1646ca](b1646ca)) * **orch:** disable the chronyd seccomp filter on Alpine when using PHC ([#3453](#3453)) ([e58af28](e58af28)) * **orchestrator:** anchor rsync CWD to root in template file copy ([#2835](#2835)) ([7160db9](7160db9)) * **orchestrator:** atomically replace metadata ([#3321](#3321)) ([0c4ad6b](0c4ad6b)) * **orchestrator:** avoid serializing upload headers twice ([#2762](#2762)) ([9b7b149](9b7b149)) * **orchestrator:** chunk readiness bug in P2P->compressed ([#3185](#3185)) ([74a6e5b](74a6e5b)) * **orchestrator:** deschedule flaky eviction-loop race in TestDiffSto… ([#3173](#3173)) ([88ff17c](88ff17c)) * **orchestrator:** discard poisoned nftables conn on firewall errors ([#3008](#3008)) ([03f10e0](03f10e0)) * **orchestrator:** drop stale pre-init logs ([#3297](#3297)) ([8ec4be5](8ec4be5)) * **orchestrator:** emit compression ratios as fractions, not BP ([#2772](#2772)) ([866f4c1](866f4c1)) * **orchestrator:** export dirty-page stall counter from process start ([#2992](#2992)) ([badc8ad](badc8ad)) * **orchestrator:** harden Firecracker process shutdown ([#2996](#2996)) ([df662e7](df662e7)) * **orchestrator:** harden shutdown network cleanup ([#3000](#3000)) ([de2f391](de2f391)) * **orchestrator:** implement Docker COPY merge semantics in template builds ([#3283](#3283)) ([9174104](9174104)) * **orchestrator:** keep dedup empty-pages telemetry scan-only ([#2991](#2991)) ([35d0832](35d0832)) * **orchestrator:** let build-cache threshold flag raise above its fal… ([#3175](#3175)) ([06393c3](06393c3)) * **orchestrator:** log missing egress proxy in startup reclaim instead of defaulting silently ([#3116](#3116)) ([6ca3163](6ca3163)) * **orchestrator:** make copy-build handle filesystem-only snapshots ([#3299](#3299)) ([62add04](62add04)) * **orchestrator:** measure ext4 free space from block groups ([#3282](#3282)) ([f18f05f](f18f05f)) * **orchestrator:** normalize upload metric file labels ([#2767](#2767)) ([6dec8b3](6dec8b3)) * **orchestrator:** order egress config/firewall updates to close BYOP enable race ([#3313](#3313)) ([7faa59e](7faa59e)) * **orchestrator:** order envd.service after local-fs.target ([#3043](#3043)) ([ea2663e](ea2663e)) * **orchestrator:** order envd.service after systemd-tmpfiles-setup ([#3130](#3130)) ([9481811](9481811)) * **orchestrator:** pause upload retain retry ([#2993](#2993)) ([4f81799](4f81799)) * **orchestrator:** pin tap device host-side MAC address ([#3271](#3271)) ([3c786ba](3c786ba)) * **orchestrator:** pin UFFD copy source buffers ([#2745](#2745)) ([837fa91](837fa91)) * **orchestrator:** preserve full ENV value across stdout chunks ([#2740](#2740)) ([4822e6d](4822e6d)) * **orchestrator:** read V3 ancestors as uncompressed instead of failing ([#2994](#2994)) ([c479dd3](c479dd3)) * **orchestrator:** reject standby while draining ([#3325](#3325)) ([475a7ee](475a7ee)) * **orchestrator:** report real V4 header compression ratio ([#2771](#2771)) ([ecd344e](ecd344e)) * **orchestrator:** resolve remaining P2P/compression/V5 issues ([#3015](#3015)) ([1e4379e](1e4379e)) * **orchestrator:** sanitize OCI pull errors ([#3096](#3096)) ([a3af6c0](a3af6c0)) * **orchestrator:** scope rootfs hash to provision default ([#3129](#3129)) ([475f955](475f955)) * **orchestrator:** stop Checks health-loop leaking ([#2739](#2739)) ([17e6e60](17e6e60)) * **orchestrator:** survive SIGBUS from failing disks under mmap'd caches ([#3385](#3385)) ([728bba3](728bba3)) * **orchestrator:** tolerate missing header for legacy templates ([#3026](#3026)) ([8a44bfe](8a44bfe)) * **orch:** fall back to ID_LIKE with a warning instead of rejecting ([#3459](#3459)) ([7167818](7167818)) * **orch:** prevent NBD dispatch read-loop stall on WRITE_ZEROES (behind flag) ([#3048](#3048)) ([efd3d4d](efd3d4d)) * **orch:** split scheduling base build id per artifact ([#2920](#2920)) ([3e35a2a](3e35a2a)) * **orch:** validate copy-build -gdb buckets before the snapshot copy ([#3446](#3446)) ([586ad74](586ad74)) * **shared:** never report a failed envd command stream as success ([#3281](#3281)) ([69c06b6](69c06b6)) * **storage:** compression upload & cache correctness fixes ([#3231](#3231)) ([980748f](980748f)) * **storage:** don't assume V4+ ancestor gaps are uncompressed ([#3447](#3447)) ([bfdbb24](bfdbb24)) * **uffd:** dedupe deferred page faults ([#2864](#2864)) ([9680a41](9680a41)) * WrapContextAsUserError should not misclassify internal timeouts as user cancellations ([#3155](#3155)) ([8f83959](8f83959)) ### Performance Improvements * **build:** cache resolved Diff per BuildId within File.ReadAt ([#2838](#2838)) ([53de07f](53de07f)) * **build:** parallelize fragmented backing reads ([#2872](#2872)) ([c7655a7](c7655a7)) * **clean-nfs-cache:** restore dirfd-relative statx ([#2766](#2766)) ([6bdbedb](6bdbedb)) * **header:** add V5 columnar varint header format ([#2847](#2847)) ([9dd931b](9dd931b)) * **header:** pack cached Header.Mapping into a compact form ([#2844](#2844)) ([7f0b13c](7f0b13c)) * **orchestrator:** add memfile dedup density threshold ([#2862](#2862)) ([7ccfa02](7ccfa02)) * **orchestrator:** avoid V3-ancestor header refresh ([#2999](#2999)) ([cb6aa0b](cb6aa0b)) * **orch:** metrics for dirty page throttling ([#2858](#2858)) ([d2aa554](d2aa554)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: e2b-release-please[bot] <298072688+e2b-release-please[bot]@users.noreply.github.com> Co-authored-by: Charlie Wyse <charlie.wyse@e2b.dev>

🤖 I have created a release *beep* *boop* --- ## 0.0.1 (2026-07-30) ### Features * **api:** add sandbox IAM workload token configuration ([13ddb3d](13ddb3d)) * **api:** add sandbox workload identity permission ([#3319](#3319)) ([13ddb3d](13ddb3d)) * **api:** SOCKS5 egress proxy on sandbox network config (BYOP) ([#2642](#2642)) ([1fc3820](1fc3820)) * **cfg:** add DISABLE_STARTUP_RECLAIM flag ([#3081](#3081)) ([7677ca6](7677ca6)) * **clickhouse:** implement multi-cluster fan-out for events and stats ([#2925](#2925)) ([39594c6](39594c6)) * dynamic sandbox log routing and ClickHouse-backed log reads ([#3236](#3236)) ([1b19a3b](1b19a3b)) * **envd:** give envd realtime IO priority, reset for user processes ([#2681](#2681)) ([f4bd1b2](f4bd1b2)) * **envd:** split collapse stats into real migrations vs already-huge ([#3021](#3021)) ([0d77614](0d77614)) * **envd:** support user-defined file metadata via xattrs ([#2732](#2732)) ([da8fbe4](da8fbe4)) * **featureflags:** support per-service context providers ([#3100](#3100)) ([65297c1](65297c1)) * freeze user cgroup across pause/resume to keep envd /init responsive ([#2688](#2688)) ([eceb741](eceb741)) * **metrics:** break down pause-snapshot latency by step ([#3426](#3426)) ([f551118](f551118)) * **metrics:** label pause telemetry by fs_only ([#3425](#3425)) ([4be33ba](4be33ba)) * **observability:** add kill_reason to sandbox.lifecycle.killed ([#2833](#2833)) ([e45418f](e45418f)) * **observability:** include kill_reason in kill-path structured logs ([#2846](#2846)) ([33c49f7](33c49f7)) * **orch:** add envd-version to LaunchDarkly sandbox context ([#3051](#3051)) ([37d3b92](37d3b92)) * **orch:** add less, nftables, iputils-ping, and jq to base provisioning ([#2736](#2736)) ([a1e010e](a1e010e)) * **orch:** collapse envd's heap into 2 MiB hugepages before pause to cut cold-resume faults ([#2997](#2997)) ([6677f73](6677f73)) * **orch:** debug a sandbox guest kernel with resume-build -gdb ([#3040](#3040)) ([37bb0dc](37bb0dc)) * **orch:** decouple warm resume from memfile dedup ([#3166](#3166)) ([77f25a0](77f25a0)) * **orch:** distro-aware template base-image provisioning ([#3411](#3411)) ([1abece1](1abece1)) * **orchestrator/cgroup:** list and destroy leaked sandbox cgroups ([#3086](#3086)) ([bce1d84](bce1d84)) * **orchestrator/nbd:** inspect and disconnect connected devices ([#3087](#3087)) ([4d47148](4d47148)) * **orchestrator/network:** list slot namespaces ([#3089](#3089)) ([c23dbc7](c23dbc7)) * **orchestrator/network:** list slot namespaces ([#3090](#3090)) ([fbfce25](fbfce25)) * **orchestrator:** add -force-reboot to resume-build to cold-boot memory-snaphsot builds ([#3208](#3208)) ([cf8f15b](cf8f15b)) * **orchestrator:** add allocated resource metrics for sandboxes ([#2943](#2943)) ([95cb6d3](95cb6d3)) * **orchestrator:** add dummy orchestrator binary for local API dev ([#2744](#2744)) ([ab56e25](ab56e25)) * **orchestrator:** add NetworkAssignHook for sandbox lifecycle extensions ([#3290](#3290)) ([3261963](3261963)) * **orchestrator:** add soft-delete marker label to the check metric ([#3144](#3144)) ([1ce64f8](1ce64f8)) * **orchestrator:** add v4HeaderForUncompressed FF bit ([#2669](#2669)) ([1f459ee](1f459ee)) * **orchestrator:** always include execution metrics in sandbox webhook events ([#2852](#2852)) ([440edfe](440edfe)) * **orchestrator:** classify envd-init by exit type ([#3139](#3139)) ([1e39a4f](1e39a4f)) * **orchestrator:** graceful sandbox drain on shutdown ([#3069](#3069)) ([6ce68e3](6ce68e3)) * **orchestrator:** graceful template-build drain on shutdown ([#3079](#3079)) ([1b3001c](1b3001c)) * **orchestrator:** improved read-path telemetry ([#3063](#3063)) ([bc3fe84](bc3fe84)) * **orchestrator:** LD-gated ClickHouse write fan-out feature flag ([#3152](#3152)) ([f046fcf](f046fcf)) * **orchestrator:** make build-reserved-disk-space-mb default 256MB ([#3065](#3065)) ([d473f98](d473f98)) * **orchestrator:** record upload compression metrics ([#2761](#2761)) ([9092e35](9092e35)) * **orchestrator:** report hugepage metrics to API ([#3182](#3182)) ([7735bae](7735bae)) * **orchestrator:** run startup reclaim on boot ([#3123](#3123)) ([79b838e](79b838e)) * **orchestrator:** single-instance flock on startup ([#3143](#3143)) ([1320d6e](1320d6e)) * **orchestrator:** soft-delete consumer enforcement for storage index ([#3034](#3034)) ([fbfc918](fbfc918)) * **orchestrator:** tag envd-init meters with start_type ([#3125](#3125)) ([4466b48](4466b48)) * **orchestrator:** track and report last status change timestamp ([#2980](#2980)) ([f79be77](f79be77)) * **orchestrator:** track sandbox lifecycles ([#2998](#2998)) ([057f20c](057f20c)) * **orchestrator:** write layer sizes (logical/mapped/diff) to object metadata ([#3122](#3122)) ([11869c0](11869c0)) * **orch:** harvest resume-prefetch trace on pause ([#3067](#3067)) ([97bd4a5](97bd4a5)) * **orch:** last-cycle memory prefetch on resume ([#3258](#3258)) ([ea94196](ea94196)) * **orch:** make resume-build -gdb work on real nodes + add copy-build -gdb ([#3108](#3108)) ([5385594](5385594)) * **orch:** opt-in DSCP marker for sandbox egress (SANDBOX_EGRESS_DSCP) ([#3039](#3039)) ([a98cf2c](a98cf2c)) * **orch:** per-start UFFD startup working-set metric ([#2960](#2960)) ([dc386b2](dc386b2)) * **orch:** premade NixOS base-image support ([#3412](#3412)) ([4bd42d2](4bd42d2)) * **orch:** record envd init duration histogram on failure with success attribute ([#2749](#2749)) ([afa7458](afa7458)) * **orch:** snapshot fragmentation metrics ([#2931](#2931)) ([842b007](842b007)) * per-team events TTL limit (tier + addons) ([#3181](#3181)) ([f76b2cb](f76b2cb)) * **shared:** add OTEL instrumentation to AWS S3 storage client ([#3172](#3172)) ([25b0fd1](25b0fd1)) * **storage:** per-role storage URLs, env-free storage library ([#3246](#3246)) ([fcbe909](fcbe909)) * **storage:** stamp provenance custom metadata on uploaded objects (incl. headers) ([#3033](#3033)) ([ba8604e](ba8604e)) * **storage:** write-through compressed templates to NFS on upload ([#2827](#2827)) ([57503c1](57503c1)) ### Bug Fixes * added api and orch ([#3454](#3454)) ([d56e0a8](d56e0a8)) * **block:** rephrase misleading error message in pwritevAll ([#2816](#2816)) ([1555f1b](1555f1b)) * **cache:** use 512-byte units for stat.Blocks in FileSize ([#2949](#2949)) ([0f632a9](0f632a9)) * **clean-nfs-cache:** exclude zombies from delete_age ([#3191](#3191)) ([3fa2aeb](3fa2aeb)) * **compression:** correctness findings from compression audit ([#2803](#2803)) ([d21a6a9](d21a6a9)) * **copy-build:** resolve compression suffix for build data files ([#2859](#2859)) ([8966f7e](8966f7e)) * correct 3 CVES ([#3218](#3218)) ([076823b](076823b)) * **envd:** stop freezing socat cgroup across pause/resume ([#2923](#2923)) ([8b6f2b9](8b6f2b9)) * **inspect-build:** adapt validate to new Chunker upstream API ([#2989](#2989)) ([2e0d3da](2e0d3da)) * **nbd:** adjust status poll sleep from 100ns to 100µs ([02bf51b](02bf51b)) * **nbd:** change NBD status poll sleep from 100ns to 100µs to avoid useless busy spinning ([#2884](#2884)) ([02bf51b](02bf51b)) * **nfsproxy:** deflake TestRoundTrip EADDRINUSE ([#2987](#2987)) ([55f4d18](55f4d18)) * **orch:** denormalize upload metric file type ([#2865](#2865)) ([b1646ca](b1646ca)) * **orch:** disable the chronyd seccomp filter on Alpine when using PHC ([#3453](#3453)) ([dfa9764](dfa9764)) * **orchestrator:** anchor rsync CWD to root in template file copy ([#2835](#2835)) ([7160db9](7160db9)) * **orchestrator:** atomically replace metadata ([#3321](#3321)) ([0c4ad6b](0c4ad6b)) * **orchestrator:** avoid serializing upload headers twice ([#2762](#2762)) ([9b7b149](9b7b149)) * **orchestrator:** chunk readiness bug in P2P->compressed ([#3185](#3185)) ([74a6e5b](74a6e5b)) * **orchestrator:** deschedule flaky eviction-loop race in TestDiffSto… ([#3173](#3173)) ([88ff17c](88ff17c)) * **orchestrator:** discard poisoned nftables conn on firewall errors ([#3008](#3008)) ([03f10e0](03f10e0)) * **orchestrator:** drop stale pre-init logs ([#3297](#3297)) ([8ec4be5](8ec4be5)) * **orchestrator:** emit compression ratios as fractions, not BP ([#2772](#2772)) ([866f4c1](866f4c1)) * **orchestrator:** export dirty-page stall counter from process start ([#2992](#2992)) ([badc8ad](badc8ad)) * **orchestrator:** harden Firecracker process shutdown ([#2996](#2996)) ([df662e7](df662e7)) * **orchestrator:** harden shutdown network cleanup ([#3000](#3000)) ([de2f391](de2f391)) * **orchestrator:** implement Docker COPY merge semantics in template builds ([#3283](#3283)) ([9174104](9174104)) * **orchestrator:** keep dedup empty-pages telemetry scan-only ([#2991](#2991)) ([35d0832](35d0832)) * **orchestrator:** let build-cache threshold flag raise above its fal… ([#3175](#3175)) ([06393c3](06393c3)) * **orchestrator:** log missing egress proxy in startup reclaim instead of defaulting silently ([#3116](#3116)) ([6ca3163](6ca3163)) * **orchestrator:** make copy-build handle filesystem-only snapshots ([#3299](#3299)) ([62add04](62add04)) * **orchestrator:** measure ext4 free space from block groups ([#3282](#3282)) ([f18f05f](f18f05f)) * **orchestrator:** normalize upload metric file labels ([#2767](#2767)) ([6dec8b3](6dec8b3)) * **orchestrator:** order egress config/firewall updates to close BYOP enable race ([#3313](#3313)) ([7faa59e](7faa59e)) * **orchestrator:** order envd.service after local-fs.target ([#3043](#3043)) ([ea2663e](ea2663e)) * **orchestrator:** order envd.service after systemd-tmpfiles-setup ([#3130](#3130)) ([9481811](9481811)) * **orchestrator:** pause upload retain retry ([#2993](#2993)) ([4f81799](4f81799)) * **orchestrator:** pin tap device host-side MAC address ([#3271](#3271)) ([3c786ba](3c786ba)) * **orchestrator:** pin UFFD copy source buffers ([#2745](#2745)) ([837fa91](837fa91)) * **orchestrator:** preserve full ENV value across stdout chunks ([#2740](#2740)) ([4822e6d](4822e6d)) * **orchestrator:** read V3 ancestors as uncompressed instead of failing ([#2994](#2994)) ([c479dd3](c479dd3)) * **orchestrator:** reject standby while draining ([#3325](#3325)) ([475a7ee](475a7ee)) * **orchestrator:** report real V4 header compression ratio ([#2771](#2771)) ([ecd344e](ecd344e)) * **orchestrator:** resolve remaining P2P/compression/V5 issues ([#3015](#3015)) ([1e4379e](1e4379e)) * **orchestrator:** sanitize OCI pull errors ([#3096](#3096)) ([a3af6c0](a3af6c0)) * **orchestrator:** scope rootfs hash to provision default ([#3129](#3129)) ([475f955](475f955)) * **orchestrator:** stop Checks health-loop leaking ([#2739](#2739)) ([17e6e60](17e6e60)) * **orchestrator:** survive SIGBUS from failing disks under mmap'd caches ([#3385](#3385)) ([8694d08](8694d08)) * **orchestrator:** tolerate missing header for legacy templates ([#3026](#3026)) ([8a44bfe](8a44bfe)) * **orch:** fall back to ID_LIKE with a warning instead of rejecting ([#3459](#3459)) ([73399b3](73399b3)) * **orch:** prevent NBD dispatch read-loop stall on WRITE_ZEROES (behind flag) ([#3048](#3048)) ([efd3d4d](efd3d4d)) * **orch:** split scheduling base build id per artifact ([#2920](#2920)) ([3e35a2a](3e35a2a)) * **orch:** validate copy-build -gdb buckets before the snapshot copy ([#3446](#3446)) ([9be382f](9be382f)) * **shared:** never report a failed envd command stream as success ([#3281](#3281)) ([69c06b6](69c06b6)) * **storage:** compression upload & cache correctness fixes ([#3231](#3231)) ([980748f](980748f)) * **storage:** don't assume V4+ ancestor gaps are uncompressed ([#3447](#3447)) ([f828d12](f828d12)) * **uffd:** dedupe deferred page faults ([#2864](#2864)) ([9680a41](9680a41)) * WrapContextAsUserError should not misclassify internal timeouts as user cancellations ([#3155](#3155)) ([8f83959](8f83959)) ### Performance Improvements * **build:** cache resolved Diff per BuildId within File.ReadAt ([#2838](#2838)) ([53de07f](53de07f)) * **build:** parallelize fragmented backing reads ([#2872](#2872)) ([c7655a7](c7655a7)) * **clean-nfs-cache:** restore dirfd-relative statx ([#2766](#2766)) ([6bdbedb](6bdbedb)) * **header:** add V5 columnar varint header format ([#2847](#2847)) ([9dd931b](9dd931b)) * **header:** pack cached Header.Mapping into a compact form ([#2844](#2844)) ([7f0b13c](7f0b13c)) * **orchestrator:** add memfile dedup density threshold ([#2862](#2862)) ([7ccfa02](7ccfa02)) * **orchestrator:** avoid V3-ancestor header refresh ([#2999](#2999)) ([cb6aa0b](cb6aa0b)) * **orch:** metrics for dirty page throttling ([#2858](#2858)) ([d2aa554](d2aa554)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: e2b-release-please[bot] <298072688+e2b-release-please[bot]@users.noreply.github.com> Co-authored-by: Charlie Wyse <charlie.wyse@e2b.dev>

rguliyev requested a review from tvi June 4, 2026 02:02

rguliyev requested review from ValentaTomas, dobrac and jakubno as code owners June 4, 2026 02:02

cla-bot Bot added the cla-signed label Jun 4, 2026

gemini-code-assist Bot reviewed Jun 4, 2026

View reviewed changes

Comment thread packages/clickhouse/pkg/hoststats/hoststats.go

Comment thread packages/orchestrator/pkg/events/events.go

Comment thread packages/clickhouse/pkg/events/delivery.go

Comment thread packages/clickhouse/pkg/hoststats/delivery.go

claude Bot reviewed Jun 4, 2026

View reviewed changes

Comment thread packages/clickhouse/pkg/clickhouse.go

chatgpt-codex-connector Bot reviewed Jun 5, 2026

View reviewed changes

Comment thread iac/provider-gcp/nomad/main.tf Outdated

chatgpt-codex-connector Bot reviewed Jun 5, 2026

View reviewed changes

Comment thread packages/clickhouse/pkg/hoststats/hoststats.go

Comment thread iac/provider-gcp/nomad/main.tf Outdated

feat(clickhouse): implement multi-cluster fan-out for events and stats

4c6c89f

Adds support for CLICKHOUSE_CONNECTION_STRINGS to allow concurrent writing to multiple clusters. Includes deduplication logic, isolated batchers for additional endpoints, and resource cleanup fixes.

rguliyev force-pushed the add-multi-write-support-eng-4250 branch from 711b593 to 4c6c89f Compare June 5, 2026 22:55

tvi approved these changes Jun 5, 2026

View reviewed changes

rguliyev enabled auto-merge (squash) June 5, 2026 22:59

rguliyev merged commit 39594c6 into main Jun 5, 2026
52 checks passed

rguliyev deleted the add-multi-write-support-eng-4250 branch June 5, 2026 23:18

e2b-release-please Bot mentioned this pull request Jul 7, 2026

chore(main): release clickhouse-migrator 1.0.0 #3212

Merged

This was referenced Jul 10, 2026

chore(main): release clickhouse-migrator 1.0.0 #3260

Closed

chore(main): release clickhouse-migrator 0.0.1 #3263

Merged

e2b-release-please Bot mentioned this pull request Jul 30, 2026

chore(main): release orchestrator 0.0.1 #3470

Merged

This was referenced Aug 3, 2026

chore(main): release orchestrator 0.1.0 #3506

Closed

chore(main): release clickhouse-migrator 0.2.0 #3525

Closed

Navigation Menu

feat(clickhouse): implement multi-cluster fan-out for events and stats - #2925

feat(clickhouse): implement multi-cluster fan-out for events and stats#2925
rguliyev merged 1 commit into
mainfrom
add-multi-write-support-eng-4250

rguliyev commented Jun 4, 2026

Uh oh!

linear-code Bot commented Jun 4, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Jun 4, 2026 •

edited

Loading

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

rguliyev commented Jun 4, 2026

Uh oh!

linear-code Bot commented Jun 4, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 4 Tests Failed:

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov Bot commented Jun 4, 2026 •

edited

Loading