Skip to content

feat(orch): per-start UFFD startup working-set metric - #2960

Merged
kalyazin merged 5 commits into
mainfrom
kalyazin/uffd-startup-metric
Jun 10, 2026
Merged

kalyazin merged 5 commits into
mainfrom
kalyazin/uffd-startup-metric

Conversation

@kalyazin

@kalyazin kalyazin commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Why

No fleet-wide signal exists for how many pages / how much data a sandbox needs
to start
(until envd init returns). uffd.serve can't answer it: it's
lifetime-cumulative (not bounded to startup) and has no per-sandbox dimension
(so Mimir gives a fleet rate, never a per-sandbox p50/p95).

What

A metric recorded once per start at the envd-init boundary, with that
start's serve-stats snapshot as the value. One observation per start in a native
histogram → histogram_quantile gives the across-sandbox p50/p95 directly.

Metric Meaning
orchestrator.sandbox.uffd.startup.pages Demand-fault pages resolved to reach envd init — the headline "pages to start".
orchestrator.sandbox.uffd.startup.source_pages Subset pulled from the source (GCS/NFS) — the pages that cost a real read. The rest were zero-filled or already resident (prefetch hits).
orchestrator.sandbox.uffd.startup.bytes Bytes faulted into the guest — the "how much data" answer.

Attributes: start_type ∈ {create, resume}, success ∈ {true, false}. No
high-cardinality labels. Counts demand faults only (resolved, no
double-counting); prefaulted pages bypass the serve loop

kalyazin added 3 commits June 9, 2026 15:48
Add per-handler cumulative counters for demand faults served — pages
resolved, the subset pulled from the source, and bytes installed — and a
ServeStats() snapshot to read them without blocking. Surface it through
the MemoryBackend interface, with a passthrough in Uffd and a zero stub
in NoopMemory.

These mirror the orchestrator.sandbox.uffd.serve metric but as a
point-in-time snapshot, so a caller can later sample how many pages a
guest needed to start. No behaviour change.

Signed-off-by: Nikita Kalyazin <nikita.kalyazin@e2b.dev>
There is no fleet-wide signal for how many pages or how much data a guest
needs to start (until envd init returns). The uffd.serve metric counts
demand faults over the whole sandbox lifetime and has no per-sandbox
dimension, so it can give a fleet rate but not a per-start distribution.

Record orchestrator.sandbox.uffd.startup.{pages,source_pages,bytes} once
per start, at the envd-init boundary, with the start's serve-stats
snapshot as the sample value. Because the observation unit is one start,
histogram_quantile yields the per-sandbox avg/p50/p95. Attributes
start_type (create/resume) and success let slow or failed starts be
correlated with page volume. source_pages isolates the data actually
pulled from the source (e.g. GCS) and doubles as a prefetch-hit signal.

Signed-off-by: Nikita Kalyazin <nikita.kalyazin@e2b.dev>
Assert ServeStats() folds a mix of finished serve attempts correctly:
resolved faults (installed/present) count as needed pages, the source
subset and installed bytes are tracked, and deferred/errored attempts are
excluded so a deferred fault is not double-counted when re-served.

Signed-off-by: Nikita Kalyazin <nikita.kalyazin@e2b.dev>
@cla-bot cla-bot Bot added the cla-signed label Jun 9, 2026
@cursor

cursor Bot commented Jun 9, 2026 •

Copy link
Copy Markdown

PR Summary

Low Risk
Observability-only: atomic counters on the existing serve hot path and one histogram sample per sandbox start; no auth, data, or resume logic changes.

Overview
Adds once-per-sandbox-start observability for how much guest memory is materialized via UFFD demand faults before envd init finishes, so fleet dashboards can compute per-start p50/p95 instead of only lifetime serve rates.

The UFFD handler now keeps cumulative ServeStats (resolved pages, source-backed pages, installed bytes) updated on each finished serve; WaitForEnvd samples that snapshot on the first call only (sync.Once), tagged with start_type (create vs resume) and success, into three histograms: orchestrator.sandbox.uffd.startup.pages, .source_pages, and .bytes. Later WaitForEnvd calls on the same sandbox (e.g. envd restart during template build) are excluded so post-startup faults do not inflate the startup working set.

Reviewed by Cursor Bugbot for commit 0374cbb. Bugbot is set up for automated code reviews on this repo. Configure here.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

@codecov

codecov Bot commented Jun 9, 2026 •

Copy link
Copy Markdown

❌ 5 Tests Failed:

Tests completed Failed Passed Skipped
2783 5 2778 5
View the top 2 failed test(s) by shortest run time
github.com/e2b-dev/infra/tests/integration/internal/tests/api/metrics::TestSandboxMetrics
Stack Traces | 6.95s run time
=== RUN   TestSandboxMetrics
=== PAUSE TestSandboxMetrics
=== CONT  TestSandboxMetrics
    sandbox_metrics_test.go:45: 
        	Error Trace:	.../api/metrics/sandbox_metrics_test.go:45
        	Error:      	Should NOT be empty, but was 0
        	Test:       	TestSandboxMetrics
--- FAIL: TestSandboxMetrics (6.95s)
github.com/e2b-dev/infra/tests/integration/internal/tests/orchestrator::TestSandboxObjectNotFound
Stack Traces | 50.2s run time
=== RUN   TestSandboxObjectNotFound
=== PAUSE TestSandboxObjectNotFound
=== CONT  TestSandboxObjectNotFound
    sandbox_object_not_found_test.go:59: sandbox creation failed due to resource exhaustion, retrying
    sandbox_object_not_found_test.go:59: sandbox creation failed due to resource exhaustion, retrying
Executing command bash in sandbox in8h27w4febjiiyew609j (user: root)
    sandbox_object_not_found_test.go:59: sandbox creation failed due to resource exhaustion, retrying
Executing command bash in sandbox i3c3gvd08wpscirq9nu5i (user: root)
    sandbox_object_not_found_test.go:59: sandbox creation failed due to resource exhaustion, retrying
    sandbox_object_not_found_test.go:59: sandbox creation failed due to resource exhaustion, retrying
    sandbox_object_not_found_test.go:59: sandbox creation failed due to resource exhaustion, retrying
    sandbox_object_not_found_test.go:59: sandbox creation failed due to resource exhaustion, retrying
Executing command bash in sandbox iz6bcjap6tkyb0o7q1xui (user: root)
    sandbox_object_not_found_test.go:59: sandbox creation failed due to resource exhaustion, retrying
    sandbox_object_not_found_test.go:59: sandbox creation failed due to resource exhaustion, retrying
    sandbox_object_not_found_test.go:59: sandbox creation failed due to resource exhaustion, retrying
    sandbox_object_not_found_test.go:70: failed to create sandbox after 10 retries
--- FAIL: TestSandboxObjectNotFound (50.19s)
View the full list of 3 ❄️ flaky test(s)
github.com/e2b-dev/infra/tests/integration/internal/tests/orchestrator::TestSandboxMemoryIntegrity

Flake rate in main: 53.11% (Passed 989 times, Failed 1120 times)

Stack Traces | 68.6s run time
=== RUN   TestSandboxMemoryIntegrity
=== PAUSE TestSandboxMemoryIntegrity
=== CONT  TestSandboxMemoryIntegrity
    sandbox_memory_integrity_test.go:27: Build completed successfully
--- FAIL: TestSandboxMemoryIntegrity (68.63s)
github.com/e2b-dev/infra/tests/integration/internal/tests/orchestrator::TestSandboxMemoryIntegrity/tmpfs_hash

Flake rate in main: 53.11% (Passed 979 times, Failed 1109 times)

Stack Traces | 204s run time
=== RUN   TestSandboxMemoryIntegrity/tmpfs_hash
=== PAUSE TestSandboxMemoryIntegrity/tmpfs_hash
=== CONT  TestSandboxMemoryIntegrity/tmpfs_hash
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{start:{pid:1259}}
Executing command bash in sandbox i8cghx6s90ijmxfis6aw5 (user: root)
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stdout:"Total memory: 985 MB\nUsed memory before tmpfs mount: 187 MB\nFree memory before tmpfs mount: 797 MB\nMemory to use in integrity test (60% of free, min 64MB): 478 MB\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"478+0 records in\n478+0 records out\n501219328 bytes (501 MB, 478 MiB) copied, 2.42207 s, 207 MB/s\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\tCommand bei"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"ng timed: \"dd if=/dev/urandom of=/mnt/testfile bs=1M count=478\"\n\tUser time (seconds): 0.00\n\tSystem "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"time (se"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"conds): 2.38\n\tPercent of CPU this job got: "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"98%\n\tElap"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"sed ("}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"wall "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"cloc"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"k) t"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"ime "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"(h:m"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"m:ss"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" or "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"m:ss"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"): 0:02.43"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\tAverage shared t"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"ext s"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"ize"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" (k"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"byte"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"s):"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" 0\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\tAv"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"era"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"ge"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" un"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"sha"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"red"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" data size (kbytes): 0\n\tAverage stack siz"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"e (k"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"byt"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"es)"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:": 0"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\n\tA"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"ve"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"rag"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"e t"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"ota"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"l s"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"ize"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" (k"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"bytes): 0\n\tMaximum resident set size "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"(kby"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"tes"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"): "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"2720\n\t"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"Ave"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"rag"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"e r"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"esi"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"den"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"t s"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"et "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"siz"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"e ("}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"kbytes): 0\n\tMajor (requiring I/O) page faults: 2\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\tMino"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"r ("}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"re"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"cla"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"iming a frame) page faults: 345\n\tVoluntary contex"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"t sw"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"itc"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"he"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"s: "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"3\n\tInvoluntary context switches: 19\n\tSwaps: 0\n\tFile system inputs: 176\n\tFile system outputs: 0\n\tSocket messages sent: 0\n\tSocket messages received: 0\n\tSignals delivered: 0\n\tPage size (bytes): 4096\n\tExit status: 0\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stdout:"Used memory after tmpfs mount and file fill: 671 MB\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{end:{exited:true status:"exit status 0"}}
    sandbox_memory_integrity_test.go:70: Command [bash] completed successfully in sandbox idqeax6x4g5mpbnmmjz1w
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{start:{pid:1275}}
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{data:{stdout:"ec9b00faa7225fa39dd25bba7a769291ca3ecf224ada766530faf10bbee5f86a\n"}}
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{end:{exited:true status:"exit status 0"}}
    sandbox_memory_integrity_test.go:80: Command [bash] completed successfully in sandbox idqeax6x4g5mpbnmmjz1w
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{start:{pid:1278}}
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
Executing command bash in sandbox idqeax6x4g5mpbnmmjz1w (user: root)
    sandbox_memory_integrity_test.go:110: 
        	Error Trace:	.../tests/orchestrator/sandbox_memory_integrity_test.go:81
        	            				.../hostedtoolcache/go/1.26.3.../src/runtime/asm_amd64.s:1771
        	Error:      	Received unexpected error:
        	            	failed to execute command bash in sandbox idqeax6x4g5mpbnmmjz1w: unavailable: HTTP status 502 Bad Gateway
    sandbox_memory_integrity_test.go:110: 
        	Error Trace:	.../tests/orchestrator/sandbox_memory_integrity_test.go:78
        	            				.../tests/orchestrator/sandbox_memory_integrity_test.go:110
        	Error:      	Condition never satisfied
        	Test:       	TestSandboxMemoryIntegrity/tmpfs_hash
--- FAIL: TestSandboxMemoryIntegrity/tmpfs_hash (204.10s)
github.com/e2b-dev/infra/tests/integration/internal/tests/proxies::TestSandboxAutoResumeViaProxy

Flake rate in main: 38.56% (Passed 983 times, Failed 617 times)

Stack Traces | 19.3s run time
=== RUN   TestSandboxAutoResumeViaProxy
=== PAUSE TestSandboxAutoResumeViaProxy
=== CONT  TestSandboxAutoResumeViaProxy
    auto_resume_test.go:97: [Status code: 502] Response body: {"sandboxId":"i22p9gsdlgp4301403kza","message":"The sandbox is running but port is not open","port":8000,"code":502}
Executing command cat in sandbox iqq3lv2g6ec103lzmy82v (user: root)
    auto_resume_test.go:116: 
        	Error Trace:	.../tests/proxies/auto_resume_test.go:116
        	Error:      	Received unexpected error:
        	            	Get "http://localhost:3002": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
        	Test:       	TestSandboxAutoResumeViaProxy
--- FAIL: TestSandboxAutoResumeViaProxy (19.28s)

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 51dd901. Configure here.

Comment thread packages/orchestrator/pkg/sandbox/sandbox.go Outdated
ServeStats() is lifetime-cumulative on the UFFD handler, and WaitForEnvd
can run more than once on the same handler: a template build swaps the
envd binary and restarts the service mid-build, then waits again. That
second wait would emit an orchestrator.sandbox.uffd.startup.* sample
holding the cumulative serve counts — startup pages plus all the
build-step demand faults since — rather than that init's working set,
polluting the distribution.

Guard the recording with a sync.Once so it fires only on the first wait,
which is the actual sandbox start and the point at which the cumulative
counters still equal the startup counts.

Signed-off-by: Nikita Kalyazin <nikita.kalyazin@e2b.dev>
@kalyazin
kalyazin marked this pull request as ready for review June 9, 2026 16:29
@kalyazin
kalyazin enabled auto-merge (squash) June 10, 2026 17:46
@kalyazin
kalyazin merged commit dc386b2 into main Jun 10, 2026
53 checks passed
@kalyazin
kalyazin deleted the kalyazin/uffd-startup-metric branch June 10, 2026 18:08
Piwriw pushed a commit to Piwriw/infra that referenced this pull request Jun 12, 2026
## Why

No fleet-wide signal exists for **how many pages / how much data a
sandbox needs
to start** (until envd init returns). `uffd.serve` can't answer it: it's
lifetime-cumulative (not bounded to startup) and has no per-sandbox
dimension
(so Mimir gives a fleet rate, never a per-sandbox p50/p95).

## What

A metric recorded **once per start** at the envd-init boundary, with
that
start's serve-stats snapshot as the value. One observation per start in
a native
histogram → `histogram_quantile` gives the **across-sandbox** p50/p95
directly.

| Metric | Meaning |
|---|---|
| `orchestrator.sandbox.uffd.startup.pages` | Demand-fault pages
resolved to reach envd init — the headline "pages to start". |
| `orchestrator.sandbox.uffd.startup.source_pages` | Subset pulled from
the source (GCS/NFS) — the pages that cost a real read. The rest were
zero-filled or already resident (prefetch hits). |
| `orchestrator.sandbox.uffd.startup.bytes` | Bytes faulted into the
guest — the "how much data" answer. |

Attributes: `start_type` ∈ `{create, resume}`, `success` ∈ `{true,
false}`. No
high-cardinality labels. Counts demand faults only (resolved, no
double-counting); prefaulted pages bypass the serve loop

---------

Signed-off-by: Nikita Kalyazin <nikita.kalyazin@e2b.dev>
charlie-e2b added a commit that referenced this pull request Jul 30, 2026
🤖 I have created a release *beep* *boop*
---


## 0.0.1 (2026-07-30)


### Features

* **api:** add sandbox IAM workload token configuration
([13ddb3d](13ddb3d))
* **api:** add sandbox workload identity permission
([#3319](#3319))
([13ddb3d](13ddb3d))
* **api:** SOCKS5 egress proxy on sandbox network config (BYOP)
([#2642](#2642))
([1fc3820](1fc3820))
* **cfg:** add DISABLE_STARTUP_RECLAIM flag
([#3081](#3081))
([7677ca6](7677ca6))
* **clickhouse:** implement multi-cluster fan-out for events and stats
([#2925](#2925))
([39594c6](39594c6))
* dynamic sandbox log routing and ClickHouse-backed log reads
([#3236](#3236))
([1b19a3b](1b19a3b))
* **envd:** give envd realtime IO priority, reset for user processes
([#2681](#2681))
([f4bd1b2](f4bd1b2))
* **envd:** split collapse stats into real migrations vs already-huge
([#3021](#3021))
([0d77614](0d77614))
* **envd:** support user-defined file metadata via xattrs
([#2732](#2732))
([da8fbe4](da8fbe4))
* **featureflags:** support per-service context providers
([#3100](#3100))
([65297c1](65297c1))
* freeze user cgroup across pause/resume to keep envd /init responsive
([#2688](#2688))
([eceb741](eceb741))
* **metrics:** break down pause-snapshot latency by step
([#3426](#3426))
([657559e](657559e))
* **metrics:** label pause telemetry by fs_only
([#3425](#3425))
([411b63e](411b63e))
* **observability:** add kill_reason to sandbox.lifecycle.killed
([#2833](#2833))
([e45418f](e45418f))
* **observability:** include kill_reason in kill-path structured logs
([#2846](#2846))
([33c49f7](33c49f7))
* **orch:** add envd-version to LaunchDarkly sandbox context
([#3051](#3051))
([37d3b92](37d3b92))
* **orch:** add less, nftables, iputils-ping, and jq to base
provisioning ([#2736](#2736))
([a1e010e](a1e010e))
* **orch:** collapse envd's heap into 2 MiB hugepages before pause to
cut cold-resume faults
([#2997](#2997))
([6677f73](6677f73))
* **orch:** debug a sandbox guest kernel with resume-build -gdb
([#3040](#3040))
([37bb0dc](37bb0dc))
* **orch:** decouple warm resume from memfile dedup
([#3166](#3166))
([77f25a0](77f25a0))
* **orch:** distro-aware template base-image provisioning
([#3411](#3411))
([f8c7b5b](f8c7b5b))
* **orchestrator/cgroup:** list and destroy leaked sandbox cgroups
([#3086](#3086))
([bce1d84](bce1d84))
* **orchestrator/nbd:** inspect and disconnect connected devices
([#3087](#3087))
([4d47148](4d47148))
* **orchestrator/network:** list slot namespaces
([#3089](#3089))
([c23dbc7](c23dbc7))
* **orchestrator/network:** list slot namespaces
([#3090](#3090))
([fbfce25](fbfce25))
* **orchestrator:** add -force-reboot to resume-build to cold-boot
memory-snaphsot builds
([#3208](#3208))
([cf8f15b](cf8f15b))
* **orchestrator:** add allocated resource metrics for sandboxes
([#2943](#2943))
([95cb6d3](95cb6d3))
* **orchestrator:** add dummy orchestrator binary for local API dev
([#2744](#2744))
([ab56e25](ab56e25))
* **orchestrator:** add NetworkAssignHook for sandbox lifecycle
extensions ([#3290](#3290))
([3261963](3261963))
* **orchestrator:** add soft-delete marker label to the check metric
([#3144](#3144))
([1ce64f8](1ce64f8))
* **orchestrator:** add v4HeaderForUncompressed FF bit
([#2669](#2669))
([1f459ee](1f459ee))
* **orchestrator:** always include execution metrics in sandbox webhook
events ([#2852](#2852))
([440edfe](440edfe))
* **orchestrator:** classify envd-init by exit type
([#3139](#3139))
([1e39a4f](1e39a4f))
* **orchestrator:** graceful sandbox drain on shutdown
([#3069](#3069))
([6ce68e3](6ce68e3))
* **orchestrator:** graceful template-build drain on shutdown
([#3079](#3079))
([1b3001c](1b3001c))
* **orchestrator:** improved read-path telemetry
([#3063](#3063))
([bc3fe84](bc3fe84))
* **orchestrator:** LD-gated ClickHouse write fan-out feature flag
([#3152](#3152))
([f046fcf](f046fcf))
* **orchestrator:** make build-reserved-disk-space-mb default 256MB
([#3065](#3065))
([d473f98](d473f98))
* **orchestrator:** record upload compression metrics
([#2761](#2761))
([9092e35](9092e35))
* **orchestrator:** report hugepage metrics to API
([#3182](#3182))
([7735bae](7735bae))
* **orchestrator:** run startup reclaim on boot
([#3123](#3123))
([79b838e](79b838e))
* **orchestrator:** single-instance flock on startup
([#3143](#3143))
([1320d6e](1320d6e))
* **orchestrator:** soft-delete consumer enforcement for storage index
([#3034](#3034))
([fbfc918](fbfc918))
* **orchestrator:** tag envd-init meters with start_type
([#3125](#3125))
([4466b48](4466b48))
* **orchestrator:** track and report last status change timestamp
([#2980](#2980))
([f79be77](f79be77))
* **orchestrator:** track sandbox lifecycles
([#2998](#2998))
([057f20c](057f20c))
* **orchestrator:** write layer sizes (logical/mapped/diff) to object
metadata ([#3122](#3122))
([11869c0](11869c0))
* **orch:** harvest resume-prefetch trace on pause
([#3067](#3067))
([97bd4a5](97bd4a5))
* **orch:** last-cycle memory prefetch on resume
([#3258](#3258))
([b22e820](b22e820))
* **orch:** make resume-build -gdb work on real nodes + add copy-build
-gdb ([#3108](#3108))
([c684bd2](c684bd2))
* **orch:** opt-in DSCP marker for sandbox egress (SANDBOX_EGRESS_DSCP)
([#3039](#3039))
([a98cf2c](a98cf2c))
* **orch:** per-start UFFD startup working-set metric
([#2960](#2960))
([dc386b2](dc386b2))
* **orch:** premade NixOS base-image support
([#3412](#3412))
([776ba39](776ba39))
* **orch:** record envd init duration histogram on failure with success
attribute ([#2749](#2749))
([afa7458](afa7458))
* **orch:** snapshot fragmentation metrics
([#2931](#2931))
([842b007](842b007))
* per-team events TTL limit (tier + addons)
([#3181](#3181))
([f76b2cb](f76b2cb))
* **shared:** add OTEL instrumentation to AWS S3 storage client
([#3172](#3172))
([25b0fd1](25b0fd1))
* **storage:** per-role storage URLs, env-free storage library
([#3246](#3246))
([fcbe909](fcbe909))
* **storage:** stamp provenance custom metadata on uploaded objects
(incl. headers) ([#3033](#3033))
([ba8604e](ba8604e))
* **storage:** write-through compressed templates to NFS on upload
([#2827](#2827))
([57503c1](57503c1))


### Bug Fixes

* added api and orch
([#3454](#3454))
([fda5e45](fda5e45))
* **block:** rephrase misleading error message in pwritevAll
([#2816](#2816))
([1555f1b](1555f1b))
* **cache:** use 512-byte units for stat.Blocks in FileSize
([#2949](#2949))
([0f632a9](0f632a9))
* **clean-nfs-cache:** exclude zombies from delete_age
([#3191](#3191))
([3fa2aeb](3fa2aeb))
* **compression:** correctness findings from compression audit
([#2803](#2803))
([d21a6a9](d21a6a9))
* **copy-build:** resolve compression suffix for build data files
([#2859](#2859))
([8966f7e](8966f7e))
* correct 3 CVES ([#3218](#3218))
([076823b](076823b))
* **envd:** stop freezing socat cgroup across pause/resume
([#2923](#2923))
([8b6f2b9](8b6f2b9))
* **inspect-build:** adapt validate to new Chunker upstream API
([#2989](#2989))
([2e0d3da](2e0d3da))
* **nbd:** adjust status poll sleep from 100ns to 100µs
([02bf51b](02bf51b))
* **nbd:** change NBD status poll sleep from 100ns to 100µs to avoid
useless busy spinning
([#2884](#2884))
([02bf51b](02bf51b))
* **nfsproxy:** deflake TestRoundTrip EADDRINUSE
([#2987](#2987))
([55f4d18](55f4d18))
* **orch:** denormalize upload metric file type
([#2865](#2865))
([b1646ca](b1646ca))
* **orch:** disable the chronyd seccomp filter on Alpine when using PHC
([#3453](#3453))
([e58af28](e58af28))
* **orchestrator:** anchor rsync CWD to root in template file copy
([#2835](#2835))
([7160db9](7160db9))
* **orchestrator:** atomically replace metadata
([#3321](#3321))
([0c4ad6b](0c4ad6b))
* **orchestrator:** avoid serializing upload headers twice
([#2762](#2762))
([9b7b149](9b7b149))
* **orchestrator:** chunk readiness bug in P2P-&gt;compressed
([#3185](#3185))
([74a6e5b](74a6e5b))
* **orchestrator:** deschedule flaky eviction-loop race in TestDiffSto…
([#3173](#3173))
([88ff17c](88ff17c))
* **orchestrator:** discard poisoned nftables conn on firewall errors
([#3008](#3008))
([03f10e0](03f10e0))
* **orchestrator:** drop stale pre-init logs
([#3297](#3297))
([8ec4be5](8ec4be5))
* **orchestrator:** emit compression ratios as fractions, not BP
([#2772](#2772))
([866f4c1](866f4c1))
* **orchestrator:** export dirty-page stall counter from process start
([#2992](#2992))
([badc8ad](badc8ad))
* **orchestrator:** harden Firecracker process shutdown
([#2996](#2996))
([df662e7](df662e7))
* **orchestrator:** harden shutdown network cleanup
([#3000](#3000))
([de2f391](de2f391))
* **orchestrator:** implement Docker COPY merge semantics in template
builds ([#3283](#3283))
([9174104](9174104))
* **orchestrator:** keep dedup empty-pages telemetry scan-only
([#2991](#2991))
([35d0832](35d0832))
* **orchestrator:** let build-cache threshold flag raise above its fal…
([#3175](#3175))
([06393c3](06393c3))
* **orchestrator:** log missing egress proxy in startup reclaim instead
of defaulting silently
([#3116](#3116))
([6ca3163](6ca3163))
* **orchestrator:** make copy-build handle filesystem-only snapshots
([#3299](#3299))
([62add04](62add04))
* **orchestrator:** measure ext4 free space from block groups
([#3282](#3282))
([f18f05f](f18f05f))
* **orchestrator:** normalize upload metric file labels
([#2767](#2767))
([6dec8b3](6dec8b3))
* **orchestrator:** order egress config/firewall updates to close BYOP
enable race ([#3313](#3313))
([7faa59e](7faa59e))
* **orchestrator:** order envd.service after local-fs.target
([#3043](#3043))
([ea2663e](ea2663e))
* **orchestrator:** order envd.service after systemd-tmpfiles-setup
([#3130](#3130))
([9481811](9481811))
* **orchestrator:** pause upload retain retry
([#2993](#2993))
([4f81799](4f81799))
* **orchestrator:** pin tap device host-side MAC address
([#3271](#3271))
([3c786ba](3c786ba))
* **orchestrator:** pin UFFD copy source buffers
([#2745](#2745))
([837fa91](837fa91))
* **orchestrator:** preserve full ENV value across stdout chunks
([#2740](#2740))
([4822e6d](4822e6d))
* **orchestrator:** read V3 ancestors as uncompressed instead of failing
([#2994](#2994))
([c479dd3](c479dd3))
* **orchestrator:** reject standby while draining
([#3325](#3325))
([475a7ee](475a7ee))
* **orchestrator:** report real V4 header compression ratio
([#2771](#2771))
([ecd344e](ecd344e))
* **orchestrator:** resolve remaining P2P/compression/V5 issues
([#3015](#3015))
([1e4379e](1e4379e))
* **orchestrator:** sanitize OCI pull errors
([#3096](#3096))
([a3af6c0](a3af6c0))
* **orchestrator:** scope rootfs hash to provision default
([#3129](#3129))
([475f955](475f955))
* **orchestrator:** stop Checks health-loop leaking
([#2739](#2739))
([17e6e60](17e6e60))
* **orchestrator:** survive SIGBUS from failing disks under mmap'd
caches ([#3385](#3385))
([728bba3](728bba3))
* **orchestrator:** tolerate missing header for legacy templates
([#3026](#3026))
([8a44bfe](8a44bfe))
* **orch:** fall back to ID_LIKE with a warning instead of rejecting
([#3459](#3459))
([7167818](7167818))
* **orch:** prevent NBD dispatch read-loop stall on WRITE_ZEROES (behind
flag) ([#3048](#3048))
([efd3d4d](efd3d4d))
* **orch:** split scheduling base build id per artifact
([#2920](#2920))
([3e35a2a](3e35a2a))
* **orch:** validate copy-build -gdb buckets before the snapshot copy
([#3446](#3446))
([586ad74](586ad74))
* **shared:** never report a failed envd command stream as success
([#3281](#3281))
([69c06b6](69c06b6))
* **storage:** compression upload & cache correctness fixes
([#3231](#3231))
([980748f](980748f))
* **storage:** don't assume V4+ ancestor gaps are uncompressed
([#3447](#3447))
([bfdbb24](bfdbb24))
* **uffd:** dedupe deferred page faults
([#2864](#2864))
([9680a41](9680a41))
* WrapContextAsUserError should not misclassify internal timeouts as
user cancellations
([#3155](#3155))
([8f83959](8f83959))


### Performance Improvements

* **build:** cache resolved Diff per BuildId within File.ReadAt
([#2838](#2838))
([53de07f](53de07f))
* **build:** parallelize fragmented backing reads
([#2872](#2872))
([c7655a7](c7655a7))
* **clean-nfs-cache:** restore dirfd-relative statx
([#2766](#2766))
([6bdbedb](6bdbedb))
* **header:** add V5 columnar varint header format
([#2847](#2847))
([9dd931b](9dd931b))
* **header:** pack cached Header.Mapping into a compact form
([#2844](#2844))
([7f0b13c](7f0b13c))
* **orchestrator:** add memfile dedup density threshold
([#2862](#2862))
([7ccfa02](7ccfa02))
* **orchestrator:** avoid V3-ancestor header refresh
([#2999](#2999))
([cb6aa0b](cb6aa0b))
* **orch:** metrics for dirty page throttling
([#2858](#2858))
([d2aa554](d2aa554))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: e2b-release-please[bot] <298072688+e2b-release-please[bot]@users.noreply.github.com>
Co-authored-by: Charlie Wyse <charlie.wyse@e2b.dev>
jakubno pushed a commit that referenced this pull request Aug 3, 2026
🤖 I have created a release *beep* *boop*
---


## 0.0.1 (2026-07-30)


### Features

* **api:** add sandbox IAM workload token configuration
([13ddb3d](13ddb3d))
* **api:** add sandbox workload identity permission
([#3319](#3319))
([13ddb3d](13ddb3d))
* **api:** SOCKS5 egress proxy on sandbox network config (BYOP)
([#2642](#2642))
([1fc3820](1fc3820))
* **cfg:** add DISABLE_STARTUP_RECLAIM flag
([#3081](#3081))
([7677ca6](7677ca6))
* **clickhouse:** implement multi-cluster fan-out for events and stats
([#2925](#2925))
([39594c6](39594c6))
* dynamic sandbox log routing and ClickHouse-backed log reads
([#3236](#3236))
([1b19a3b](1b19a3b))
* **envd:** give envd realtime IO priority, reset for user processes
([#2681](#2681))
([f4bd1b2](f4bd1b2))
* **envd:** split collapse stats into real migrations vs already-huge
([#3021](#3021))
([0d77614](0d77614))
* **envd:** support user-defined file metadata via xattrs
([#2732](#2732))
([da8fbe4](da8fbe4))
* **featureflags:** support per-service context providers
([#3100](#3100))
([65297c1](65297c1))
* freeze user cgroup across pause/resume to keep envd /init responsive
([#2688](#2688))
([eceb741](eceb741))
* **metrics:** break down pause-snapshot latency by step
([#3426](#3426))
([f551118](f551118))
* **metrics:** label pause telemetry by fs_only
([#3425](#3425))
([4be33ba](4be33ba))
* **observability:** add kill_reason to sandbox.lifecycle.killed
([#2833](#2833))
([e45418f](e45418f))
* **observability:** include kill_reason in kill-path structured logs
([#2846](#2846))
([33c49f7](33c49f7))
* **orch:** add envd-version to LaunchDarkly sandbox context
([#3051](#3051))
([37d3b92](37d3b92))
* **orch:** add less, nftables, iputils-ping, and jq to base
provisioning ([#2736](#2736))
([a1e010e](a1e010e))
* **orch:** collapse envd's heap into 2 MiB hugepages before pause to
cut cold-resume faults
([#2997](#2997))
([6677f73](6677f73))
* **orch:** debug a sandbox guest kernel with resume-build -gdb
([#3040](#3040))
([37bb0dc](37bb0dc))
* **orch:** decouple warm resume from memfile dedup
([#3166](#3166))
([77f25a0](77f25a0))
* **orch:** distro-aware template base-image provisioning
([#3411](#3411))
([1abece1](1abece1))
* **orchestrator/cgroup:** list and destroy leaked sandbox cgroups
([#3086](#3086))
([bce1d84](bce1d84))
* **orchestrator/nbd:** inspect and disconnect connected devices
([#3087](#3087))
([4d47148](4d47148))
* **orchestrator/network:** list slot namespaces
([#3089](#3089))
([c23dbc7](c23dbc7))
* **orchestrator/network:** list slot namespaces
([#3090](#3090))
([fbfce25](fbfce25))
* **orchestrator:** add -force-reboot to resume-build to cold-boot
memory-snaphsot builds
([#3208](#3208))
([cf8f15b](cf8f15b))
* **orchestrator:** add allocated resource metrics for sandboxes
([#2943](#2943))
([95cb6d3](95cb6d3))
* **orchestrator:** add dummy orchestrator binary for local API dev
([#2744](#2744))
([ab56e25](ab56e25))
* **orchestrator:** add NetworkAssignHook for sandbox lifecycle
extensions ([#3290](#3290))
([3261963](3261963))
* **orchestrator:** add soft-delete marker label to the check metric
([#3144](#3144))
([1ce64f8](1ce64f8))
* **orchestrator:** add v4HeaderForUncompressed FF bit
([#2669](#2669))
([1f459ee](1f459ee))
* **orchestrator:** always include execution metrics in sandbox webhook
events ([#2852](#2852))
([440edfe](440edfe))
* **orchestrator:** classify envd-init by exit type
([#3139](#3139))
([1e39a4f](1e39a4f))
* **orchestrator:** graceful sandbox drain on shutdown
([#3069](#3069))
([6ce68e3](6ce68e3))
* **orchestrator:** graceful template-build drain on shutdown
([#3079](#3079))
([1b3001c](1b3001c))
* **orchestrator:** improved read-path telemetry
([#3063](#3063))
([bc3fe84](bc3fe84))
* **orchestrator:** LD-gated ClickHouse write fan-out feature flag
([#3152](#3152))
([f046fcf](f046fcf))
* **orchestrator:** make build-reserved-disk-space-mb default 256MB
([#3065](#3065))
([d473f98](d473f98))
* **orchestrator:** record upload compression metrics
([#2761](#2761))
([9092e35](9092e35))
* **orchestrator:** report hugepage metrics to API
([#3182](#3182))
([7735bae](7735bae))
* **orchestrator:** run startup reclaim on boot
([#3123](#3123))
([79b838e](79b838e))
* **orchestrator:** single-instance flock on startup
([#3143](#3143))
([1320d6e](1320d6e))
* **orchestrator:** soft-delete consumer enforcement for storage index
([#3034](#3034))
([fbfc918](fbfc918))
* **orchestrator:** tag envd-init meters with start_type
([#3125](#3125))
([4466b48](4466b48))
* **orchestrator:** track and report last status change timestamp
([#2980](#2980))
([f79be77](f79be77))
* **orchestrator:** track sandbox lifecycles
([#2998](#2998))
([057f20c](057f20c))
* **orchestrator:** write layer sizes (logical/mapped/diff) to object
metadata ([#3122](#3122))
([11869c0](11869c0))
* **orch:** harvest resume-prefetch trace on pause
([#3067](#3067))
([97bd4a5](97bd4a5))
* **orch:** last-cycle memory prefetch on resume
([#3258](#3258))
([ea94196](ea94196))
* **orch:** make resume-build -gdb work on real nodes + add copy-build
-gdb ([#3108](#3108))
([5385594](5385594))
* **orch:** opt-in DSCP marker for sandbox egress (SANDBOX_EGRESS_DSCP)
([#3039](#3039))
([a98cf2c](a98cf2c))
* **orch:** per-start UFFD startup working-set metric
([#2960](#2960))
([dc386b2](dc386b2))
* **orch:** premade NixOS base-image support
([#3412](#3412))
([4bd42d2](4bd42d2))
* **orch:** record envd init duration histogram on failure with success
attribute ([#2749](#2749))
([afa7458](afa7458))
* **orch:** snapshot fragmentation metrics
([#2931](#2931))
([842b007](842b007))
* per-team events TTL limit (tier + addons)
([#3181](#3181))
([f76b2cb](f76b2cb))
* **shared:** add OTEL instrumentation to AWS S3 storage client
([#3172](#3172))
([25b0fd1](25b0fd1))
* **storage:** per-role storage URLs, env-free storage library
([#3246](#3246))
([fcbe909](fcbe909))
* **storage:** stamp provenance custom metadata on uploaded objects
(incl. headers) ([#3033](#3033))
([ba8604e](ba8604e))
* **storage:** write-through compressed templates to NFS on upload
([#2827](#2827))
([57503c1](57503c1))


### Bug Fixes

* added api and orch
([#3454](#3454))
([d56e0a8](d56e0a8))
* **block:** rephrase misleading error message in pwritevAll
([#2816](#2816))
([1555f1b](1555f1b))
* **cache:** use 512-byte units for stat.Blocks in FileSize
([#2949](#2949))
([0f632a9](0f632a9))
* **clean-nfs-cache:** exclude zombies from delete_age
([#3191](#3191))
([3fa2aeb](3fa2aeb))
* **compression:** correctness findings from compression audit
([#2803](#2803))
([d21a6a9](d21a6a9))
* **copy-build:** resolve compression suffix for build data files
([#2859](#2859))
([8966f7e](8966f7e))
* correct 3 CVES ([#3218](#3218))
([076823b](076823b))
* **envd:** stop freezing socat cgroup across pause/resume
([#2923](#2923))
([8b6f2b9](8b6f2b9))
* **inspect-build:** adapt validate to new Chunker upstream API
([#2989](#2989))
([2e0d3da](2e0d3da))
* **nbd:** adjust status poll sleep from 100ns to 100µs
([02bf51b](02bf51b))
* **nbd:** change NBD status poll sleep from 100ns to 100µs to avoid
useless busy spinning
([#2884](#2884))
([02bf51b](02bf51b))
* **nfsproxy:** deflake TestRoundTrip EADDRINUSE
([#2987](#2987))
([55f4d18](55f4d18))
* **orch:** denormalize upload metric file type
([#2865](#2865))
([b1646ca](b1646ca))
* **orch:** disable the chronyd seccomp filter on Alpine when using PHC
([#3453](#3453))
([dfa9764](dfa9764))
* **orchestrator:** anchor rsync CWD to root in template file copy
([#2835](#2835))
([7160db9](7160db9))
* **orchestrator:** atomically replace metadata
([#3321](#3321))
([0c4ad6b](0c4ad6b))
* **orchestrator:** avoid serializing upload headers twice
([#2762](#2762))
([9b7b149](9b7b149))
* **orchestrator:** chunk readiness bug in P2P-&gt;compressed
([#3185](#3185))
([74a6e5b](74a6e5b))
* **orchestrator:** deschedule flaky eviction-loop race in TestDiffSto…
([#3173](#3173))
([88ff17c](88ff17c))
* **orchestrator:** discard poisoned nftables conn on firewall errors
([#3008](#3008))
([03f10e0](03f10e0))
* **orchestrator:** drop stale pre-init logs
([#3297](#3297))
([8ec4be5](8ec4be5))
* **orchestrator:** emit compression ratios as fractions, not BP
([#2772](#2772))
([866f4c1](866f4c1))
* **orchestrator:** export dirty-page stall counter from process start
([#2992](#2992))
([badc8ad](badc8ad))
* **orchestrator:** harden Firecracker process shutdown
([#2996](#2996))
([df662e7](df662e7))
* **orchestrator:** harden shutdown network cleanup
([#3000](#3000))
([de2f391](de2f391))
* **orchestrator:** implement Docker COPY merge semantics in template
builds ([#3283](#3283))
([9174104](9174104))
* **orchestrator:** keep dedup empty-pages telemetry scan-only
([#2991](#2991))
([35d0832](35d0832))
* **orchestrator:** let build-cache threshold flag raise above its fal…
([#3175](#3175))
([06393c3](06393c3))
* **orchestrator:** log missing egress proxy in startup reclaim instead
of defaulting silently
([#3116](#3116))
([6ca3163](6ca3163))
* **orchestrator:** make copy-build handle filesystem-only snapshots
([#3299](#3299))
([62add04](62add04))
* **orchestrator:** measure ext4 free space from block groups
([#3282](#3282))
([f18f05f](f18f05f))
* **orchestrator:** normalize upload metric file labels
([#2767](#2767))
([6dec8b3](6dec8b3))
* **orchestrator:** order egress config/firewall updates to close BYOP
enable race ([#3313](#3313))
([7faa59e](7faa59e))
* **orchestrator:** order envd.service after local-fs.target
([#3043](#3043))
([ea2663e](ea2663e))
* **orchestrator:** order envd.service after systemd-tmpfiles-setup
([#3130](#3130))
([9481811](9481811))
* **orchestrator:** pause upload retain retry
([#2993](#2993))
([4f81799](4f81799))
* **orchestrator:** pin tap device host-side MAC address
([#3271](#3271))
([3c786ba](3c786ba))
* **orchestrator:** pin UFFD copy source buffers
([#2745](#2745))
([837fa91](837fa91))
* **orchestrator:** preserve full ENV value across stdout chunks
([#2740](#2740))
([4822e6d](4822e6d))
* **orchestrator:** read V3 ancestors as uncompressed instead of failing
([#2994](#2994))
([c479dd3](c479dd3))
* **orchestrator:** reject standby while draining
([#3325](#3325))
([475a7ee](475a7ee))
* **orchestrator:** report real V4 header compression ratio
([#2771](#2771))
([ecd344e](ecd344e))
* **orchestrator:** resolve remaining P2P/compression/V5 issues
([#3015](#3015))
([1e4379e](1e4379e))
* **orchestrator:** sanitize OCI pull errors
([#3096](#3096))
([a3af6c0](a3af6c0))
* **orchestrator:** scope rootfs hash to provision default
([#3129](#3129))
([475f955](475f955))
* **orchestrator:** stop Checks health-loop leaking
([#2739](#2739))
([17e6e60](17e6e60))
* **orchestrator:** survive SIGBUS from failing disks under mmap'd
caches ([#3385](#3385))
([8694d08](8694d08))
* **orchestrator:** tolerate missing header for legacy templates
([#3026](#3026))
([8a44bfe](8a44bfe))
* **orch:** fall back to ID_LIKE with a warning instead of rejecting
([#3459](#3459))
([73399b3](73399b3))
* **orch:** prevent NBD dispatch read-loop stall on WRITE_ZEROES (behind
flag) ([#3048](#3048))
([efd3d4d](efd3d4d))
* **orch:** split scheduling base build id per artifact
([#2920](#2920))
([3e35a2a](3e35a2a))
* **orch:** validate copy-build -gdb buckets before the snapshot copy
([#3446](#3446))
([9be382f](9be382f))
* **shared:** never report a failed envd command stream as success
([#3281](#3281))
([69c06b6](69c06b6))
* **storage:** compression upload & cache correctness fixes
([#3231](#3231))
([980748f](980748f))
* **storage:** don't assume V4+ ancestor gaps are uncompressed
([#3447](#3447))
([f828d12](f828d12))
* **uffd:** dedupe deferred page faults
([#2864](#2864))
([9680a41](9680a41))
* WrapContextAsUserError should not misclassify internal timeouts as
user cancellations
([#3155](#3155))
([8f83959](8f83959))


### Performance Improvements

* **build:** cache resolved Diff per BuildId within File.ReadAt
([#2838](#2838))
([53de07f](53de07f))
* **build:** parallelize fragmented backing reads
([#2872](#2872))
([c7655a7](c7655a7))
* **clean-nfs-cache:** restore dirfd-relative statx
([#2766](#2766))
([6bdbedb](6bdbedb))
* **header:** add V5 columnar varint header format
([#2847](#2847))
([9dd931b](9dd931b))
* **header:** pack cached Header.Mapping into a compact form
([#2844](#2844))
([7f0b13c](7f0b13c))
* **orchestrator:** add memfile dedup density threshold
([#2862](#2862))
([7ccfa02](7ccfa02))
* **orchestrator:** avoid V3-ancestor header refresh
([#2999](#2999))
([cb6aa0b](cb6aa0b))
* **orch:** metrics for dirty page throttling
([#2858](#2858))
([d2aa554](d2aa554))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: e2b-release-please[bot] <298072688+e2b-release-please[bot]@users.noreply.github.com>
Co-authored-by: Charlie Wyse <charlie.wyse@e2b.dev>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants