Skip to content

refactor(storage): FrameTable vs FullFrameTable - #2975

Merged
levb merged 4 commits into
mainfrom
lev-frametable-full-partial-split
Jun 10, 2026
Merged

levb merged 4 commits into
mainfrom
lev-frametable-full-partial-split

Conversation

@levb

@levb levb commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Add a FullFrameTable wrapper for the two narrow contexts where the FT covers a whole file: (a) fresh out of compressStream / CompressBytes / StoreFile / UploadFramed, and (b) the future StorageDiff fallback latch (#2919). Producers return *FullFrameTable; downcast via Table() to feed the hot read path or assign to BuildData.FrameData.

FrameTable itself stays the authoritative type — what BuildData stores, what DeserializeFrameTable and TrimToRanges return, what ReadAt and OpenRangeReader accept. No on-disk format change, no behavior change.

Add a FullFrameTable wrapper for the two narrow contexts where the FT
covers a whole file: (a) fresh out of compressStream / CompressBytes /
StoreFile / UploadFramed, and (b) the future StorageDiff fallback latch
(#2919). Producers return *FullFrameTable; downcast via Table() to feed
the hot read path or assign to BuildData.FrameData.

FrameTable itself stays the authoritative type — what BuildData stores,
what DeserializeFrameTable and TrimToRanges return, what ReadAt and
OpenRangeReader accept. No on-disk format change, no behavior change.
@codecov

codecov Bot commented Jun 10, 2026 •

Copy link
Copy Markdown

❌ 5 Tests Failed:

Tests completed Failed Passed Skipped
2781 5 2776 7
View the top 3 failed test(s) by shortest run time
github.com/e2b-dev/infra/tests/integration/internal/tests/api/sandboxes::TestSnapshotTemplateCreate
Stack Traces | 0s run time
=== RUN   TestSnapshotTemplateCreate
=== PAUSE TestSnapshotTemplateCreate
=== CONT  TestSnapshotTemplateCreate
--- FAIL: TestSnapshotTemplateCreate (0.00s)
github.com/e2b-dev/infra/packages/shared/pkg/storage::TestMultipartUploader_HighConcurrency_StressTest
Stack Traces | 0.55s run time
=== RUN   TestMultipartUploader_HighConcurrency_StressTest
=== PAUSE TestMultipartUploader_HighConcurrency_StressTest
=== CONT  TestMultipartUploader_HighConcurrency_StressTest
    gcp_multipart_test.go:398: 
        	Error Trace:	.../pkg/storage/gcp_multipart_test.go:398
        	Error:      	"1" is not greater than "1"
        	Test:       	TestMultipartUploader_HighConcurrency_StressTest
        	Messages:   	Should have concurrent uploads
--- FAIL: TestMultipartUploader_HighConcurrency_StressTest (0.55s)
github.com/e2b-dev/infra/tests/integration/internal/tests/api/sandboxes::TestSnapshotTemplateCreate/create_snapshot_with_name_and_tag
Stack Traces | 33.4s run time
=== RUN   TestSnapshotTemplateCreate/create_snapshot_with_name_and_tag
=== PAUSE TestSnapshotTemplateCreate/create_snapshot_with_name_and_tag
=== CONT  TestSnapshotTemplateCreate/create_snapshot_with_name_and_tag
    snapshot_template_test.go:97: 
        	Error Trace:	.../api/sandboxes/snapshot_template_test.go:97
        	Error:      	Not equal: 
        	            	expected: 201
        	            	actual  : 409
        	Test:       	TestSnapshotTemplateCreate/create_snapshot_with_name_and_tag
--- FAIL: TestSnapshotTemplateCreate/create_snapshot_with_name_and_tag (33.37s)
View the full list of 3 ❄️ flaky test(s)
github.com/e2b-dev/infra/tests/integration/internal/tests/api/sandboxes::TestSandboxListPaginationRunningLargerLimit

Flake rate in main: 38.21% (Passed 1001 times, Failed 619 times)

Stack Traces | 93.8s run time
=== RUN   TestSandboxListPaginationRunningLargerLimit
    sandbox_list_test.go:327: Created sandbox 1/12: ib0m6b5hb1s27wx5vwpxo
    sandbox_list_test.go:327: Created sandbox 2/12: ib3xbgl3owjtl8xl4qhpr
    sandbox_list_test.go:327: Created sandbox 3/12: i7ac0z65rywwvijc3x3r1
    sandbox_list_test.go:327: Created sandbox 4/12: icejbw3074y4gyp0hle88
    sandbox_list_test.go:327: Created sandbox 5/12: ikfe0q2ob7ofh0hgm9hgx
    sandbox_list_test.go:327: Created sandbox 6/12: ikws1o9cyzbghcqg557wm
    sandbox_list_test.go:327: Created sandbox 7/12: iiup131bdezlffqwa7od8
    sandbox_list_test.go:327: Created sandbox 8/12: ig6qyuuveuef3vbuvg49l
    sandbox_list_test.go:327: Created sandbox 9/12: iwm75jkgo3y0e10psxp1n
    sandbox_list_test.go:327: Created sandbox 10/12: irm4aab3jmug65ox0mfmj
    sandbox_list_test.go:327: Created sandbox 11/12: i32h1dby81nvnd2xsh18x
    sandbox_list_test.go:327: Created sandbox 12/12: ij1svf180ge091ny1ztpj
    sandbox_list_test.go:330: 
        	Error Trace:	.../api/sandboxes/sandbox_list_test.go:340
        	            				.../hostedtoolcache/go/1.26.3.../src/runtime/asm_amd64.s:1771
        	Error:      	"[]" should have 12 item(s), but has 0
    sandbox_list_test.go:330: 
        	Error Trace:	.../api/sandboxes/sandbox_list_test.go:330
        	Error:      	Condition never satisfied
        	Test:       	TestSandboxListPaginationRunningLargerLimit
--- FAIL: TestSandboxListPaginationRunningLargerLimit (93.84s)
github.com/e2b-dev/infra/tests/integration/internal/tests/orchestrator::TestSandboxMemoryIntegrity

Flake rate in main: 53.00% (Passed 994 times, Failed 1121 times)

Stack Traces | 62.5s run time
=== RUN   TestSandboxMemoryIntegrity
=== PAUSE TestSandboxMemoryIntegrity
=== CONT  TestSandboxMemoryIntegrity
    sandbox_memory_integrity_test.go:27: Build completed successfully
--- FAIL: TestSandboxMemoryIntegrity (62.53s)
github.com/e2b-dev/infra/tests/integration/internal/tests/orchestrator::TestSandboxMemoryIntegrity/tmpfs_hash

Flake rate in main: 53.01% (Passed 984 times, Failed 1110 times)

Stack Traces | 191s run time
=== RUN   TestSandboxMemoryIntegrity/tmpfs_hash
=== PAUSE TestSandboxMemoryIntegrity/tmpfs_hash
=== CONT  TestSandboxMemoryIntegrity/tmpfs_hash
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{start:{pid:1259}}
Executing command bash in sandbox ijat3v0h1sdt89akxhsy0 (user: root)
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stdout:"Total memory: 985 MB\nUsed memory before tmpfs mount: 189 MB\nFree memory before tmpfs mount: 795 MB\nMemory to use in integrity test (60% of free, min 64MB): 477 MB\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"477+0 records in\n477+0 records out\n500170752 bytes (500 MB, 477 MiB) copied, 1.99389 s, 251 MB/s\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\tCommand being timed: \"dd if=/dev/urandom of=/mnt/testfile bs=1M count=477\"\n\tUser time (se"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"conds): "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"0.00\n\tSystem time (seconds): 1.93\n\tPercent of CPU this job got: 96%\n\tElapsed (wall clock) time (h:mm:ss or m:ss): "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"0:02.00\n\tAverage shared te"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"xt size (kbytes): 0\n\tA"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"verage unshared data size (kbytes): 0\n\tAverage stack size (kbytes): 0\n\tAverage total size (kbytes): 0\n\tMaximum "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"resident se"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"t size ("}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"kbytes): 2704\n\tAverage r"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"esident s"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"et size"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" (kbytes"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"): 0\n\tM"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"ajor (requir"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"ing I/O"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:") page "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"faults: 3\n\tMinor (reclaiming a frame) page faults: 342\n\tVoluntary context switches: 4\n\tInvoluntary context switches: 14\n\tSwaps: 0"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\n\tFile system inputs: "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"176\n\tFile "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"system o"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"utputs:"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" 0\n\tSock"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"et messages"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" sent: "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"0\n\tSocke"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"t messa"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"ges rece"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"ived: 0\n\tSi"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"gnals de"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"livered: 0\n\tPage size (bytes): 4096\n\tExit status: 0\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stdout:"Used memory after tmpfs mount and file fill: 670 MB\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{end:{exited:true status:"exit status 0"}}
    sandbox_memory_integrity_test.go:70: Command [bash] completed successfully in sandbox iz41yy8s7atocljr66fp3
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{start:{pid:1275}}
Executing command bash in sandbox ix45g80fptm0rh7nw5l58 (user: root)
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{data:{stdout:"5b1c94117610cfe1d6fc2a59ecec7c7f487fd73f82465d661f60094700b8c76e\n"}}
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{end:{exited:true status:"exit status 0"}}
    sandbox_memory_integrity_test.go:80: Command [bash] completed successfully in sandbox iz41yy8s7atocljr66fp3
Executing command bash in sandbox ix45g80fptm0rh7nw5l58 (user: root)
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{start:{pid:1279}}
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
Executing command bash in sandbox iz41yy8s7atocljr66fp3 (user: root)
    sandbox_memory_integrity_test.go:110: 
        	Error Trace:	.../tests/orchestrator/sandbox_memory_integrity_test.go:81
        	            				.../hostedtoolcache/go/1.26.3.../src/runtime/asm_amd64.s:1771
        	Error:      	Received unexpected error:
        	            	failed to execute command bash in sandbox iz41yy8s7atocljr66fp3: unavailable: HTTP status 502 Bad Gateway
    sandbox_memory_integrity_test.go:110: 
        	Error Trace:	.../tests/orchestrator/sandbox_memory_integrity_test.go:78
        	            				.../tests/orchestrator/sandbox_memory_integrity_test.go:110
        	Error:      	Condition never satisfied
        	Test:       	TestSandboxMemoryIntegrity/tmpfs_hash
--- FAIL: TestSandboxMemoryIntegrity/tmpfs_hash (191.39s)

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

Comment thread packages/orchestrator/pkg/sandbox/build_upload_v4.go Outdated

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

In packages/orchestrator/pkg/sandbox/build_upload_v4.go, fullFT can be nil for uncompressed uploads, which will cause a nil pointer dereference when calling UncompressedSize, CompressedSize, or IsCompressed. Using the nil-safe Table() method to get the underlying *FrameTable before calling these methods will prevent this panic.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread packages/orchestrator/pkg/sandbox/build_upload_v4.go Outdated
github-actions Bot and others added 3 commits June 10, 2026 18:05
Make the FullFrameTable wrapper unembedded (private named field), so
methods on the inner FrameTable are no longer promoted. The only way
to use a *FullFrameTable is now its nil-safe Table() — calling
methods directly on a nil *FullFrameTable becomes a compile error
instead of a runtime panic.

This catches the bug Gemini flagged on build_upload_v4.go: fullFT is
nil for uncompressed uploads, and Go's embed selector would deref it
before reaching the underlying nil-safe FrameTable methods. Now fixed
by extracting ft := fullFT.Table() once and routing all calls through
it. Same fix applied to debug-logging in storage_fs/storage_google.

TrimToRanges, DeserializeFrameTable, Serialize, and
newFrameTableFromEntries are all reverted to their original bodies
(no behavior delta in FrameTable itself).
@levb
levb merged commit 4dbd594 into main Jun 10, 2026
96 of 98 checks passed
@levb
levb deleted the lev-frametable-full-partial-split branch June 10, 2026 22:12
Piwriw pushed a commit to Piwriw/infra that referenced this pull request Jun 12, 2026
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants