Skip to content

perf(iac): increase client dirty page limits - #2861

Merged
ValentaTomas merged 1 commit into
mainfrom
valenta/increase-host-dirty-pages
May 30, 2026
Merged

ValentaTomas merged 1 commit into
mainfrom
valenta/increase-host-dirty-pages

Conversation

@ValentaTomas

Copy link
Copy Markdown
Member

Summary

Raise client host dirty page thresholds so writeback-heavy workloads can keep more dirty data in flight before throttling.

Test plan

Not run; bootstrap sysctl-only change.

@cla-bot cla-bot Bot added the cla-signed label May 29, 2026
@cursor

cursor Bot commented May 29, 2026 •

Copy link
Copy Markdown

PR Summary

Low Risk
Boot-time sysctl tuning only; slightly higher dirty memory use before throttling, with no auth or application logic changes.

Overview
Nomad client bootstrap now sets higher kernel dirty-page thresholds (vm.dirty_background_ratio and vm.dirty_ratio) so write-heavy workloads can accumulate more dirty memory before the kernel starts background or synchronous writeback throttling.

Reviewed by Cursor Bugbot for commit ecdac81. Bugbot is set up for automated code reviews on this repo. Configure here.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the Nomad client start script to configure larger host writeback bursts by setting vm.dirty_background_ratio and vm.dirty_ratio. There are no review comments or critical findings to report.

@codecov

codecov Bot commented May 29, 2026 •

Copy link
Copy Markdown

❌ 3 Tests Failed:

Tests completed Failed Passed Skipped
2697 3 2694 5
View the full list of 4 ❄️ flaky test(s)
github.com/e2b-dev/infra/tests/integration/internal/tests/api/metrics::TestSandboxMetrics

Flake rate in main: 44.59% (Passed 707 times, Failed 569 times)

Stack Traces | 18.7s run time
=== RUN   TestSandboxMetrics
=== PAUSE TestSandboxMetrics
=== CONT  TestSandboxMetrics
    sandbox_metrics_test.go:26: 
        	Error Trace:	.../api/metrics/sandbox_metrics_test.go:26
        	Error:      	Condition never satisfied
        	Test:       	TestSandboxMetrics
        	Messages:   	sandbox metrics not available in time
--- FAIL: TestSandboxMetrics (18.70s)
github.com/e2b-dev/infra/tests/integration/internal/tests/api/sandboxes::TestSandboxListPaginationRunningLargerLimit

Flake rate in main: 42.25% (Passed 719 times, Failed 526 times)

Stack Traces | 97.5s run time
=== RUN   TestSandboxListPaginationRunningLargerLimit
    sandbox_list_test.go:327: Created sandbox 1/12: iuq52hwh96my0mgc84kk4
    sandbox_list_test.go:327: Created sandbox 2/12: iy4eh5e8zrsa5m9u6atr4
    sandbox_list_test.go:327: Created sandbox 3/12: imtit0em2ojo2pfpi86hc
    sandbox_list_test.go:327: Created sandbox 4/12: i1zxkyk9vr8pcnm75avl9
    sandbox_list_test.go:327: Created sandbox 5/12: i9v2q5ur2qvhlua0b9c1x
    sandbox_list_test.go:327: Created sandbox 6/12: iabyfhxok0gew1cxcq5pr
    sandbox_list_test.go:327: Created sandbox 7/12: igro69ovf6yr36sxwpad6
    sandbox_list_test.go:327: Created sandbox 8/12: i5n6mu1qo1ape8zqitmgh
    sandbox_list_test.go:327: Created sandbox 9/12: iymhlbnfpad856zv8cjkc
    sandbox_list_test.go:327: Created sandbox 10/12: iap8gd1zsj2wiwnxaqpqs
    sandbox_list_test.go:327: Created sandbox 11/12: i8k2zsoq3bll117kptl07
    sandbox_list_test.go:327: Created sandbox 12/12: iy22nuke4m83mb9s1caak
    sandbox_list_test.go:330: 
        	Error Trace:	.../api/sandboxes/sandbox_list_test.go:340
        	            				.../hostedtoolcache/go/1.26.3.../src/runtime/asm_amd64.s:1771
        	Error:      	"[]" should have 12 item(s), but has 0
    sandbox_list_test.go:330: 
        	Error Trace:	.../api/sandboxes/sandbox_list_test.go:330
        	Error:      	Condition never satisfied
        	Test:       	TestSandboxListPaginationRunningLargerLimit
--- FAIL: TestSandboxListPaginationRunningLargerLimit (97.50s)
github.com/e2b-dev/infra/tests/integration/internal/tests/orchestrator::TestSandboxMemoryIntegrity

Flake rate in main: 57.67% (Passed 712 times, Failed 970 times)

Stack Traces | 66.1s run time
=== RUN   TestSandboxMemoryIntegrity
=== PAUSE TestSandboxMemoryIntegrity
=== CONT  TestSandboxMemoryIntegrity
    sandbox_memory_integrity_test.go:27: Build completed successfully
--- FAIL: TestSandboxMemoryIntegrity (66.10s)
github.com/e2b-dev/infra/tests/integration/internal/tests/orchestrator::TestSandboxMemoryIntegrity/tmpfs_hash

Flake rate in main: 57.79% (Passed 702 times, Failed 961 times)

Stack Traces | 202s run time
=== RUN   TestSandboxMemoryIntegrity/tmpfs_hash
=== PAUSE TestSandboxMemoryIntegrity/tmpfs_hash
=== CONT  TestSandboxMemoryIntegrity/tmpfs_hash
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{start:{pid:1271}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stdout:"Total memory: 985 MB\nUsed memory before tmpfs mount: 190 MB\nFree memory before tmpfs mount: 794 MB\nMemory to use in integrity test (60% of free, min 64MB): 476 MB\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"476+0 records in\n476+0 records out\n499122176 bytes (499 MB, 476 MiB) copied, 1.85487 s, 269 MB/s\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\t"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"C"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"o"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"m"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"m"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"a"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"d"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"b"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"e"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"i"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"g"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"t"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"i"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"m"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"e"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"d"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:":"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\""}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"dd"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"if=/dev/urandom"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"of=/mnt/testfile"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"bs=1M"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"count=476"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\""}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\t"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"U"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"s"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"e"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"r"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"t"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"i"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"m"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"e"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"("}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"s"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"e"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"c"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"o"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"d"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"s"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:")"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:":"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"0.00"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\t"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"S"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"y"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"s"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"t"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"e"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"m"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"t"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"ime (seconds): 1.84\n\tPercent of CPU this job got: 99%\n\tElapsed (wall clock) time (h:mm:ss or m:ss): 0:01.85\n\tAverage shared text size (kbytes): 0\n\tAverage unshared data size (kbytes): 0\n\tAverage stack size (kbytes): 0\n\tAverage total size (kbytes): 0\n\tMaximum resident set size (kbytes): 2712\n\tAverage resident set size (kbytes): 0\n\tMajor (requiring I/O) page faults: 2\n\tMinor ("}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"reclaiming a frame) page faults: 345\n\tVoluntary context switches: 3\n\tInvoluntary context switches: 28\n\tSwaps: 0\n\tFile system inputs: 176\n\tFile system outputs: 0\n\tSocket messages sent: 0\n\tSo"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"cket messages received: 0\n\tSignals delivered: 0\n\tPage size (bytes): 4096\n\tExit status: 0\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stdout:"Used memory after tmpfs mount and file fill: 670 MB\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{end:{exited:true  status:"exit status 0"}}
    sandbox_memory_integrity_test.go:70: Command [bash] completed successfully in sandbox ihatvye4lepsh2zr8jjlz
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{start:{pid:1287}}
Executing command bash in sandbox infr7gbquuldjsj9qs8yf (user: root)
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{data:{stdout:"d402ea6a01d7748230a4fcb45ce61a8ed64167fc8fdcb2504a2c972a7115e53d\n"}}
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{end:{exited:true  status:"exit status 0"}}
    sandbox_memory_integrity_test.go:80: Command [bash] completed successfully in sandbox ihatvye4lepsh2zr8jjlz
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{start:{pid:1290}}
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
Executing command bash in sandbox ihatvye4lepsh2zr8jjlz (user: root)
    sandbox_memory_integrity_test.go:110: 
        	Error Trace:	.../tests/orchestrator/sandbox_memory_integrity_test.go:81
        	            				.../hostedtoolcache/go/1.26.3.../src/runtime/asm_amd64.s:1771
        	Error:      	Received unexpected error:
        	            	failed to execute command bash in sandbox ihatvye4lepsh2zr8jjlz: unavailable: HTTP status 502 Bad Gateway
    sandbox_memory_integrity_test.go:110: 
        	Error Trace:	.../tests/orchestrator/sandbox_memory_integrity_test.go:78
        	            				.../tests/orchestrator/sandbox_memory_integrity_test.go:110
        	Error:      	Condition never satisfied
        	Test:       	TestSandboxMemoryIntegrity/tmpfs_hash
--- FAIL: TestSandboxMemoryIntegrity/tmpfs_hash (201.61s)

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@ValentaTomas
ValentaTomas marked this pull request as ready for review May 29, 2026 21:30
@ValentaTomas
ValentaTomas enabled auto-merge (squash) May 29, 2026 21:30
@ValentaTomas
ValentaTomas requested a review from optix2000 May 30, 2026 00:09
@ValentaTomas
ValentaTomas merged commit bb920f4 into main May 30, 2026
52 checks passed
@ValentaTomas
ValentaTomas deleted the valenta/increase-host-dirty-pages branch May 30, 2026 00:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants