[EN-1603] fix(api): prevent uint64 underflow in node allocated metrics by jakubno · Pull Request #3216 · e2b-dev/runtime

jakubno · 2026-07-07T15:11:02Z

OptimisticRemove subtracted from unsigned counters without a guard. When the periodic ServiceInfo poll overwrote the local counter after a sandbox already left the orchestrator's map, the subsequent remove underflowed

cursor · 2026-07-07T15:11:09Z

PR Summary

Medium Risk
Touches optimistic resource accounting used during sandbox deletion and scheduling; skipping a remove can briefly leave counters slightly high until the next sync, but avoids the worse failure mode of massive bogus “free” capacity from underflow.

Overview
OptimisticRemove no longer blindly subtracts sandbox CPU and memory from unsigned node allocation counters. When the amount to remove exceeds what is currently recorded—typically because a ServiceInfo poll already reset counts after the sandbox left the orchestrator—it logs a warning and skips the subtraction instead of wrapping to huge values that would distort placement.

Tests now seed initial allocated memory where needed and cover both partial mismatch and a fresh node with zero allocation.

^{Reviewed by Cursor Bugbot for commit f1f7fe2. Bugbot is set up for automated code reviews on this repo. Configure here.}

codecov · 2026-07-07T15:12:33Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

📢 Thoughts on this report? Let us know!

gemini-code-assist

Code Review

This pull request introduces safety checks to prevent integer underflow and wrapping when calculating and updating allocated sandbox resources. Specifically, it adds guards in OptimisticRemove to avoid subtracting resources below zero and refactors the sandbox resource summation into a validated helper function sumAllocatedResources with accompanying unit tests. The review feedback suggests further strengthening these safety measures by adding defensive nil checks for sandbox metadata configurations to prevent runtime panics, and explicitly guarding against negative input values before casting them to unsigned integers in OptimisticRemove.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 239ff4d26a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Co-authored-by: Petr Vaněk <petr.vanek@e2b.dev>

linear-code · 2026-07-07T15:56:55Z

EN-1603

🤖 I have created a release *beep* *boop* --- ## 0.0.1 (2026-07-30) ### Features * add workspace admin API foundations ([#3314](#3314)) ([0f72030](0f72030)) * **api:** add admin team API key routes ([#2825](#2825)) ([4a1e083](4a1e083)) * **api:** add feature flag to stop accepting E2B access tokens ([#3240](#3240)) ([2cf489b](2cf489b)) * **api:** add sandbox fork endpoint ([#3202](#3202)) ([643d726](643d726)) * **api:** add sandbox IAM workload token configuration ([13ddb3d](13ddb3d)) * **api:** add sandbox workload identity permission ([#3319](#3319)) ([13ddb3d](13ddb3d)) * **api:** add user agent integration attribution to PostHog events ([#3303](#3303)) ([d83be18](d83be18)) * **api:** discover orchestrators via nomad service ([#3176](#3176)) ([32af250](32af250)) * **api:** e2b access token deprecation feature flag rename ([#3110](#3110)) ([ebc2daa](ebc2daa)) * **api:** enforce blocked-team restrictions at mutating API endpoints ([#2659](#2659)) ([db848ab](db848ab)) * **api:** filter snapshots by name ([#3184](#3184)) ([6fa1bc7](6fa1bc7)) * **api:** gate access token issuance behind feature flag ([#3101](#3101)) ([2f7811e](2f7811e)) * **api:** LD-gated ClickHouse read switcher ([#3061](#3061)) ([29e74ca](29e74ca)) * **api:** limit template build name to 128 characters ([#3109](#3109)) ([84aa186](84aa186)) * **api:** paginated GET /v2/templates (EN-603) ([#3059](#3059)) ([91e02e4](91e02e4)) * **api:** per-region volume type defaults from node-derived region ([#3435](#3435)) ([baf5559](baf5559)) * **api:** pin resume retries to the node a previous resume timed out on ([#3066](#3066)) ([a4fd0f2](a4fd0f2)) * **api:** SOCKS5 egress proxy on sandbox network config (BYOP) ([#2642](#2642)) ([1fc3820](1fc3820)) * **api:** soft-delete build layers in DB on user delete ([#3121](#3121)) ([ee88776](ee88776)) * **auth:** support admin token team auth ([#2934](#2934)) ([5496666](5496666)) * dynamic sandbox log routing and ClickHouse-backed log reads ([#3236](#3236)) ([1b19a3b](1b19a3b)) * **evictor:** make max concurrent evictions a feature flag ([#2727](#2727)) ([0b33013](0b33013)) * **metrics:** distinguish joined from regular requests (ENG-4072) ([#2699](#2699)) ([390e296](390e296)) * **observability:** add kill_reason to sandbox.lifecycle.killed ([#2833](#2833)) ([e45418f](e45418f)) * **observability:** include kill_reason in kill-path structured logs ([#2846](#2846)) ([33c49f7](33c49f7)) * **orchestrator:** add dummy orchestrator binary for local API dev ([#2744](#2744)) ([ab56e25](ab56e25)) * **orchestrator:** report hugepage metrics to API ([#3182](#3182)) ([7735bae](7735bae)) * **orchestrator:** track and report last status change timestamp ([#2980](#2980)) ([f79be77](f79be77)) * **otel:** instrument auth service HTTP client with otelhttp ([#2722](#2722)) ([69b085d](69b085d)) * per-team events TTL limit (tier + addons) ([#3181](#3181)) ([f76b2cb](f76b2cb)) * **storage:** stamp provenance custom metadata on uploaded objects (incl. headers) ([#3033](#3033)) ([ba8604e](ba8604e)) ### Bug Fixes * added api and orch ([#3454](#3454)) ([fda5e45](fda5e45)) * **api:** check template alias tags in exists endpoint ([#2916](#2916)) ([9574cdf](9574cdf)) * **api:** copy auth/internal into api and dashboard-api image builds ([#3323](#3323)) ([bda1fee](bda1fee)) * **api:** discover the local orchestrator as a template builder ([#3386](#3386)) ([9ea005a](9ea005a)) * **api:** expose pagination headers via CORS ([#3388](#3388)) ([e832b1e](e832b1e)) * **api:** handle corrupted data in sandbox stop time ([#3203](#3203)) ([a98a178](a98a178)) * **api:** include exhaustion reason in "Node exhausted" placement warning ([#3279](#3279)) ([eb3797b](eb3797b)) * **api:** invalidate auth cache on API key deletion ([#3324](#3324)) ([8b02910](8b02910)) * **api:** keep API alive until in-flight requests finish ([#2708](#2708)) ([06378c7](06378c7)) * **api:** let the analytics collector address carry a port ([#3394](#3394)) ([6d41cb5](6d41cb5)) * **api:** parse the pause body regardless of Content-Length ([#3056](#3056)) ([d66aab8](d66aab8)) * **api:** prevent uint64 underflow in node allocated metrics ([#3216](#3216)) ([fed38e1](fed38e1)) * **api:** push api and db-migrator images to both latest and commit S… ([#2951](#2951)) ([6f010fc](6f010fc)) * **api:** reject non-positive timeout on sandbox create, resume, and fork ([#3419](#3419)) ([b672bd1](b672bd1)) * **api:** report invalid tag errors as bad requests ([#2799](#2799)) ([10085a1](10085a1)) * **api:** stop evicting the local node during sync ([#2881](#2881)) ([5455905](5455905)) * **api:** use correct error variable in processCustomErrors ([#3135](#3135)) ([a131a00](a131a00)) * **auth:** rename X-Team-Id header to X-Team-ID ([#2723](#2723)) ([f92ecc0](f92ecc0)) * correct 3 CVES ([#3218](#3218)) ([076823b](076823b)) * **orchestrator:** reject standby while draining ([#3325](#3325)) ([475a7ee](475a7ee)) * Support snapshots for non-default clusters ([#2947](#2947)) ([28eeb72](28eeb72)) ### Performance Improvements * **api:** wake reservation waiters via pub/sub instead of 20ms polling [ENG-4070] ([#2729](#2729)) ([2944d06](2944d06)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: e2b-release-please[bot] <298072688+e2b-release-please[bot]@users.noreply.github.com>

🤖 I have created a release *beep* *boop* --- ## 0.0.1 (2026-07-30) ### Features * add workspace admin API foundations ([#3314](#3314)) ([0f72030](0f72030)) * **api:** add admin team API key routes ([#2825](#2825)) ([4a1e083](4a1e083)) * **api:** add feature flag to stop accepting E2B access tokens ([#3240](#3240)) ([2cf489b](2cf489b)) * **api:** add sandbox fork endpoint ([#3202](#3202)) ([643d726](643d726)) * **api:** add sandbox IAM workload token configuration ([13ddb3d](13ddb3d)) * **api:** add sandbox workload identity permission ([#3319](#3319)) ([13ddb3d](13ddb3d)) * **api:** add user agent integration attribution to PostHog events ([#3303](#3303)) ([d83be18](d83be18)) * **api:** discover orchestrators via nomad service ([#3176](#3176)) ([32af250](32af250)) * **api:** e2b access token deprecation feature flag rename ([#3110](#3110)) ([ebc2daa](ebc2daa)) * **api:** enforce blocked-team restrictions at mutating API endpoints ([#2659](#2659)) ([db848ab](db848ab)) * **api:** filter snapshots by name ([#3184](#3184)) ([6fa1bc7](6fa1bc7)) * **api:** gate access token issuance behind feature flag ([#3101](#3101)) ([2f7811e](2f7811e)) * **api:** LD-gated ClickHouse read switcher ([#3061](#3061)) ([29e74ca](29e74ca)) * **api:** limit template build name to 128 characters ([#3109](#3109)) ([84aa186](84aa186)) * **api:** paginated GET /v2/templates (EN-603) ([#3059](#3059)) ([91e02e4](91e02e4)) * **api:** per-region volume type defaults from node-derived region ([#3435](#3435)) ([1bded44](1bded44)) * **api:** pin resume retries to the node a previous resume timed out on ([#3066](#3066)) ([a4fd0f2](a4fd0f2)) * **api:** SOCKS5 egress proxy on sandbox network config (BYOP) ([#2642](#2642)) ([1fc3820](1fc3820)) * **api:** soft-delete build layers in DB on user delete ([#3121](#3121)) ([ee88776](ee88776)) * **auth:** support admin token team auth ([#2934](#2934)) ([5496666](5496666)) * dynamic sandbox log routing and ClickHouse-backed log reads ([#3236](#3236)) ([1b19a3b](1b19a3b)) * **evictor:** make max concurrent evictions a feature flag ([#2727](#2727)) ([0b33013](0b33013)) * **metrics:** distinguish joined from regular requests (ENG-4072) ([#2699](#2699)) ([390e296](390e296)) * **observability:** add kill_reason to sandbox.lifecycle.killed ([#2833](#2833)) ([e45418f](e45418f)) * **observability:** include kill_reason in kill-path structured logs ([#2846](#2846)) ([33c49f7](33c49f7)) * **orchestrator:** add dummy orchestrator binary for local API dev ([#2744](#2744)) ([ab56e25](ab56e25)) * **orchestrator:** report hugepage metrics to API ([#3182](#3182)) ([7735bae](7735bae)) * **orchestrator:** track and report last status change timestamp ([#2980](#2980)) ([f79be77](f79be77)) * **otel:** instrument auth service HTTP client with otelhttp ([#2722](#2722)) ([69b085d](69b085d)) * per-team events TTL limit (tier + addons) ([#3181](#3181)) ([f76b2cb](f76b2cb)) * **storage:** stamp provenance custom metadata on uploaded objects (incl. headers) ([#3033](#3033)) ([ba8604e](ba8604e)) ### Bug Fixes * added api and orch ([#3454](#3454)) ([d56e0a8](d56e0a8)) * **api:** check template alias tags in exists endpoint ([#2916](#2916)) ([9574cdf](9574cdf)) * **api:** copy auth/internal into api and dashboard-api image builds ([#3323](#3323)) ([bda1fee](bda1fee)) * **api:** discover the local orchestrator as a template builder ([#3386](#3386)) ([9ea005a](9ea005a)) * **api:** expose pagination headers via CORS ([#3388](#3388)) ([e832b1e](e832b1e)) * **api:** handle corrupted data in sandbox stop time ([#3203](#3203)) ([a98a178](a98a178)) * **api:** include exhaustion reason in "Node exhausted" placement warning ([#3279](#3279)) ([eb3797b](eb3797b)) * **api:** invalidate auth cache on API key deletion ([#3324](#3324)) ([8b02910](8b02910)) * **api:** keep API alive until in-flight requests finish ([#2708](#2708)) ([06378c7](06378c7)) * **api:** let the analytics collector address carry a port ([#3394](#3394)) ([6d41cb5](6d41cb5)) * **api:** parse the pause body regardless of Content-Length ([#3056](#3056)) ([d66aab8](d66aab8)) * **api:** prevent uint64 underflow in node allocated metrics ([#3216](#3216)) ([fed38e1](fed38e1)) * **api:** push api and db-migrator images to both latest and commit S… ([#2951](#2951)) ([6f010fc](6f010fc)) * **api:** reject non-positive timeout on sandbox create, resume, and fork ([#3419](#3419)) ([5a4b631](5a4b631)) * **api:** report invalid tag errors as bad requests ([#2799](#2799)) ([10085a1](10085a1)) * **api:** stop evicting the local node during sync ([#2881](#2881)) ([5455905](5455905)) * **api:** use correct error variable in processCustomErrors ([#3135](#3135)) ([a131a00](a131a00)) * **auth:** rename X-Team-Id header to X-Team-ID ([#2723](#2723)) ([f92ecc0](f92ecc0)) * correct 3 CVES ([#3218](#3218)) ([076823b](076823b)) * **orchestrator:** reject standby while draining ([#3325](#3325)) ([475a7ee](475a7ee)) * Support snapshots for non-default clusters ([#2947](#2947)) ([28eeb72](28eeb72)) ### Performance Improvements * **api:** wake reservation waiters via pub/sub instead of 20ms polling [ENG-4070] ([#2729](#2729)) ([2944d06](2944d06)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: e2b-release-please[bot] <298072688+e2b-release-please[bot]@users.noreply.github.com>

jakubno requested review from ValentaTomas and dobrac as code owners July 7, 2026 15:11

cla-bot Bot added the cla-signed label Jul 7, 2026

gemini-code-assist Bot reviewed Jul 7, 2026

View reviewed changes

Comment thread packages/orchestrator/pkg/service/service_info.go Outdated

Comment thread packages/api/internal/orchestrator/nodemanager/node.go

chatgpt-codex-connector Bot reviewed Jul 7, 2026

View reviewed changes

Comment thread packages/orchestrator/pkg/service/service_info.go Outdated

fix(api): prevent uint64 underflow in node allocated metrics

3f1cb09

jakubno force-pushed the fix/node-metrics-underflow branch from 239ff4d to 3f1cb09 Compare July 7, 2026 15:16

chore: simplify

8e9f6d8

arkamar approved these changes Jul 7, 2026

View reviewed changes

Comment thread packages/api/internal/orchestrator/nodemanager/node.go Outdated

Apply suggestion from @arkamar

f1f7fe2

Co-authored-by: Petr Vaněk <petr.vanek@e2b.dev>

jakubno enabled auto-merge (squash) July 7, 2026 15:55

jakubno changed the title ~~fix(api): prevent uint64 underflow in node allocated metrics~~ [EN-1603} fix(api): prevent uint64 underflow in node allocated metrics Jul 7, 2026

jakubno changed the title ~~[EN-1603} fix(api): prevent uint64 underflow in node allocated metrics~~ [EN-1603] fix(api): prevent uint64 underflow in node allocated metrics Jul 7, 2026

jakubno merged commit fed38e1 into main Jul 7, 2026
41 checks passed

jakubno deleted the fix/node-metrics-underflow branch July 7, 2026 16:27

e2b-release-please Bot mentioned this pull request Jul 30, 2026

chore(main): release api 0.0.1 #3469

Merged

e2b-release-please Bot mentioned this pull request Aug 3, 2026

chore(main): release api 0.1.0 #3491

Closed

Navigation Menu

[EN-1603] fix(api): prevent uint64 underflow in node allocated metrics - #3216

[EN-1603] fix(api): prevent uint64 underflow in node allocated metrics#3216
jakubno merged 3 commits into
mainfrom
fix/node-metrics-underflow

jakubno commented Jul 7, 2026 •

edited

Loading

Uh oh!

cursor Bot commented Jul 7, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Jul 7, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

linear-code Bot commented Jul 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jakubno commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cursor Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

Uh oh!

codecov Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

linear-code Bot commented Jul 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jakubno commented Jul 7, 2026 •

edited

Loading

cursor Bot commented Jul 7, 2026 •

edited

Loading

codecov Bot commented Jul 7, 2026 •

edited

Loading