Skip to content

fix(api): stop evicting the local node during sync - #2881

Merged
ValentaTomas merged 1 commit into
mainfrom
fix/local-node-sync-eviction
Jun 1, 2026
Merged

ValentaTomas merged 1 commit into
mainfrom
fix/local-node-sync-eviction

Conversation

@ValentaTomas

Copy link
Copy Markdown
Member

Summary

In local/single-node mode the node sync loop runs with an empty Nomad discovery list (skipSyncingWithNomad). The statically-connected local orchestrator is registered with NomadNodeShortID="local", so IsNomadManaged() is true and it was validated against that empty list in syncNode — failing the membership check and getting Closed + deregistered on every ~20s sync cycle.

Sandbox creates landing in the eviction/reconnect window (or racing the connection close) then fail placement with 500 "Failed to place sandbox". This is the most likely driver of the high integration-test flake rate (~42% on main) where unrelated tests intermittently get that 500 / a follow-up 502.

The fix syncs the local node directly when Nomad syncing is skipped, instead of evicting it against a list it can never appear in. Node health is still tracked by node.Sync (it is marked unhealthy if the orchestrator is unreachable), so a genuinely dead orchestrator is still handled.

Test plan

  • go test ./packages/api/internal/orchestrator/...
  • Watch integration-test flake rate on Failed to place sandbox.

In local mode the node sync runs with an empty Nomad discovery list, so the
statically-connected local orchestrator (NomadNodeShortID="local", reported
as Nomad-managed) failed the discovery membership check in syncNode and was
deregistered on every ~20s sync cycle. Sandbox creates landing in the
eviction/reconnect window then failed placement with "Failed to place
sandbox". Sync the local node directly when Nomad syncing is skipped; node
health is still tracked via node.Sync.
@cla-bot cla-bot Bot added the cla-signed label Jun 1, 2026
@cursor

cursor Bot commented Jun 1, 2026 •

Copy link
Copy Markdown

PR Summary

Low Risk
Narrow orchestrator sync-path change for local mode only; production Nomad sync behavior is unchanged.

Overview
Fixes local/single-node mode where the periodic node pool sync treated the static orchestrator as Nomad-managed and checked it against an empty discovery list, so it was closed and deregistered about every 20 seconds and sandbox placement could fail with intermittent 500 Failed to place sandbox. When Nomad syncing is skipped, Nomad-managed nodes in the pool are now health-synced with node.Sync instead of membership validation against Nomad, while cluster nodes and full Nomad deployments keep their existing sync paths.

Reviewed by Cursor Bugbot for commit bd996cf. Bugbot is set up for automated code reviews on this repo. Configure here.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

There are no critical findings to report.

@codecov

codecov Bot commented Jun 1, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 10 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
packages/api/internal/orchestrator/cache.go 0.00% 10 Missing ⚠️

📢 Thoughts on this report? Let us know!

@ValentaTomas
ValentaTomas marked this pull request as ready for review June 1, 2026 03:29
@ValentaTomas
ValentaTomas enabled auto-merge (squash) June 1, 2026 03:30

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small, well-scoped fix for the local-mode sync loop.

Extended reasoning...

Overview

This PR modifies a single function in packages/api/internal/orchestrator/cache.go. It refactors an if/else into a switch and adds a new case: when skipSyncingWithNomad is true, Nomad-managed nodes are health-synced via node.Sync directly instead of being validated against an empty Nomad discovery list (which always fails, causing the local orchestrator to get Closed and deregistered every ~20s).

Security risks

None. The change is purely about how the existing local node is reconciled in the sync loop. No new external inputs, auth, crypto, or permission boundaries are touched. The behavior for production Nomad and clustered deployments is unchanged (the default branch still calls the original syncNode).

Level of scrutiny

Low. The change is mechanical and the new code path is narrow — it only fires when skipSyncingWithNomad is true (local/single-node mode). node.Sync is already the call used by syncNode and syncClusterNode after their respective membership checks succeed, so this is reusing an established primitive. Health tracking still works because node.Sync marks the orchestrator unhealthy if it is unreachable.

Other factors

  • Codecov reports all modified lines are covered and all tests pass.
  • PR description provides a clear root-cause analysis (local node has NomadNodeShortID="local", IsNomadManaged() returns true, empty discovery list ⇒ eviction loop).
  • Cursor Bugbot also flagged this as low risk.
  • The fix addresses a concrete integration-test flake (~42% rate on main), making the value/risk tradeoff strongly in favor of merging.

@ValentaTomas
ValentaTomas merged commit 5455905 into main Jun 1, 2026
51 checks passed
@ValentaTomas
ValentaTomas deleted the fix/local-node-sync-eviction branch June 1, 2026 03:52
charlie-e2b pushed a commit that referenced this pull request Jul 30, 2026
🤖 I have created a release *beep* *boop*
---


## 0.0.1 (2026-07-30)


### Features

* add workspace admin API foundations
([#3314](#3314))
([0f72030](0f72030))
* **api:** add admin team API key routes
([#2825](#2825))
([4a1e083](4a1e083))
* **api:** add feature flag to stop accepting E2B access tokens
([#3240](#3240))
([2cf489b](2cf489b))
* **api:** add sandbox fork endpoint
([#3202](#3202))
([643d726](643d726))
* **api:** add sandbox IAM workload token configuration
([13ddb3d](13ddb3d))
* **api:** add sandbox workload identity permission
([#3319](#3319))
([13ddb3d](13ddb3d))
* **api:** add user agent integration attribution to PostHog events
([#3303](#3303))
([d83be18](d83be18))
* **api:** discover orchestrators via nomad service
([#3176](#3176))
([32af250](32af250))
* **api:** e2b access token deprecation feature flag rename
([#3110](#3110))
([ebc2daa](ebc2daa))
* **api:** enforce blocked-team restrictions at mutating API endpoints
([#2659](#2659))
([db848ab](db848ab))
* **api:** filter snapshots by name
([#3184](#3184))
([6fa1bc7](6fa1bc7))
* **api:** gate access token issuance behind feature flag
([#3101](#3101))
([2f7811e](2f7811e))
* **api:** LD-gated ClickHouse read switcher
([#3061](#3061))
([29e74ca](29e74ca))
* **api:** limit template build name to 128 characters
([#3109](#3109))
([84aa186](84aa186))
* **api:** paginated GET /v2/templates (EN-603)
([#3059](#3059))
([91e02e4](91e02e4))
* **api:** per-region volume type defaults from node-derived region
([#3435](#3435))
([baf5559](baf5559))
* **api:** pin resume retries to the node a previous resume timed out on
([#3066](#3066))
([a4fd0f2](a4fd0f2))
* **api:** SOCKS5 egress proxy on sandbox network config (BYOP)
([#2642](#2642))
([1fc3820](1fc3820))
* **api:** soft-delete build layers in DB on user delete
([#3121](#3121))
([ee88776](ee88776))
* **auth:** support admin token team auth
([#2934](#2934))
([5496666](5496666))
* dynamic sandbox log routing and ClickHouse-backed log reads
([#3236](#3236))
([1b19a3b](1b19a3b))
* **evictor:** make max concurrent evictions a feature flag
([#2727](#2727))
([0b33013](0b33013))
* **metrics:** distinguish joined from regular requests (ENG-4072)
([#2699](#2699))
([390e296](390e296))
* **observability:** add kill_reason to sandbox.lifecycle.killed
([#2833](#2833))
([e45418f](e45418f))
* **observability:** include kill_reason in kill-path structured logs
([#2846](#2846))
([33c49f7](33c49f7))
* **orchestrator:** add dummy orchestrator binary for local API dev
([#2744](#2744))
([ab56e25](ab56e25))
* **orchestrator:** report hugepage metrics to API
([#3182](#3182))
([7735bae](7735bae))
* **orchestrator:** track and report last status change timestamp
([#2980](#2980))
([f79be77](f79be77))
* **otel:** instrument auth service HTTP client with otelhttp
([#2722](#2722))
([69b085d](69b085d))
* per-team events TTL limit (tier + addons)
([#3181](#3181))
([f76b2cb](f76b2cb))
* **storage:** stamp provenance custom metadata on uploaded objects
(incl. headers) ([#3033](#3033))
([ba8604e](ba8604e))


### Bug Fixes

* added api and orch
([#3454](#3454))
([fda5e45](fda5e45))
* **api:** check template alias tags in exists endpoint
([#2916](#2916))
([9574cdf](9574cdf))
* **api:** copy auth/internal into api and dashboard-api image builds
([#3323](#3323))
([bda1fee](bda1fee))
* **api:** discover the local orchestrator as a template builder
([#3386](#3386))
([9ea005a](9ea005a))
* **api:** expose pagination headers via CORS
([#3388](#3388))
([e832b1e](e832b1e))
* **api:** handle corrupted data in sandbox stop time
([#3203](#3203))
([a98a178](a98a178))
* **api:** include exhaustion reason in "Node exhausted" placement
warning ([#3279](#3279))
([eb3797b](eb3797b))
* **api:** invalidate auth cache on API key deletion
([#3324](#3324))
([8b02910](8b02910))
* **api:** keep API alive until in-flight requests finish
([#2708](#2708))
([06378c7](06378c7))
* **api:** let the analytics collector address carry a port
([#3394](#3394))
([6d41cb5](6d41cb5))
* **api:** parse the pause body regardless of Content-Length
([#3056](#3056))
([d66aab8](d66aab8))
* **api:** prevent uint64 underflow in node allocated metrics
([#3216](#3216))
([fed38e1](fed38e1))
* **api:** push api and db-migrator images to both latest and commit S…
([#2951](#2951))
([6f010fc](6f010fc))
* **api:** reject non-positive timeout on sandbox create, resume, and
fork ([#3419](#3419))
([b672bd1](b672bd1))
* **api:** report invalid tag errors as bad requests
([#2799](#2799))
([10085a1](10085a1))
* **api:** stop evicting the local node during sync
([#2881](#2881))
([5455905](5455905))
* **api:** use correct error variable in processCustomErrors
([#3135](#3135))
([a131a00](a131a00))
* **auth:** rename X-Team-Id header to X-Team-ID
([#2723](#2723))
([f92ecc0](f92ecc0))
* correct 3 CVES ([#3218](#3218))
([076823b](076823b))
* **orchestrator:** reject standby while draining
([#3325](#3325))
([475a7ee](475a7ee))
* Support snapshots for non-default clusters
([#2947](#2947))
([28eeb72](28eeb72))


### Performance Improvements

* **api:** wake reservation waiters via pub/sub instead of 20ms polling
[ENG-4070] ([#2729](#2729))
([2944d06](2944d06))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: e2b-release-please[bot] <298072688+e2b-release-please[bot]@users.noreply.github.com>
jakubno pushed a commit that referenced this pull request Aug 3, 2026
🤖 I have created a release *beep* *boop*
---


## 0.0.1 (2026-07-30)


### Features

* add workspace admin API foundations
([#3314](#3314))
([0f72030](0f72030))
* **api:** add admin team API key routes
([#2825](#2825))
([4a1e083](4a1e083))
* **api:** add feature flag to stop accepting E2B access tokens
([#3240](#3240))
([2cf489b](2cf489b))
* **api:** add sandbox fork endpoint
([#3202](#3202))
([643d726](643d726))
* **api:** add sandbox IAM workload token configuration
([13ddb3d](13ddb3d))
* **api:** add sandbox workload identity permission
([#3319](#3319))
([13ddb3d](13ddb3d))
* **api:** add user agent integration attribution to PostHog events
([#3303](#3303))
([d83be18](d83be18))
* **api:** discover orchestrators via nomad service
([#3176](#3176))
([32af250](32af250))
* **api:** e2b access token deprecation feature flag rename
([#3110](#3110))
([ebc2daa](ebc2daa))
* **api:** enforce blocked-team restrictions at mutating API endpoints
([#2659](#2659))
([db848ab](db848ab))
* **api:** filter snapshots by name
([#3184](#3184))
([6fa1bc7](6fa1bc7))
* **api:** gate access token issuance behind feature flag
([#3101](#3101))
([2f7811e](2f7811e))
* **api:** LD-gated ClickHouse read switcher
([#3061](#3061))
([29e74ca](29e74ca))
* **api:** limit template build name to 128 characters
([#3109](#3109))
([84aa186](84aa186))
* **api:** paginated GET /v2/templates (EN-603)
([#3059](#3059))
([91e02e4](91e02e4))
* **api:** per-region volume type defaults from node-derived region
([#3435](#3435))
([1bded44](1bded44))
* **api:** pin resume retries to the node a previous resume timed out on
([#3066](#3066))
([a4fd0f2](a4fd0f2))
* **api:** SOCKS5 egress proxy on sandbox network config (BYOP)
([#2642](#2642))
([1fc3820](1fc3820))
* **api:** soft-delete build layers in DB on user delete
([#3121](#3121))
([ee88776](ee88776))
* **auth:** support admin token team auth
([#2934](#2934))
([5496666](5496666))
* dynamic sandbox log routing and ClickHouse-backed log reads
([#3236](#3236))
([1b19a3b](1b19a3b))
* **evictor:** make max concurrent evictions a feature flag
([#2727](#2727))
([0b33013](0b33013))
* **metrics:** distinguish joined from regular requests (ENG-4072)
([#2699](#2699))
([390e296](390e296))
* **observability:** add kill_reason to sandbox.lifecycle.killed
([#2833](#2833))
([e45418f](e45418f))
* **observability:** include kill_reason in kill-path structured logs
([#2846](#2846))
([33c49f7](33c49f7))
* **orchestrator:** add dummy orchestrator binary for local API dev
([#2744](#2744))
([ab56e25](ab56e25))
* **orchestrator:** report hugepage metrics to API
([#3182](#3182))
([7735bae](7735bae))
* **orchestrator:** track and report last status change timestamp
([#2980](#2980))
([f79be77](f79be77))
* **otel:** instrument auth service HTTP client with otelhttp
([#2722](#2722))
([69b085d](69b085d))
* per-team events TTL limit (tier + addons)
([#3181](#3181))
([f76b2cb](f76b2cb))
* **storage:** stamp provenance custom metadata on uploaded objects
(incl. headers) ([#3033](#3033))
([ba8604e](ba8604e))


### Bug Fixes

* added api and orch
([#3454](#3454))
([d56e0a8](d56e0a8))
* **api:** check template alias tags in exists endpoint
([#2916](#2916))
([9574cdf](9574cdf))
* **api:** copy auth/internal into api and dashboard-api image builds
([#3323](#3323))
([bda1fee](bda1fee))
* **api:** discover the local orchestrator as a template builder
([#3386](#3386))
([9ea005a](9ea005a))
* **api:** expose pagination headers via CORS
([#3388](#3388))
([e832b1e](e832b1e))
* **api:** handle corrupted data in sandbox stop time
([#3203](#3203))
([a98a178](a98a178))
* **api:** include exhaustion reason in "Node exhausted" placement
warning ([#3279](#3279))
([eb3797b](eb3797b))
* **api:** invalidate auth cache on API key deletion
([#3324](#3324))
([8b02910](8b02910))
* **api:** keep API alive until in-flight requests finish
([#2708](#2708))
([06378c7](06378c7))
* **api:** let the analytics collector address carry a port
([#3394](#3394))
([6d41cb5](6d41cb5))
* **api:** parse the pause body regardless of Content-Length
([#3056](#3056))
([d66aab8](d66aab8))
* **api:** prevent uint64 underflow in node allocated metrics
([#3216](#3216))
([fed38e1](fed38e1))
* **api:** push api and db-migrator images to both latest and commit S…
([#2951](#2951))
([6f010fc](6f010fc))
* **api:** reject non-positive timeout on sandbox create, resume, and
fork ([#3419](#3419))
([5a4b631](5a4b631))
* **api:** report invalid tag errors as bad requests
([#2799](#2799))
([10085a1](10085a1))
* **api:** stop evicting the local node during sync
([#2881](#2881))
([5455905](5455905))
* **api:** use correct error variable in processCustomErrors
([#3135](#3135))
([a131a00](a131a00))
* **auth:** rename X-Team-Id header to X-Team-ID
([#2723](#2723))
([f92ecc0](f92ecc0))
* correct 3 CVES ([#3218](#3218))
([076823b](076823b))
* **orchestrator:** reject standby while draining
([#3325](#3325))
([475a7ee](475a7ee))
* Support snapshots for non-default clusters
([#2947](#2947))
([28eeb72](28eeb72))


### Performance Improvements

* **api:** wake reservation waiters via pub/sub instead of 20ms polling
[ENG-4070] ([#2729](#2729))
([2944d06](2944d06))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: e2b-release-please[bot] <298072688+e2b-release-please[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants