fix(api): stop evicting the local node during sync - #2881
Conversation
In local mode the node sync runs with an empty Nomad discovery list, so the statically-connected local orchestrator (NomadNodeShortID="local", reported as Nomad-managed) failed the discovery membership check in syncNode and was deregistered on every ~20s sync cycle. Sandbox creates landing in the eviction/reconnect window then failed placement with "Failed to place sandbox". Sync the local node directly when Nomad syncing is skipped; node health is still tracked via node.Sync.
PR SummaryLow Risk Overview Reviewed by Cursor Bugbot for commit bd996cf. Bugbot is set up for automated code reviews on this repo. Configure here. |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
There was a problem hiding this comment.
Small, well-scoped fix for the local-mode sync loop.
Extended reasoning...
Overview
This PR modifies a single function in packages/api/internal/orchestrator/cache.go. It refactors an if/else into a switch and adds a new case: when skipSyncingWithNomad is true, Nomad-managed nodes are health-synced via node.Sync directly instead of being validated against an empty Nomad discovery list (which always fails, causing the local orchestrator to get Closed and deregistered every ~20s).
Security risks
None. The change is purely about how the existing local node is reconciled in the sync loop. No new external inputs, auth, crypto, or permission boundaries are touched. The behavior for production Nomad and clustered deployments is unchanged (the default branch still calls the original syncNode).
Level of scrutiny
Low. The change is mechanical and the new code path is narrow — it only fires when skipSyncingWithNomad is true (local/single-node mode). node.Sync is already the call used by syncNode and syncClusterNode after their respective membership checks succeed, so this is reusing an established primitive. Health tracking still works because node.Sync marks the orchestrator unhealthy if it is unreachable.
Other factors
- Codecov reports all modified lines are covered and all tests pass.
- PR description provides a clear root-cause analysis (local node has
NomadNodeShortID="local",IsNomadManaged()returns true, empty discovery list ⇒ eviction loop). - Cursor Bugbot also flagged this as low risk.
- The fix addresses a concrete integration-test flake (~42% rate on main), making the value/risk tradeoff strongly in favor of merging.
🤖 I have created a release *beep* *boop* --- ## 0.0.1 (2026-07-30) ### Features * add workspace admin API foundations ([#3314](#3314)) ([0f72030](0f72030)) * **api:** add admin team API key routes ([#2825](#2825)) ([4a1e083](4a1e083)) * **api:** add feature flag to stop accepting E2B access tokens ([#3240](#3240)) ([2cf489b](2cf489b)) * **api:** add sandbox fork endpoint ([#3202](#3202)) ([643d726](643d726)) * **api:** add sandbox IAM workload token configuration ([13ddb3d](13ddb3d)) * **api:** add sandbox workload identity permission ([#3319](#3319)) ([13ddb3d](13ddb3d)) * **api:** add user agent integration attribution to PostHog events ([#3303](#3303)) ([d83be18](d83be18)) * **api:** discover orchestrators via nomad service ([#3176](#3176)) ([32af250](32af250)) * **api:** e2b access token deprecation feature flag rename ([#3110](#3110)) ([ebc2daa](ebc2daa)) * **api:** enforce blocked-team restrictions at mutating API endpoints ([#2659](#2659)) ([db848ab](db848ab)) * **api:** filter snapshots by name ([#3184](#3184)) ([6fa1bc7](6fa1bc7)) * **api:** gate access token issuance behind feature flag ([#3101](#3101)) ([2f7811e](2f7811e)) * **api:** LD-gated ClickHouse read switcher ([#3061](#3061)) ([29e74ca](29e74ca)) * **api:** limit template build name to 128 characters ([#3109](#3109)) ([84aa186](84aa186)) * **api:** paginated GET /v2/templates (EN-603) ([#3059](#3059)) ([91e02e4](91e02e4)) * **api:** per-region volume type defaults from node-derived region ([#3435](#3435)) ([baf5559](baf5559)) * **api:** pin resume retries to the node a previous resume timed out on ([#3066](#3066)) ([a4fd0f2](a4fd0f2)) * **api:** SOCKS5 egress proxy on sandbox network config (BYOP) ([#2642](#2642)) ([1fc3820](1fc3820)) * **api:** soft-delete build layers in DB on user delete ([#3121](#3121)) ([ee88776](ee88776)) * **auth:** support admin token team auth ([#2934](#2934)) ([5496666](5496666)) * dynamic sandbox log routing and ClickHouse-backed log reads ([#3236](#3236)) ([1b19a3b](1b19a3b)) * **evictor:** make max concurrent evictions a feature flag ([#2727](#2727)) ([0b33013](0b33013)) * **metrics:** distinguish joined from regular requests (ENG-4072) ([#2699](#2699)) ([390e296](390e296)) * **observability:** add kill_reason to sandbox.lifecycle.killed ([#2833](#2833)) ([e45418f](e45418f)) * **observability:** include kill_reason in kill-path structured logs ([#2846](#2846)) ([33c49f7](33c49f7)) * **orchestrator:** add dummy orchestrator binary for local API dev ([#2744](#2744)) ([ab56e25](ab56e25)) * **orchestrator:** report hugepage metrics to API ([#3182](#3182)) ([7735bae](7735bae)) * **orchestrator:** track and report last status change timestamp ([#2980](#2980)) ([f79be77](f79be77)) * **otel:** instrument auth service HTTP client with otelhttp ([#2722](#2722)) ([69b085d](69b085d)) * per-team events TTL limit (tier + addons) ([#3181](#3181)) ([f76b2cb](f76b2cb)) * **storage:** stamp provenance custom metadata on uploaded objects (incl. headers) ([#3033](#3033)) ([ba8604e](ba8604e)) ### Bug Fixes * added api and orch ([#3454](#3454)) ([fda5e45](fda5e45)) * **api:** check template alias tags in exists endpoint ([#2916](#2916)) ([9574cdf](9574cdf)) * **api:** copy auth/internal into api and dashboard-api image builds ([#3323](#3323)) ([bda1fee](bda1fee)) * **api:** discover the local orchestrator as a template builder ([#3386](#3386)) ([9ea005a](9ea005a)) * **api:** expose pagination headers via CORS ([#3388](#3388)) ([e832b1e](e832b1e)) * **api:** handle corrupted data in sandbox stop time ([#3203](#3203)) ([a98a178](a98a178)) * **api:** include exhaustion reason in "Node exhausted" placement warning ([#3279](#3279)) ([eb3797b](eb3797b)) * **api:** invalidate auth cache on API key deletion ([#3324](#3324)) ([8b02910](8b02910)) * **api:** keep API alive until in-flight requests finish ([#2708](#2708)) ([06378c7](06378c7)) * **api:** let the analytics collector address carry a port ([#3394](#3394)) ([6d41cb5](6d41cb5)) * **api:** parse the pause body regardless of Content-Length ([#3056](#3056)) ([d66aab8](d66aab8)) * **api:** prevent uint64 underflow in node allocated metrics ([#3216](#3216)) ([fed38e1](fed38e1)) * **api:** push api and db-migrator images to both latest and commit S… ([#2951](#2951)) ([6f010fc](6f010fc)) * **api:** reject non-positive timeout on sandbox create, resume, and fork ([#3419](#3419)) ([b672bd1](b672bd1)) * **api:** report invalid tag errors as bad requests ([#2799](#2799)) ([10085a1](10085a1)) * **api:** stop evicting the local node during sync ([#2881](#2881)) ([5455905](5455905)) * **api:** use correct error variable in processCustomErrors ([#3135](#3135)) ([a131a00](a131a00)) * **auth:** rename X-Team-Id header to X-Team-ID ([#2723](#2723)) ([f92ecc0](f92ecc0)) * correct 3 CVES ([#3218](#3218)) ([076823b](076823b)) * **orchestrator:** reject standby while draining ([#3325](#3325)) ([475a7ee](475a7ee)) * Support snapshots for non-default clusters ([#2947](#2947)) ([28eeb72](28eeb72)) ### Performance Improvements * **api:** wake reservation waiters via pub/sub instead of 20ms polling [ENG-4070] ([#2729](#2729)) ([2944d06](2944d06)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: e2b-release-please[bot] <298072688+e2b-release-please[bot]@users.noreply.github.com>
🤖 I have created a release *beep* *boop* --- ## 0.0.1 (2026-07-30) ### Features * add workspace admin API foundations ([#3314](#3314)) ([0f72030](0f72030)) * **api:** add admin team API key routes ([#2825](#2825)) ([4a1e083](4a1e083)) * **api:** add feature flag to stop accepting E2B access tokens ([#3240](#3240)) ([2cf489b](2cf489b)) * **api:** add sandbox fork endpoint ([#3202](#3202)) ([643d726](643d726)) * **api:** add sandbox IAM workload token configuration ([13ddb3d](13ddb3d)) * **api:** add sandbox workload identity permission ([#3319](#3319)) ([13ddb3d](13ddb3d)) * **api:** add user agent integration attribution to PostHog events ([#3303](#3303)) ([d83be18](d83be18)) * **api:** discover orchestrators via nomad service ([#3176](#3176)) ([32af250](32af250)) * **api:** e2b access token deprecation feature flag rename ([#3110](#3110)) ([ebc2daa](ebc2daa)) * **api:** enforce blocked-team restrictions at mutating API endpoints ([#2659](#2659)) ([db848ab](db848ab)) * **api:** filter snapshots by name ([#3184](#3184)) ([6fa1bc7](6fa1bc7)) * **api:** gate access token issuance behind feature flag ([#3101](#3101)) ([2f7811e](2f7811e)) * **api:** LD-gated ClickHouse read switcher ([#3061](#3061)) ([29e74ca](29e74ca)) * **api:** limit template build name to 128 characters ([#3109](#3109)) ([84aa186](84aa186)) * **api:** paginated GET /v2/templates (EN-603) ([#3059](#3059)) ([91e02e4](91e02e4)) * **api:** per-region volume type defaults from node-derived region ([#3435](#3435)) ([1bded44](1bded44)) * **api:** pin resume retries to the node a previous resume timed out on ([#3066](#3066)) ([a4fd0f2](a4fd0f2)) * **api:** SOCKS5 egress proxy on sandbox network config (BYOP) ([#2642](#2642)) ([1fc3820](1fc3820)) * **api:** soft-delete build layers in DB on user delete ([#3121](#3121)) ([ee88776](ee88776)) * **auth:** support admin token team auth ([#2934](#2934)) ([5496666](5496666)) * dynamic sandbox log routing and ClickHouse-backed log reads ([#3236](#3236)) ([1b19a3b](1b19a3b)) * **evictor:** make max concurrent evictions a feature flag ([#2727](#2727)) ([0b33013](0b33013)) * **metrics:** distinguish joined from regular requests (ENG-4072) ([#2699](#2699)) ([390e296](390e296)) * **observability:** add kill_reason to sandbox.lifecycle.killed ([#2833](#2833)) ([e45418f](e45418f)) * **observability:** include kill_reason in kill-path structured logs ([#2846](#2846)) ([33c49f7](33c49f7)) * **orchestrator:** add dummy orchestrator binary for local API dev ([#2744](#2744)) ([ab56e25](ab56e25)) * **orchestrator:** report hugepage metrics to API ([#3182](#3182)) ([7735bae](7735bae)) * **orchestrator:** track and report last status change timestamp ([#2980](#2980)) ([f79be77](f79be77)) * **otel:** instrument auth service HTTP client with otelhttp ([#2722](#2722)) ([69b085d](69b085d)) * per-team events TTL limit (tier + addons) ([#3181](#3181)) ([f76b2cb](f76b2cb)) * **storage:** stamp provenance custom metadata on uploaded objects (incl. headers) ([#3033](#3033)) ([ba8604e](ba8604e)) ### Bug Fixes * added api and orch ([#3454](#3454)) ([d56e0a8](d56e0a8)) * **api:** check template alias tags in exists endpoint ([#2916](#2916)) ([9574cdf](9574cdf)) * **api:** copy auth/internal into api and dashboard-api image builds ([#3323](#3323)) ([bda1fee](bda1fee)) * **api:** discover the local orchestrator as a template builder ([#3386](#3386)) ([9ea005a](9ea005a)) * **api:** expose pagination headers via CORS ([#3388](#3388)) ([e832b1e](e832b1e)) * **api:** handle corrupted data in sandbox stop time ([#3203](#3203)) ([a98a178](a98a178)) * **api:** include exhaustion reason in "Node exhausted" placement warning ([#3279](#3279)) ([eb3797b](eb3797b)) * **api:** invalidate auth cache on API key deletion ([#3324](#3324)) ([8b02910](8b02910)) * **api:** keep API alive until in-flight requests finish ([#2708](#2708)) ([06378c7](06378c7)) * **api:** let the analytics collector address carry a port ([#3394](#3394)) ([6d41cb5](6d41cb5)) * **api:** parse the pause body regardless of Content-Length ([#3056](#3056)) ([d66aab8](d66aab8)) * **api:** prevent uint64 underflow in node allocated metrics ([#3216](#3216)) ([fed38e1](fed38e1)) * **api:** push api and db-migrator images to both latest and commit S… ([#2951](#2951)) ([6f010fc](6f010fc)) * **api:** reject non-positive timeout on sandbox create, resume, and fork ([#3419](#3419)) ([5a4b631](5a4b631)) * **api:** report invalid tag errors as bad requests ([#2799](#2799)) ([10085a1](10085a1)) * **api:** stop evicting the local node during sync ([#2881](#2881)) ([5455905](5455905)) * **api:** use correct error variable in processCustomErrors ([#3135](#3135)) ([a131a00](a131a00)) * **auth:** rename X-Team-Id header to X-Team-ID ([#2723](#2723)) ([f92ecc0](f92ecc0)) * correct 3 CVES ([#3218](#3218)) ([076823b](076823b)) * **orchestrator:** reject standby while draining ([#3325](#3325)) ([475a7ee](475a7ee)) * Support snapshots for non-default clusters ([#2947](#2947)) ([28eeb72](28eeb72)) ### Performance Improvements * **api:** wake reservation waiters via pub/sub instead of 20ms polling [ENG-4070] ([#2729](#2729)) ([2944d06](2944d06)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: e2b-release-please[bot] <298072688+e2b-release-please[bot]@users.noreply.github.com>
Summary
In local/single-node mode the node sync loop runs with an empty Nomad discovery list (
skipSyncingWithNomad). The statically-connected local orchestrator is registered withNomadNodeShortID="local", soIsNomadManaged()is true and it was validated against that empty list insyncNode— failing the membership check and gettingClosed + deregistered on every ~20s sync cycle.Sandbox creates landing in the eviction/reconnect window (or racing the connection close) then fail placement with
500 "Failed to place sandbox". This is the most likely driver of the high integration-test flake rate (~42% on main) where unrelated tests intermittently get that 500 / a follow-up 502.The fix syncs the local node directly when Nomad syncing is skipped, instead of evicting it against a list it can never appear in. Node health is still tracked by
node.Sync(it is marked unhealthy if the orchestrator is unreachable), so a genuinely dead orchestrator is still handled.Test plan
go test ./packages/api/internal/orchestrator/...Failed to place sandbox.