Skip to content

fix(api): add per-stage node-rejection breakdown to placement failures - #3554

Open
AdaAibaby wants to merge 1 commit into
e2b-dev:mainfrom
AdaAibaby:fix/placement-error-diagnosis
Open

AdaAibaby wants to merge 1 commit into
e2b-dev:mainfrom
AdaAibaby:fix/placement-error-diagnosis

Conversation

@AdaAibaby

@AdaAibaby AdaAibaby commented Aug 9, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #3553

Rebased onto the placement error taxonomy (placement/errors.go + orchestrator/placement_errors.go) that has since landed on main. That taxonomy already classifies placement failures (timeout / no-capacity / no-compatible-node / unsupported-feature / create-failed), maps each to a semantic error_code and HTTP status, and forwards a curated, safe client message. This PR fills the two observability gaps it left, without touching what the client sees.

What

  1. FailedToPlaceSandboxError now says why the fleet had no candidate. Previously it only listed the requirements (machine, labels, features). sample() now counts how many nodes each filter stage discarded — excluded, not-accepting, cpu-incompatible, feature-incompatible, label-filtered — and the error appends that breakdown.
  2. The no-eligible-node return path in placeSandbox now logs. It was the only failure return with no logger.L() call (the create-failed and ResourceExhausted paths already log), so a sandbox_no_compatible_node failure left no trace in structured logs. Added a logger.L().Error with sandbox/template/build IDs and the full breakdown.

Why this does not re-open the topology-exposure concern

placementAPIError returns a fixed "Failed to place sandbox: no compatible node for this template's requirements" for this class and never calls err.Error(). The rejection counts and CPU details live only in the internal Err field, the OTel span, and the new log line — visible to operators, never to callers.

Example log / internal error

no node available with required metadata: machine={...}, labels=[gpu]; rejected 38 node(s): 0 excluded, 0 not-accepting, 38 cpu-incompatible, 0 feature-incompatible, 0 label-filtered

Changes

  • placement/placement_best_of_K.go — add nodeRejectionCounts, tally per stage in sample(), thread into FailedToPlaceSandboxError, append breakdown in Error()
  • placement/placement.go — log at the no-eligible-node site
  • placement/placement_best_of_K_test.go — cover the breakdown in Error(), add sample() rejection-count test and end-to-end chooseNode propagation test

Testing

go test ./internal/orchestrator/placement/... passes (39 tests). go vet clean. Client-facing TestPlacementAPIError still passes, confirming the breakdown does not leak into ClientMsg.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ffeca63463

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread packages/api/internal/orchestrator/create_instance.go Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3a6ab5b69e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread packages/api/internal/orchestrator/create_instance.go Outdated
@leonmeijer

Copy link
Copy Markdown

This change is already incorporated on main by 063aa20, including typed and sanitized placement-failure reasons plus regression tests. Merging this branch would duplicate the current implementation, so this PR can be closed as superseded.

The placement error taxonomy classifies a no-candidate placement as
FailedToPlaceSandboxError, but the error only listed the requirements
(machine, labels, features), not why the fleet failed to meet them. That
return path in placeSandbox was also the only one with no structured log
line, so a 'no compatible node' failure left no trace in Loki.

Count how many nodes each filter stage in sample() rejected (excluded,
not-accepting, cpu-incompatible, feature-incompatible, label-filtered),
thread the tally into FailedToPlaceSandboxError, and append it to the
error message. Add a logger.L().Error at the no-eligible-node site with
sandbox/template/build IDs and the full breakdown.

The breakdown stays internal: placementAPIError returns a fixed
'no compatible node for this template's requirements' for this class and
never calls err.Error(), so node counts and CPU details are not exposed
to callers.
@AdaAibaby
AdaAibaby force-pushed the fix/placement-error-diagnosis branch from 1468cfe to 884b23d Compare September 29, 2026 06:54
@AdaAibaby AdaAibaby changed the title fix(api): expose sandbox placement failure reason in error message fix(api): add per-stage node-rejection breakdown to placement failures Sep 29, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

api: "Failed to place sandbox" gives no diagnostic - expose why no node accepted the sandbox

3 participants