fix: preserve retries for transient egress ledger conflicts by danny-avila · Pull Request #179 · LibreChat-AI/code-interpreter

danny-avila · 2026-09-11T19:09:48Z

Summary

I preserve retries for transient egress-ledger conflicts while keeping genuine authorization denials fail-fast. The gateway currently uses HTTP 403 for both scope denials and exhausted Redis WATCH conflicts; after #177 stopped all 403 retries, a temporary ledger conflict could terminate input preparation immediately.

Return HTTP 503 with ledger_conflict and Retry-After when optimistic ledger updates exhaust their retries.
Expose X-CodeAPI-Error-Code on classified gateway errors so the runner can distinguish permanent scope denials.
Stop HTTP 403 retries for classified gateway denials and direct-server denials; retain retries for unclassified responses from older gateways in both marker discovery and downloads.
Forward gateway error classification and retry hints through the worker relay while keeping unrelated headers filtered.
Honor bounded gateway retry hints during object downloads while preserving exponential backoff and cancellation.
Preserve HTTP 401 fail-fast behavior, batch failure accounting, and transient download retries.

old gateway -> unclassified 403 -> preserve legacy retry
new gateway -> 503 ledger_conflict -> retry
new gateway -> 403 scope_mismatch -> abort input batch

The additive header and retained legacy retry path support both gateway-first and runner-first rollouts. This does not change authorization scope or grant budgets.

Change Type

Bug fix (non-breaking change which fixes an issue)

Testing

Run cd api && bun test src/download.test.ts src/job-cleanup.test.ts src/inline-prime-atomicity.test.ts src/session-inputs.prime.test.ts: 39 passed.
Run cd service && bun test src/egress-gateway.test.ts src/egress-ledger.test.ts: 34 passed.
Force Redis transaction conflicts through the actual gateway/ledger path: verify HTTP 503, retry classification, unchanged accounting, and no upstream file request.
Verify classified 403 aborts a 240-file gateway batch while legacy 403 recovers on retry, including marker discovery before priming.
Compile relay source/tests with TypeScript and run with Node: 12 passed, including classified-denial and ledger-conflict forwarding.
Run package workspace typechecking: 11 diagnostics from unavailable native dependencies also reproduce on the untouched baseline; targeted relay compilation passes.
Run npx tsc --noEmit in API and service: eight API and six service diagnostics also reproduce on the untouched baseline with the same dependencies, with no additional diagnostics (line-number shifts normalized).
Run git diff --check: passed.

Test Configuration

macOS, Bun 1.3.13, existing API/service dependencies; local HTTP listeners and Redis transaction-conflict injection.

Checklist

My code adheres to this project's style guidelines
I have performed a self-review of my own code
I have commented in any complex areas of my code
I have written tests demonstrating that my changes are effective or that my feature works
Local unit tests pass with my changes

danny-avila · 2026-09-11T19:11:36Z

@codex review the current remote head 16d6ac3. Please check retry classification, mixed-version compatibility, and gateway error accounting.

chatgpt-codex-connector · 2026-09-11T19:11:55Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-11T19:27:29.013247Z	`17e5a0b`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 16d6ac3fb4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

danny-avila · 2026-09-11T19:18:47Z

@codex review the current PR head af22f55. Marker discovery, direct-server denials, and relay header propagation are covered by new regression tests. Please confirm this exact commit is reviewed.

danny-avila · 2026-09-11T19:19:22Z

@codex review the current PR head 7c4cf3d. This includes the reviewed fixes plus explicit gateway configuration in classified-denial regression cases. Please review this exact head rather than the preceding revision.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7c4cf3d335

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

danny-avila · 2026-09-11T19:25:01Z

@codex review the current PR head 17e5a0b. I audited the gateway/relay/marker/download/cancellation path together. Object downloads now honor the same bounded numeric retry hint as marker discovery without shortening exponential backoff; tests cover the one-second conflict delay and cancellation while waiting. Please confirm this exact head is reviewed.

chatgpt-codex-connector · 2026-09-11T19:27:27Z

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit: 17e5a0be14

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

* fix: fail denied input downloads once per batch (LibreChat-AI#177) * fix: preserve retries for transient egress ledger conflicts (LibreChat-AI#179) * fix: distinguish retryable ledger contention from scope denials * fix: preserve error classification through marker discovery and relay * test: exercise classified denials through gateway configuration * fix: honor bounded gateway retry hints during object downloads * perf: reuse authorized input versions and make egress accounting atomic (LibreChat-AI#180) * perf: reuse authorized input versions and make egress accounting atomic * perf: resolve authorized input manifests once per execution * test: preserve fetch signature in revocation fixture * fix: isolate shared-grant failures and prevent ledger replay * 🧹 fix: Evict Stale File-Object Index Entries (LibreChat-AI#182) * fix: evict stale file-object index entries Forget cached locators after successful deletion and missing-object downloads so replacement keys resolve immediately. Reuse exact resolver matching in the delete route to avoid prefix collisions. Fixes LibreChat-AI#181 * fix: keep file-object deletion storage-authoritative * fix: retire superseded upload objects * fix: canonicalize replacement object keys * fix: collapse legacy object-key siblings * fix: recover reads from stale locators * fix: namespace canonical object identities * perf: enable authorized input reuse by default (LibreChat-AI#183) * fix: honor requested input destinations (LibreChat-AI#184) * fix: disambiguate legacy dotted object identities (LibreChat-AI#186) * fix: helm egress deployment getting stuck on install (LibreChat-AI#176) * fix: helm egress deployment getting stuck on install * only wait for redis if the ledger is required * Update helm/codeapi/templates/egress-gateway-deployment.yaml Co-authored-by: Danny Avila <danacordially@gmail.com> --------- Co-authored-by: Danny Avila <danacordially@gmail.com> * feat: Add Trusted VM Command Policy (LibreChat-AI#187) * 🛰️ feat: Add trusted VM command policy * docs: clarify trusted VM socket boundary * fix: Retry Clean Cancelled BYOM Settlements Through Stop Grace (LibreChat-AI#188) A clean atomic workspace mutation rejection was settled with retries cut off at the original execution deadline. When Stop arrived near that deadline, process-tree termination finished after it, so the first settlement attempt was aborted immediately while Code API was still draining the cancellation. The client received ASSIGNMENT_EXPIRED and the durable mutation guard stayed armed. Route clean workspace mutation rejections through the known-clean rejection recovery path: a transient-retrying heartbeat and settlement retries through the rejection acknowledgement grace, floored at the bridge cancellation settlement grace. Worker shutdown still fails closed. Share the grace constant from the protocol module so the bridge and worker stay aligned. Closes LibreChat-AI#173 * fix: Release Unassigned Workspace Slots When Dispatch Cleanup Fails (LibreChat-AI#189) Closes LibreChat-AI#170 * fix: Exit Cleanly When Native Executor Shuts Down Concurrently (LibreChat-AI#191) * fix: Exit Cleanly When Native Executor Shuts Down Concurrently A native BYOM worker under systemd KillMode=control-group receives SIGTERM at the same time as its forked SRT executor. The child ignores IPC once it is shutting down, so the parent's close handshake is left pending until the child exits, which rejects it with 'Native executor is unavailable'. That rejection escaped the CLI finally block and turned an idle administrative stop into exit status 1. Treat the close handshake as best-effort: the executor is terminated in finally regardless, and the active command has already drained, so a lost or stalled reply carries no mutation risk. Also make the child report exit status 0 when its own SRT teardown succeeded. Closes LibreChat-AI#190 * fix: Surface Explicit Executor Cleanup Failures During Close Only a lost, refused, or stalled close handshake is benign at shutdown. A negative close reply from the executor is a real cleanup failure and still rejects so pool shutdown can aggregate it. * fix: authenticate GitHub App Git operations (LibreChat-AI#192) * fix: isolate file deletion rate limits (LibreChat-AI#193) * feat: broker GitHub CLI authentication (LibreChat-AI#194) * feat: Run PTC in Selected BYOM Workspaces (LibreChat-AI#195) * feat: run PTC in selected BYOM workspaces * fix: harden native workspace PTC replay * fix: preserve replay isolation and bridge limits * test: tolerate hosts without filesystem cloning * test: surface copy-on-write clone faults * fix: harden native workspace PTC admission * fix: close native replay effect and finalization boundaries * feat: Report Truncated Output Artifacts (LibreChat-AI#199) * feat: report truncated output artifacts * fix: classify omitted artifacts precisely * fix: preserve artifact scan invariants * fix: bound depth truncation probes * fix: bound capped directory enumeration * fix: constrain truncation probes across the job * fix: stop exhausted artifact probes * fix: cancel selected-workspace PTC across processes (LibreChat-AI#196) * fix: cancel replay jobs across API and worker processes * fix: drain worker cancellation watches promptly * fix: close programmatic cancellation races * fix: preserve cancellation response ordering * fix: close distributed cancellation races * fix: harden cancellation under concurrent load * fix: make cancellation ownership durable through completion * fix: recover durable replay outcomes across lost replies * fix: return atomic cancellation outcomes with aligned retention * fix: commit native results inside the workspace mutation fence * fix: claim programmatic execution before stalled-job redelivery * fix: classify capped artifact probe candidates (LibreChat-AI#206) * feat: Report Deleted Code Session Files (LibreChat-AI#200) * fix: report deleted persisted files * fix: reconcile deletions across code runtimes * fix: preserve protected session inputs * fix: classify reserved runtime paths * fix: preserve trusted jq path for PTC (LibreChat-AI#207) * fix: isolate native PTC readiness and watchdog phases (LibreChat-AI#208) * feat: Declare Named Worker Project Environments (LibreChat-AI#209) * feat: declare named worker project environments * fix: preserve environment trust and negotiated action boundaries * Harden environment loading and executor identity * Protect environment root traversal and exact config bytes * Reject self-controlled environment root aliases * Check filesystem identities at environment trust boundaries * Validate environment containment across Linux mount aliases * Handle stacked mounts conservatively without blocking unrelated paths * fix: Allow Trusted Own-Root Environment Symlinks (LibreChat-AI#212) * fix: Allow Trusted Own-Root Environment Symlinks * fix: Check Alias Parent Ownership by Filesystem Identity * fix: Enforce Parent Ownership Across Every Environment Path * fix: Retain Quarantine After Failed Environment Setup (LibreChat-AI#213) * fix: Retain Quarantine After Failed Environment Setup * test: Run Native Environment Setup Lifecycle in CI * fix: Document and Verify Local Setup Quarantine Recovery * fix: Allow Lambda MicroVM metadata in hardened mode (LibreChat-AI#215) * docs: add self-hosted worker setup runbook (LibreChat-AI#219) * fix: bound memory buffering for streamed file uploads (LibreChat-AI#218) * ci: Automate Auditable Main Releases (LibreChat-AI#216) * fix: decouple repository release versions * ci: automate releases after successful main builds * fix: forward input-file limit into sandbox guests (LibreChat-AI#217) * feat: Inspect Local Coding Projects (LibreChat-AI#221) * feat: add bounded local project inventory * fix: report incomplete Git metadata reads * fix: preserve incomplete discovery and remote identities * fix: finalize discovery budgets and nested remote identities * fix(code): stop project traversal at filesystem budget boundaries * fix: make automatic release tip check read-only (LibreChat-AI#225) * feat: Discover Bounded Repository Instructions for Attached Workspaces (LibreChat-AI#226) * Discover bounded repository instructions for opted-in workspaces * Verify snapshot digests and cross-platform confinement * feat: Register Selected Coding Projects (LibreChat-AI#222) * feat(code): register explicitly selected project roots * fix(code): reject shared Git metadata for selected projects * fix(code): pin selected project identity through executor admission * fix(code): preserve full filesystem identity precision * Check selected project identity before replay staging * Anchor replay copies to the verified working directory * fix: Bind Selected Project Operations to Held Directory Descriptors * test: Cover Selected Project PTC and Load Native Fixtures Before Platform Simulation * fix: Keep Native Root Bindings Worker-Local and Verify Directory Ancestry * fix: Anchor Project Admission and Preserve Search Permissions * fix: Report workspace admission capacity without ambiguous timeout errors (LibreChat-AI#227) * fix: Distinguish workspace admission capacity from execution expiry * test: Preserve execution uncertainty while classifying blocked follow-ups * fix: Classify admission expiry at the enqueue boundary * ci: Fix Release Version Resolution for Untagged and Resumed Runs (LibreChat-AI#233) * ci: Fix Release Version Resolution for Untagged and Resumed Runs The release workflow resolved its version in one inline shell block under `set -euo pipefail`, where two paths could not succeed. Filtering tags through `grep` made a no-match fatal. On the ordinary untagged tip of `main`, `git tag --points-at HEAD | grep -E '^v[0-9]+...'` exits 1, and the step died before reaching its skip handling or `next-release-version.sh`, so a deployable commit could not obtain a release version (LibreChat-AI#228). Selecting stable tags now reads exit 1 as an empty answer while exit 2 and above still fail the release, which also lets the missing-previous-tag case report its own error. The rerun-resume path then rejected the tag it had itself chosen. With a stable tag already pointing at `HEAD` and no release published, the version comes from that tag, and the following existence check failed merely because the ref existed (LibreChat-AI#229). It now compares the tag's commit against the release commit, so only a tag on some other commit is a collision; `Create tag` already tolerates a tag that exists. The block moved into `.github/scripts/resolve-release-version.sh`, beside the `next-release-version.sh` it calls, so `tests/release-version-resolution.sh` can cover every path: automatic, resumed, skipped, dispatched, pushed-tag, and the runs that must be refused, each against a throwaway repository with a stubbed `gh`. * fix: harden release resolver execution --------- Co-authored-by: Lia <lia@librechat.ai> Co-authored-by: Danny Avila <danny@librechat.ai> * feat: Route GitHub App credentials per repository (LibreChat-AI#236) * feat: Route GitHub App credentials per repository * test: Make repository routing assertion deterministic * fix: Harden repository credential routing * fix: Bound shared GitHub credential refreshes * fix: Bind GitHub credentials to admitted workspaces * fix: Authenticate GitHub App bot identity lookup (LibreChat-AI#237) * feat: Advertise Workspace Command Timeout Ceiling (LibreChat-AI#238) * 🌳 feat: Provision Conversation-Scoped Code Worktrees (LibreChat-AI#239) * feat: provision conversation-scoped code worktrees * fix: isolate conversation checkout metadata * docs: clarify isolated conversation checkouts * fix: revalidate conversation checkout sources * fix: preserve synchronous legacy execution startup * fix: harden conversation worktree lifecycle * test: use canonical workspace isolation keys * fix: secure conversation worktree provisioning * fix: preserve isolated workspace lifecycle * fix: harden conversation worktree provisioning * fix: fence worktree setup and credential routing * fix: use kernel-backed provisioning locks * fix: load worktree locking only when provisioned * fix: retain conversation provisioning ownership through recovery * fix: reserve provisioning before launching checkout writers * fix: pin Git provisioning inputs and close instance admission gaps * fix: Close native scratch directory streams (LibreChat-AI#240) * feat(code): route trusted VM GitHub App tokens by checkout (LibreChat-AI#248) * feat(code): route trusted VM GitHub App tokens by checkout * fix(code): resolve checkout credentials within canonical root * fix(code): reject cross-root credential aliases --------- Co-authored-by: Danny Avila <danny@librechat.ai> Co-authored-by: Ignaz "Ian" Kraft <ignaz.k@live.de> Co-authored-by: Danny Avila <danacordially@gmail.com> Co-authored-by: Jackson Riding <99007683+jacksonriding@users.noreply.github.com> Co-authored-by: lia-by-librechat[bot] <328778573+lia-by-librechat[bot]@users.noreply.github.com> Co-authored-by: Lia <lia@librechat.ai> Co-authored-by: busla <3162968+busla@users.noreply.github.com>

fix: distinguish retryable ledger contention from scope denials

16d6ac3

danny-avila mentioned this pull request Sep 11, 2026

fix: fail denied input downloads once per batch #177

Merged

5 tasks done

chatgpt-codex-connector Bot reviewed Sep 11, 2026

View reviewed changes

Comment thread api/src/job.ts Outdated

Comment thread api/src/job.ts Outdated

Comment thread api/src/job.ts Outdated

fix: preserve error classification through marker discovery and relay

af22f55

test: exercise classified denials through gateway configuration

7c4cf3d

chatgpt-codex-connector Bot reviewed Sep 11, 2026

View reviewed changes

Comment thread service/src/egress-gateway.ts

fix: honor bounded gateway retry hints during object downloads

17e5a0b

danny-avila merged commit 794df9e into main Sep 11, 2026
10 checks passed

Navigation Menu

fix: preserve retries for transient egress ledger conflicts - #179

fix: preserve retries for transient egress ledger conflicts#179
danny-avila merged 4 commits into
mainfrom
danny-avila/codeapi-ledger-retry-classification

danny-avila commented Sep 11, 2026 •

edited

Loading

Uh oh!

danny-avila commented Sep 11, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 11, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

danny-avila commented Sep 11, 2026

Uh oh!

danny-avila commented Sep 11, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

danny-avila commented Sep 11, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

danny-avila commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Change Type

Testing

Test Configuration

Checklist

Uh oh!

danny-avila commented Sep 11, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

danny-avila commented Sep 11, 2026

Uh oh!

danny-avila commented Sep 11, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

danny-avila commented Sep 11, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

danny-avila commented Sep 11, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Sep 11, 2026 •

edited

Loading