Skip to content

fix(agent): pre-send compaction gate + in-turn recovery for context-window overflow (#193) - #195

Merged
justrach merged 1 commit into
mainfrom
fix/193-presend-compact-gate
Jul 13, 2026
Merged

fix(agent): pre-send compaction gate + in-turn recovery for context-window overflow (#193)#195
justrach merged 1 commit into
mainfrom
fix/193-presend-compact-gate

Conversation

@justrach

Copy link
Copy Markdown
Owner

What

Fixes #193 — a single turn's tool-output burst could push the request input past the model's context window before auto-compaction ever fired, so codex / gpt-5.x rejected with exceeds the context window and the turn died. That makes graff a poor harness exactly when it's working hardest (many large parallel tool reads in one turn).

Root cause

Auto-compaction was gated on last_context_tokens, which only updates from the server's returned usage — so it lags tool output appended within a turn. The between-turns 80% meter (mainloop.zig) never saw the mid-turn spike. graff already had a local estimator (fullInputEstimateTokens, #174) but only used it to correct the meter after a response, never as a pre-send gate.

Fix (two layers, mirroring codex run_pre_sampling_compact and opencode isOverflow / compactAfterOverflow)

  1. Pre-send gate (agent.zig runTurn): estimate the full input locally before every request and compact when it's already ≥ compactAt(), so we never ship an over-cap request. The gate brackets compactOrRecover with closeCodexWs() on both sides — mid-turn the codex WS holds a prev_id / codex_sent_upto watermark keyed to the pre-trim history, so the first reset lets compact()'s own summary request run against a clean session (a stale prev_id makes the server re-prepend the full pre-trim context and defeat/overflow the summary), and the second re-anchors the next in-turn request() to the trimmed full input instead of a delta referencing dropped messages. This matches how every other compactOrRecover call site is bookended by runTurn's closeCodexWs.

  2. In-turn recovery (agent_request.zig): the local estimate is a bytes/4 lower bound, so the backend can still reject. On exceeds the context window, emergency-trim and retry the same request once (single-retry guard), then fall through to the original error — the turn recovers invisibly instead of failing.

Verification

  • zig build test176/176 (adds an inputOverCompactThreshold unit test), zig fmt clean.
  • Adversarial multi-agent review of the diff — which caught the missing closeCodexWs bracket above and drove that fix, then confirmed the fixed version correct.
  • End-to-end drive against a local mock: a would-be-overflow turn now emits [compacting][history compacted] → completes, instead of dying with exceeds the context window.

Known follow-ups (documented on #193, out of scope here)

  • Per-tool-output truncation at ingestion (codex TruncationPolicy / opencode max_bytes) — the in-turn recovery is codex/.responses-only; a dense burst on anthropic/openai that both undercounts and overflows still dies as before (same as pre-fix baseline).
  • The [compacting ~N tokens…] line reads ~0 when the gate fires off the local estimate (the server meter is still 0). Cosmetic.

…indow overflow (#193)

Auto-compaction was gated on last_context_tokens, which only updates from the
server's returned usage, so it lags tool output appended *within* a turn. A
burst of large parallel tool results pushes this turn's input past the model's
window before the between-turns 80% meter (mainloop.zig) ever sees it — codex /
gpt-5.x then rejects with "exceeds the context window" and the turn dies. graff
already had a local estimator (fullInputEstimateTokens, #174) but only used it
to correct the meter after a response, never as a pre-send gate.

Two layers, mirroring codex (run_pre_sampling_compact) and opencode
(isOverflow-before-each-turn / compactAfterOverflow):

1. Pre-send gate (agent.zig runTurn): estimate the full input locally before
   every request and compact when it is already >= compactAt(), so we never
   ship an over-cap request. The gate brackets compactOrRecover with
   closeCodexWs() on both sides — mid-turn the codex WS holds a prev_id /
   codex_sent_upto watermark keyed to the pre-trim history, so (a) the first
   reset lets compact()'s own summary request run against a clean session (a
   stale prev_id makes the server re-prepend the full pre-trim context and
   defeat/overflow the summary), and (b) the second re-anchors the next in-turn
   request() to the trimmed full input instead of a delta referencing dropped
   messages. This matches how every other compactOrRecover call site is
   bookended by runTurn's closeCodexWs.

2. In-turn recovery (agent_request.zig): the local estimate is a byte/4 lower
   bound, so the backend can still reject. On "exceeds the context window",
   emergency-trim and retry the same request once (guarded to a single retry),
   then fall through to the original error — the turn recovers invisibly
   instead of failing. closeCodexWs re-anchors so the retry sends the trimmed
   full input, not a stale WS delta.

Verified: zig build test 176/176 (adds an inputOverCompactThreshold unit test),
zig fmt clean; adversarial multi-agent review of the diff (which caught and
drove the closeCodexWs-bracket fix above); end-to-end drive against a local
mock — a would-be-overflow turn now emits [compacting] -> [history compacted]
-> completes instead of dying with "exceeds the context window".

Co-Authored-By: blackfloofie <265516171+blackfloofie@users.noreply.github.com>
@justrach
justrach merged commit 346c846 into main Jul 13, 2026
6 checks passed
@justrach
justrach deleted the fix/193-presend-compact-gate branch July 13, 2026 08:58
justrach added a commit that referenced this pull request Aug 6, 2026
CI's codex PTY scenario caught the note turn. Investigating it turned up a
real gap and corrected a wrong assumption about where the boundary is.

THE GAP, now closed. #391 is about a PLANNED rollover: context is filling, so
reserve a buffer and spend one call before the window turns over. It is not
about a rescue. Once the provider has rejected a request for exceeding the
window (last_request_context_overflow), or the meter is at the destructive-
recovery boundary where compactOrRecover may drop real history
(Provider.nearContextLimit, 95%), the session has demonstrably run out of room
and a note is the last thing it can afford. decideRoom becomes decideContext
and refuses those outright, BEFORE the token-buffer question — which would
have said yes, because at 95% of a 200k window there are still 10k tokens of
nominal room. contextOf reads the same two signals compactOrRecover uses to
authorize destructive trimming, so the note fires exactly when compaction is
scheduled and never when it is damage control. Two unit tests, one pure and
one through maybeWrite with a live provider.

THE ASSUMPTION, corrected with evidence. run_midturn_compaction_scenario is
NOT the recovery path. test-pty-codex-ws.py sets the server meter to 90% of
the window and codex_ws_test.py says why: "Cross compact@ (80%) but stay below
the destructive recovery boundary (95%)." runTurn's mid-turn gate is
inputOverCompactThreshold (80%), and trim_on_fail is false there. So it is the
planned rollover, and the single most representative instance of #391 in the
tree — a long tool-loop turn crossing the threshold mid-flight, the failure
mode the issue was filed over. Suppressing the note there would suppress it on
the main path.

So the scenario is taught the new shape rather than the note suppressed, and
the #195 WS invariant is CHECKED rather than argued: the note turn is a quiet
tool-less SSE request that opens no WebSocket (connection_id is None), runs
inside runTurn's existing closeCodexWs bracket against a throwaway clone of
history, and leaves the choreography untouched — ws_connections stays 2, the
post-compaction turn still re-anchors on a fresh socket, and no request
carries previous_response_id. Verified by running it, not by reasoning.

Both compaction fixtures are re-keyed on the request's last USER TURN instead
of its ordinal. An ordinal-keyed fixture re-targets silently when compaction
gains a step: the summary reply landed on the note turn, the note reply became
the handoff summary, and midturn still went green while proving something
else. The transactional scenario failed outright for the same reason.

And the scenario now pins #391 end to end on the wire, which no unit test can
reach: the note is absent from `instructions` before it is written, present
VERBATIM in every request after it, and absent from `input` on the
post-compaction turn — state, not conversation, with nothing in the history
for the next compaction to paraphrase away. Confirmed load-bearing by
disabling the injection and watching it fail.

Co-Authored-By: Codegraff <blackfloofie@codegraff.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

codex/gpt-5.x: a single-turn tool-output burst overflows the context window before auto-compaction fires (no pre-send local-estimate gate)

1 participant