Skip to content

fix(agent): context-overflow follow-up — recovery-path parity + local-model wedge (#201-#204) - #207

Closed
justrach wants to merge 3 commits into
fix/193-tool-output-capfrom
fix/overflow-followup
Closed

fix(agent): context-overflow follow-up — recovery-path parity + local-model wedge (#201-#204)#207
justrach wants to merge 3 commits into
fix/193-tool-output-capfrom
fix/overflow-followup

Conversation

@justrach

Copy link
Copy Markdown
Owner

Follow-up to #196, closing the ranked backlog from the context-overflow audit.
Together these break the compound local-model wedge chain end to end: bad window
guess -> undercount -> gate never fires -> needle miss -> no recovery -> hard wedge
(previously escapable only by a manual /compact).

Stacked on #196 (base = fix/193-tool-output-cap); GitHub will retarget this to
main when #196 merges, at which point the Closes keywords fire.

Commits

What changed

Gap Fix
G1 #201 perOutputCap was ~50% of the window each; with keep_recent=4 verbatim, four recent outputs pinned ~2x the window past recovery. Now ~1/8 window + 256 KB ceiling, so 4 recent stay reclaimable.
G3 #202 recordUsage froze the meter on missing usage; now floors at max(fullInputEstimateTokens, body/4) (the fallback the codex path already had).
G6 #202 capOversizedToolOutputs truncated silently; now emits a tracer note + status line.
G10 #202 in-place trims zeroed the meter; now re-estimate the trimmed size.
G8 #203 unknown/local models defaulted to a 200k guess. GRAFF_CONTEXT declares the real window, applied only to non-catalogued models (pricing.isKnownModel).
G4 #203 the pre-send gate estimate omitted system+tools; adds a window-clamped baseline (kept out of fullInputEstimateTokens, which stays pure).
G2 #203 overflow detected only by English substrings; now matches structured error.code first (codex parity), substrings as fallback.
G5 #204 a same-format provider switch kept the old model's token count; now re-estimates.
G9 #204 80% compaction threshold hardcoded; GRAFF_COMPACT_PCT overrides it (both directions).

Design note: the openai/codex research behind the local-model decisions (window
override vs probe vs catalog, structured detection) is captured in #203.

Tests

zig build test 183/183 (was 179; +perOutputCap, +recordUsage-fallback,
+contextWindowFor, +compactAt-override; overflow-detection tests gain structured-code
cases), zig fmt clean.

Closes #201
Closes #202
Closes #203
Closes #204

justrach and others added 3 commits July 14, 2026 09:57
…202)

The #193 follow-up (b227ef8) added the per-output cap + in-turn overflow recovery
for anthropic/openai but left four holes the codex/.responses path had already
closed (#174/#175/#195). This brings the other providers to parity and makes
truncation visible everywhere.

G1 (#201) — recent-output wedge. perOutputCap() was context*2 bytes (~50% of the
window each), but trimOldestToolOutputs keeps keep_recent=4 outputs verbatim, so
four recent large results pinned ~2x the window past what recovery could reclaim
-> hard wedge. Cap is now ~1/8 of the window (context/2 bytes) with a 256 KB
absolute ceiling, so 4 recent outputs occupy ~50% of the window and stay
reclaimable; the ceiling also bounds one pathological result on huge windows
(codex uses a fixed cap with no recency exemption).

G3 (#202) — recordUsage froze the meter when the provider omitted usage (early
`orelse return`), so the between-turns compaction gate could never fire. It now
floors last_context_tokens at max(fullInputEstimateTokens, req_body_len/4), the
same fallback recordUsageResponses already applies (#174). recordUsage now takes
req_body_len; both call sites pass body.len.

G6 (#202) — capOversizedToolOutputs truncated silently (call site discarded the
count). It now emits a tracer note + an interactive status line when it fires;
the model already saw the inline marker.

G10 (#202) — every in-place trim zeroed the meter, opening a transient blind
window and clobbering an overflow recover-pin. Both trim sites now re-estimate
(fullInputEstimateTokens) instead of setting 0.

fullInputEstimateTokens is now member-aliased on Agent so agent_compact can call
it. Tests: +perOutputCap (#201), +recordUsage-fallback (#202); the trimOldest /
capOversized tests now assert the re-measure contract. zig build test 181/181,
zig fmt clean.

Closes #201
Closes #202

Co-Authored-By: blackfloofie <265516171+blackfloofie@users.noreply.github.com>
…red overflow detection + metering parity (#203, #204)

Follows ea230cc. Closes the local/unknown-model wedge cluster (#203) and the two
metering-parity items (#204). Together with #201/#202 this breaks the compound
wedge chain end to end (bad window guess -> undercount -> gate never fires ->
needle miss -> no recovery).

G8 (#203) — unknown/local models defaulted to a 200k guess, mis-sizing every cap
and threshold. GRAFF_CONTEXT / GRAFF_CONTEXT_WINDOW now declares the real window;
contextWindowFor applies it ONLY when the model isn't catalogued (new
pricing.isKnownModel), so a global override can't shrink a known model that
happens to be 200k. Root fix — cascades to compactAt, perOutputCap, and the gate.

G4 (#203) — the pre-send gate estimate omitted the system prompt + tool schemas
and under-fired. inputOverCompactThreshold now adds a baseline for that fixed
prefill, clamped to 1/8 of the window so it can't dominate a small local window.
Kept OUT of fullInputEstimateTokens, which must stay pure over self.messages for
the unit tests.

G2 (#203) — overflow was detected only by ~6 English substrings. isContextOverflow
now matches the structured error.code (context_length_exceeded /
context_window_exceeded) first, so a local or non-English provider whose message
differs still recovers; substrings remain the fallback. error.code is extracted at
the two envelope branches (streamed + non-streamed); codex/.responses and the
generic path pass null (their English messages match). HTTP-status detection was
intentionally not added — codex itself reads the code from the stream, not a 400,
and the code check already covers the local-provider case.

G5 (#204) — a same-format provider switch kept the previous model's absolute token
count against the new window. applyProviderInner now re-estimates the meter from
the (kept/translated) history after the switch.

G9 (#204) — the 80% compaction threshold was hardcoded. GRAFF_COMPACT_PCT overrides
it (1..100), and unlike codex's one-directional clamp it may lower OR raise.

Tests: +contextWindowFor (#203), +compactAt-override (#204); isContextOverflow gains
structured-code cases; recoverContextOverflow / isContextOverflow call sites updated
for the new code param. zig build test 183/183, zig fmt clean.

Closes #203
Closes #204

Co-Authored-By: blackfloofie <265516171+blackfloofie@users.noreply.github.com>
… test (#203)

Extends the G2 work in 3a603d0. The structured-code detection covered the
anthropic-style {"type":"error"} envelopes, but the GENERIC apiErrorMessage path —
where openai-compatible errors ({"error":{...}}, no top-level "type") actually
land — still passed null for the code. So a local provider (LM Studio, mlx, any
openai-compat base_url) whose message text we don't match on would NOT be
recognized as an overflow. New errorCode(root) pulls root.error.code (or a
top-level root.code) and threads it into recoverContextOverflow there.

E2E test (scripts/test-pty-overflow.py): points graff at a mock lmstudio backend
(127.0.0.1:1234) that injects an error and drives a real terminal turn.
  A. code=context_length_exceeded with a Dutch message (matches no English
     substring) -> graff pins the ctx meter to the window (200k/200k, 100%),
     proving detection went through the structured code, and stays responsive
     (runs /help, exits cleanly) instead of wedging.
  B. code=rate_limit_exceeded, non-overflow message -> meter must NOT pin,
     proving detection is precise.
Deterministic across repeated runs; skips only if 127.0.0.1:1234 is already held.

Tests: zig build test 184/184 (+errorCode unit test), zig fmt clean, +E2E PTY test.

Refs #201 #202 #203
@justrach

Copy link
Copy Markdown
Owner Author

Added an end-to-end overflow PTY test (scripts/test-pty-overflow.py) that points graff at a mock lmstudio backend injecting a context_length_exceeded error and drives a real terminal turn.

Writing it caught a real gap in the G2 fix: the structured-error.code detection only covered the anthropic-style {"type":"error"} envelopes. OpenAI-compatible errors are {"error":{...}} (no top-level "type") and land on the generic apiErrorMessage path, which was still passing null for the code — so a local provider (LM Studio / mlx / any openai-compat base_url) whose message text we don't match on would not have been recognized as an overflow. That's the exact case G2 exists for. Fixed with errorCode(root) (aa2b995).

The test proves both directions:

  • code=context_length_exceeded + a Dutch message (matches no English needle) → meter pins to 200k/200k (100%) and the REPL stays responsive → detection went through the structured code, no wedge.
  • code=rate_limit_exceeded + non-overflow message → meter must not pin → detection is precise.

Deterministic across repeated runs (SO_REUSEADDR on the port probe); skips cleanly if 127.0.0.1:1234 is already held by a real LM Studio.

@justrach
justrach deleted the branch fix/193-tool-output-cap July 15, 2026 10:23
@justrach justrach closed this Jul 15, 2026
@justrach
justrach deleted the fix/overflow-followup branch July 15, 2026 10:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant