Skip to content

fix(v1): end the Bash chat program cleanly on context overflow when it cannot compact - #2633

Merged
mikasenghaas merged 1 commit into
mainfrom
fix/bash-overflow-without-compaction
Sep 19, 2026
Merged

mikasenghaas merged 1 commit into
mainfrom
fix/bash-overflow-without-compaction

Conversation

@parkerpettit

@parkerpettit parkerpettit commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

The bundled chat program re-raises a context-overflow 400 whenever it runs as the Bash harness, on the assumption that Bash compacts. With compaction disabled the Compactor re-raises the overflow untouched, so a Bash rollout that runs out of context exits 1 and is recorded as stop=ProviderError (and retried, if ProviderError is in the retry list) instead of ending with the transcript so far like the Null program does. The same happens with compaction enabled but no threshold, i.e. none given and none discoverable from the provider's model card: the compactor never acts in that state.

Gate the re-raise on whether the compactor can act (enabled with a threshold) instead of on args.bash. When it can, nothing changes: an overflow that reaches the loop is still a real failure, and CompactionFailed still propagates (#2584). No other harness ships this program (browser_use bundles its own).

Checked through the real rollout (interception server, subprocess runtime, scoring) against a scripted fake OpenAI server: Bash without compaction, and Bash with compaction but no discoverable window, now end ok and scored on overflow, like Null; Bash with a threshold still recovers from a mid-run overflow; a rebuilt conversation that still overflows and an unrelated 400 both still fail the rollout.

Note

Fix run_chat_loop context overflow handling based on compaction state

  • Removes args.bash special-casing for API errors in the run_chat_loop conversation loop.
  • Uses the compactor's enabled state and discovered threshold to determine if compaction is available.
  • Ends the run with the transcript on context-overflow errors when compaction is unavailable, and propagates the error when compaction is available. Non-overflow API errors still raise.
  • Behavioral Change: context-overflow errors now raise when compaction is enabled and has a threshold, instead of ending the run with the partial transcript.

Macroscope summarized bc4f235.

@parkerpettit parkerpettit changed the title fix(v1): end the Bash chat program cleanly on context overflow when compaction is off fix(v1): end Bash cleanly on context overflow without compaction, stop the trace with context_length Sep 19, 2026
@parkerpettit
parkerpettit force-pushed the fix/bash-overflow-without-compaction branch from 06a76cd to 63cc520 Compare September 19, 2026 16:44
@parkerpettit parkerpettit changed the title fix(v1): end Bash cleanly on context overflow without compaction, stop the trace with context_length fix(v1): end the Bash chat program cleanly on context overflow when compaction is off Sep 19, 2026
@parkerpettit
parkerpettit marked this pull request as ready for review September 19, 2026 16:59
Comment thread verifiers/v1/harnesses/utils/core.py Outdated
@macroscopeapp

macroscopeapp Bot commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Approved at bc4f235

Macroscope's review found this PR approvable — This one-file fix cleanly distinguishes unavailable compaction from failed compaction, preserving partial transcripts when no threshold exists while still surfacing genuine compaction failures. The change is small, localized, and does not introduce broader API or infrastructure risk.

You can add or adjust custom eligibility rules. Learn more.

…t cannot compact

The bundled chat program re-raised a context-overflow 400 whenever it ran as the
Bash harness, on the assumption that Bash compacts. With compaction disabled the
Compactor re-raises the overflow untouched, so a Bash rollout that ran out of
context crashed with exit 1 and was recorded as a retryable error instead of
ending with the transcript so far, as the Null program does. The same happened
with compaction enabled but no threshold (none given and none discoverable from
the provider's model card): the compactor never acts in that state.

Gate the re-raise on whether the compactor can act (enabled with a threshold)
rather than on the harness: when it cannot, overflow ends the run cleanly for
both programs; when it can, an overflow that reaches the loop is a real failure
and still propagates (CompactionFailed included).
@parkerpettit
parkerpettit force-pushed the fix/bash-overflow-without-compaction branch from 63cc520 to bc4f235 Compare September 19, 2026 17:14
@parkerpettit parkerpettit changed the title fix(v1): end the Bash chat program cleanly on context overflow when compaction is off fix(v1): end the Bash chat program cleanly on context overflow when it cannot compact Sep 19, 2026
@mikasenghaas
mikasenghaas merged commit 35edbc9 into main Sep 19, 2026
13 checks passed
@mikasenghaas
mikasenghaas deleted the fix/bash-overflow-without-compaction branch September 19, 2026 21:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants