Skip to content

fix(stream): retry mid-assembly chunk timeouts instead of halting - #547

Merged
yogthos merged 1 commit into
mainfrom
stream-timeout-retry
Jun 30, 2026
Merged

fix(stream): retry mid-assembly chunk timeouts instead of halting#547
yogthos merged 1 commit into
mainfrom
stream-timeout-retry

Conversation

@yogthos

@yogthos yogthos commented Jun 30, 2026

Copy link
Copy Markdown
Collaborator

Closes #545.

Problem

When a provider stalled while a tool call was mid-assembly, the stream-chunk timeout fired as a hard error and the run halted (Once I see the error, everything halts, essentially.). A retry layer already existed, but it never engaged: any thinking/text delta streamed before the tool call marked the attempt committed, and the retry layer refused to replay a committed attempt. For reasoning models (e.g. ChatGPT-5.5) that emit thinking before a tool call, a single stall was fatal.

All of the recovery machinery was already in place — the consumer discards the partial assistant message on StreamEvent::Retry (PROV-5) and shows a retry banner. The only thing blocking recovery was the if !committed guard.

Changes

  • src/agent/agent_loop/retry.rs — retry timeout errors even after content has committed. The partial is stalled/incomplete, so discarding it on Retry and restarting is correct. Non-timeout committed errors (hard resets, auth) still surface to avoid duplicating tokens already shown.
  • src/timeout.rsDEFAULT_TOOL_CALL_GAP_SECS 30 → 60. 30s was too tight for reasoning models behind proxies.
  • src/agent/agent_loop/rig_stream.rs — the mid-assembly error now states it is retried automatically and names timeouts.tool_call_gap_secs.
  • docs/config.md — updated default + note that this timeout is auto-retried.

Verification

  • New test retries_timeout_after_content_committed (TDD: red → green).
  • Existing does_not_retry_after_content_committed (connection reset) stays green by design — only timeouts carve out the committed guard.
  • Modules green: agent_loop 523, recovery 33, retry 13, rig_stream 23, timeout 5.
  • cargo clippy --bin dirge and cargo fmt --check clean.

On retry exhaustion (5 attempts) the error still surfaces cleanly via TurnEnd and returns to idle — worst case is a recoverable prompt, not a freeze.

A stream-chunk timeout that fired while a tool call was mid-assembly
surfaced directly instead of retrying: any thinking/text delta streamed
beforehand marked the attempt "committed", and the retry layer refused
to replay a committed attempt. For reasoning models that emit thinking
before a tool call, a single provider stall halted the whole run.

Retry timeout errors even after content commits. The consumer already
discards the partial assistant message on StreamEvent::Retry (PROV-5),
so the next attempt starts clean; non-timeout committed errors still
surface to avoid duplicating tokens already shown.

Also raise the mid-assembly gap timeout default 30s -> 60s (too tight
for reasoning models behind proxies) and make the mid-assembly error
name timeouts.tool_call_gap_secs.
@yogthos
yogthos merged commit 8b0d2bc into main Jun 30, 2026
13 checks passed
yogthos pushed a commit that referenced this pull request Jun 30, 2026
PTY-backed !/!! bang commands (interactive gh/editors, vt100 in-place
render), /prompt <name> <text> runs the text, write_todo_list backed by
the issue board, clipboard copied tooltip, CI clippy gate, visible failed
MCP servers, stream retry on mid-tool-call chunk timeouts, and bounded DB
loads in agent construction. (#538, #539, #540, #541, #542, #544, #546, #547)
yogthos added a commit that referenced this pull request Jun 30, 2026
* fix(stream): don't retry chunk timeouts after text has committed

#547 made stream-chunk timeouts retryable even once content had
committed, on the assumption that StreamEvent::Retry makes the consumer
discard the partial. But that reset (stream.rs PROV-5) only clears the
message history — it doesn't touch text already rendered to the UI. So
attempt 2 re-streamed the whole response and appended a second copy,
producing the duplicate output users saw when a provider stalled
mid-response.

Gate retry on !committed again. The #545 mid-assembly-stall case (a
timeout before any text commits, e.g. a lone tool-call fragment) still
retries; only timeouts after committed text now surface instead of
duplicating.

Also corrects the now-inaccurate "retried automatically" wording in the
timeout error message and docs/config.md, which only holds for the
pre-commit case.

* retry: don't nudge on plain transport stream-chunk timeouts

The stall-recovery nudge ("you may have been stuck in a long reasoning
loop") fired for any pre-commit timeout. For a plain stream-chunk
timeout — "provider stalled or connection silently dropped", no tool
call open — that blame is mis-attributed: it's a transport drop, not a
reasoning loop, and the retry should just re-run clean.

Narrow is_stall_timeout to exclude the "connection silently dropped"
message. Request-level and mid-assembly timeouts (the real stall cases)
still nudge.

---------

Co-authored-by: Yogthos <yogthos@gmail.com>
yogthos pushed a commit that referenced this pull request Jun 30, 2026
[AGENTS] left-panel box listing running subagents (id_short disambiguates
same-profile rows), slash commands unified to a single source of truth,
critic scoped/disabled for read-only and design prompts, and the 0.14.0
(#547) duplicate-output regression fixed by gating stream-chunk retry on
!committed again. Plus a README note on building with newer libclang.
(#548, #549, #552, #553, #554)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Long Running Tasks: error: stream chunk timed out

1 participant