[bug] Chat times out at 25s on the tool loop's second round — a 4096-token cap under a 25s non-streaming ceiling is an impossible combination

[serge-ivo]

Not transient, and not the deploy — the tool loop's second round cannot fit a 25s ceiling

Reported live: chat showing AI request timed out (25s).

What actually happened — production, reconstructed from the trace + message history

05:02:36 → 05:04:37   Deploy API Worker (success)
05:04:01              turn OK — "Issue #112 is live"
05:05:49              503 "The service is updating"      ← the deploy, a SEPARATE event
05:08:52              chat 504  AI request timed out (25s)
05:15:38              chat 504  AI request timed out (25s)
05:20:39              turn OK — github_list_pulls → answered

Message history for bd43f4de-… shows the failing turn, twice, with the same shape:

user   | "Start implementing"
system | ✅ github_read_issue { "number": 111, "title": "Feature: Masters Games Training section", … }
system | Error: AI request timed out (25s)

The tool call succeeded. The generation that consumed its result did not. Round 1 (decide to call github_read_issue) returned fast; round 2 — same request, now carrying issue #111's full JSON on top of an already-large coding context — exceeded 25s. The user retried and got the identical failure 6 minutes later.

So: deterministic, reproducible from the same message, and unrelated to the 05:05:49 deploy 503. The chat recovered by itself only because the next turn (github_list_pulls → empty array → one-line answer) was short.

The defect: two constants that contradict each other

lib/user-ai.ts:147-167 — the Anthropic call is not streamed:

const timeoutMs = body.timeoutMs ?? 25_000;
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), timeoutMs);
res = await fetch("https://api.anthropic.com/v1/messages", { …, signal: controller.signal });
…
const data = await res.json();

There is no stream: true anywhere in user-ai.ts. So the 25s must cover the entire generation, not time-to-first-token.

lib/reply-truncation.ts:45 — export const CHAT_MAX_TOKENS = 4096;, passed at agent-think.ts:700.

These cannot both hold. A 4096-token completion from claude-sonnet-4-6 needs roughly 30-70s at realistic throughput. The system is configured to permit a reply it is configured not to wait for. Any turn that runs long fails by construction — retrying reproduces it exactly, which is what happened here.

And the tool loop makes it worse in the precise place users feel it: round 2+ carries the tool result, so it is simultaneously the longest prompt and the round the user reaches after the agent has already done real work (created an issue, read a repo). The side effects land; the answer does not.

Note also that up to 3 rounds × 25s = 75s of provider time inside one request, so the 25s is not protecting a 30s request budget the way a single-call reading of it suggests.

What to do

1. Stream the Anthropic call (the real fix). With stream: true, the deadline becomes time-to-first token plus an inter-chunk stall timeout, instead of total generation time. A long reply then succeeds, and the 4096-token cap stops being a trap. It also removes the dead air the user currently stares at.

This is the same class as #421 — a non-streaming request under a wall-clock ceiling, where the ceiling measures the wrong thing. Worth fixing with the same shape in both places.

2. Until then, make the two constants consistent. Either raise timeoutMs to something a 4096-token reply can actually finish in, or lower CHAT_MAX_TOKENS to what fits 25s. Shipping them contradictory is the bug; either direction is defensible, silently permitting an impossible combination is not.

3. Cap tool results fed back into the prompt. github_read_issue returned a full issue JSON verbatim into round 2. A size cap on tool results before they re-enter the context (with a "truncated" marker) directly shortens the round most likely to time out. Grep found no such cap.

4. Say something actionable. AI request timed out (25s) tells the user nothing they can act on, and "try again" is precisely the wrong advice for a deterministic failure — they took it, and it failed identically. It should say the reply was too long to complete and offer the action that works: ask for a shorter answer, or narrow the request.

5. Do not lose the turn. The user's message and the tool result both survived here (good — better than #420's behaviour). Keep that, and make the failure resumable: the tool result is already in hand, so a retry should be able to re-run only the generation rather than re-calling the tool.

Alternatives considered and rejected

• Treat it as transient and auto-retry. Rejected — this is deterministic. An automatic retry burns the user's own Anthropic credit on a call that cannot succeed, twice as slowly.
• Raise timeoutMs alone. Insufficient on its own: with 3 tool rounds it multiplies, and it does nothing about the dead air. Acceptable only as the stopgap in (2).
• Blame the deploy. The 503 at 05:05:49 is real and is a genuinely different failure (isTransientInfraError → "The service is updating"). Conflating them would send the implementer to the wrong subsystem — the stack is unambiguous: UserAiProviderError … at runAnthropic.

Acceptance criteria

• A turn whose reply approaches CHAT_MAX_TOKENS completes instead of timing out.
• A tool-loop round carrying a large tool result completes, or the result is capped before it re-enters the prompt.
• The timeout and the output cap are consistent, with a test that fails if one is changed without the other.
• The user-facing message names a next action that works, and does not suggest a bare retry for a deterministic failure.

Regression risk

• Streaming changes the response contract for /chat and every caller of runAnthropic (chat, loop-decide, co-pilot, overseer, workflows). Worth doing behind the existing function signature so callers are untouched — collect the stream server-side, keep returning a whole message, and only the timeout semantics change. That gets the fix with none of the client work.
• A tool-result cap can hide information the model needed. Truncate with an explicit marker and keep the head, so the model can ask for more rather than silently reasoning over a fragment.

Related: #421 (the same non-streaming-under-a-deadline shape, in the voice path), #424 (this failure logged correctly — source:"chat" from agent-do.ts:549 — which is why it was reconstructible), #423 (the log noise that made it harder to spot).

[serge-ivo]

Confirmed — third occurrence, and the discriminator is now isolated

A third timeout landed at 05:33:55Z on the same instance, ~25 minutes after the first. All four relevant turns, from the live message history:

Time	User turn	Tool result	Outcome
05:08:52	"Start implementing"	github_read_issue (1658 chars)	timeout
05:15:38	"Start implementing"	github_read_issue (1982 chars)	timeout
05:20:39	"What pull requests are open…"	github_list_pulls (26 chars — [])	OK
05:33:55	"Start implementing the issues that we've filed recently."	github_list_issues (1282 chars)	timeout

Same instance, same model, same account, same key, spread across 25 minutes. 3 of 3 long-form action requests failed; the one short lookup in the middle succeeded.

The discriminator is the OUTPUT, not the input

I framed the original report around round 2's prompt size. That is a contributing factor but it is not the variable that separates these four turns:

• The tool results are 1282-1982 chars — roughly 400-500 tokens. That is nowhere near enough input to cost 25 seconds.
• What actually separates them is what the model was asked to produce. "Start implementing the issues we've filed" is an inherently long-form answer — a plan across multiple issues. "What pull requests are open" is one line, and it returned fine with a tool call in the same shape.

So the failure is generation-length bound, and the arithmetic makes it forced rather than incidental:

CHAT_MAX_TOKENS = 4096              (reply-truncation.ts:45)
timeoutMs       = 25_000, no stream (user-ai.ts:147)

At realistic claude-sonnet-4-6 throughput (~50-80 tok/s) the 25s ceiling admits roughly 1250-2000 output tokens. The configured cap is 4096 — about 2.5× what the timeout can ever wait for. Every reply between the effective ceiling and the configured cap fails deterministically, and the user cannot tell which requests those are until after the agent has already spent their credit and run its tools.

This does not change the fix, but it sharpens the priority: streaming is the fix, not a larger timeout. Raising timeoutMs only moves the invisible cliff; streaming removes it, because the deadline then measures first-token latency and inter-chunk stalls rather than total length.

Additional evidence for fix (5) — the turn is not resumable

Each failure left the tool result persisted and the reply missing. The user retried "Start implementing" and paid for github_read_issue twice (05:08 and 05:15) for identical data already in the thread. A retry that re-ran only the generation would have cost nothing extra and, with streaming, would have succeeded.

Immediate workaround for anyone hitting this now

Ask for a narrower answer — one issue at a time, or "give me the plan for issue #112 only". Short replies complete. This is a workaround for a deterministic bug, not a fix, and it should not be needed.

[serge-ivo]

Items 1–4 are shipped and deployed. Item 5 is not — details and reasoning below.

d5c6d1a — streaming, three deadlines, and the consistency test (items 1, 2, 4)

Done exactly the way the regression note asks: behind the existing runAnthropic signature. The stream is collected server-side and callers still get one whole message, so chat, /loop-decide, the co-pilot, the overseer and both workflows needed no edit at all — the sixty lines below the fetch (tool_use extraction, the #212 four-way usage split, #397's stopReason) are untouched, because lib/anthropic-stream.ts rebuilds byte-for-byte the {content, usage, stop_reason} body the non-streaming endpoint returned.

The single 25s number became three, in lib/ai-deadlines.ts, because "the model is still writing" and "the provider has gone away" are different facts that one number could not tell apart:

first token	25s	nothing yet ⇒ outage; nothing generated, nothing lost
stall	20s of complete silence, re-armed per chunk	Anthropic pings during a long generation, so this is a dead socket, not a thinking model
total	180s	an absolute ceiling, so a runaway cannot hold a request open

Item 2's consistency requirement is a test rather than a comment: generationBudgetMs(CHAT_MAX_TOKENS) <= AI_TOTAL_TIMEOUT_MS, in ai-deadlines.test.ts. The defect was never one wrong number — it was two numbers in two files that nothing related. Raise the cap without raising the ceiling and that test fails, naming both.

Item 4: the three deadlines read differently because they mean different things. first-token invites a retry (there, it works). total says "Sending the same message again will fail the same way" and asks for a shorter or narrower request — asserted by a test that fails if the string ever regains "try again".

Two things deliberately not done, both from the issue's own reasoning: a partial message is never returned when a stream dies mid-reply (that is #397 with a new cause — a half-answer stored and displayed as whole), and a tool_use whose arguments arrive truncated fails rather than executing, because the loop would act on a half-built call.

efe285a — the tool-result backstop (item 3)

One correction to the issue: "grep found no such cap" is true only of a generic one. Per-tool caps do exist and are good — github_read_issue cuts a body at 8KB (github-issues.ts BODY_CAP), repo_read_file 8KB, repo_git 12KB, fetch_url 4,000 chars, an MCP resource read at RESOURCE_MAX_CHARS = 20,000. So this is scoped as a backstop, not a policy: TOOL_RESULT_MAX_CHARS = 24,000, deliberately above the largest deliberate cap, with a test asserting that relationship. Below it, it would silently overrule a judgement someone made on purpose about their own tool's data. It can only fire on a tool that bounded nothing — which is the case that had no answer.

It sits at record in agent-think.ts, the one seam where an outcome re-enters the next round — twice, as prose and keyed by tool_use id, and both are read by the model, so capping one is not capping. Truncation is visible with both numbers and an instruction, per the regression note.

Being straight about what this does and does not fix: the reported timeout was the generation exceeding a total-time deadline, and the streaming is what fixes it. github_read_issue's payload was already capped at 8KB and was not the cause. This bounds the input side, which was genuinely unbounded whether or not it had bitten yet.

Item 5 — resumability — NOT done, and left open

"Do not lose the turn" already holds and still holds (the user message and the tool result both survive). But "a retry should re-run only the generation rather than re-calling the tool" is a real feature, not a corner of this fix: it needs the round's tool_result blocks persisted against the failed turn and a resume path that re-enters agent-think mid-loop. That is separable work and I am not smuggling it in under a timeout ticket. Worth a follow-up issue — say the word and I will have one filed.

Acceptance criteria

• A turn approaching CHAT_MAX_TOKENS completes instead of timing out — user-ai-stall.test.ts proves a stream whose total duration exceeds the stall budget several times over still completes; progress, not duration, keeps the call alive.
• A tool-loop round carrying a large tool result completes, or the result is capped — capped, visibly, at the seam.
• Timeout and output cap consistent, with a test that fails if one moves without the other.
• The message names an action that works and does not suggest a bare retry for the deterministic case.
• (Item 5) Resumable generation — not attempted, see above.

Verified

Both commits: pnpm typecheck (11 projects), pnpm vitest run (6,525 tests), biome, and all seven guard scripts. CI green on both, including pnpm test:e2e (chromium + webkit). Both Deploy API Worker runs succeeded; api.proagentstore.online/health is 200 after each.

Not verified, and worth saying plainly: I could not exercise a real Anthropic generation in production. The streaming path only runs for a signed-in user with a stored BYOK key, and the public trial endpoint has none — POST /v1/public/agents/portal-watch/try returns the expected 402 "Add an API key", which confirms the chat path reaches provider selection normally but proves nothing about the stream itself. The next real chat turn on a BYOK key is the actual test. If a reply now stalls or comes back truncated, it is this change and I will fix it forward.

One honest caveat on the 180s ceiling: raising it can only ever make a request fail later than the old 25s, never earlier, so this is a strict improvement. But a chat response is still a single JSON body, so anything beyond the edge's own idle tolerance would surface as a network-level failure rather than as this message. Streaming to the client is the fix for that, and it is the client work the regression note deliberately scoped out.

[serge-ivo]

Item 5 is now #442, so this can close once you're happy with items 1-4 in production.

One correction I put in the follow-up because it changes the scope: "the tool result survives" is true of the message, but the result itself survives only as a 120-character preview — agent-think.ts:996-998 slices it before pushing to allToolLog, and outcomeById (the full, capped, id-keyed blocks) is function-local and dies with the throw. So the retry isn't choosing to re-call the tool; it has nothing else to work from.

[serge-ivo]

All four acceptance criteria met. Item 5 is #442. Closing.

Verified against main and against the deployed API (d5c6d1a and efe285a are both ancestors of
b6f6c9f6, which "Deploy API Worker" shipped successfully at 13:02 UTC today).

The fix, where it landed

• Streaming (item 1). lib/user-ai.ts:171 sets anthropicBody.stream = true, and the reader at
  :335-366 re-arms a per-chunk budget: the first read carries the first-token deadline, every read
  after it carries the stall deadline. The wait is now bounded by silence, not by length — which
  is what makes the 4,096-token cap stop being a trap. The Workers-AI fallback stays non-streamed and
  says why (:389).
• The two constants made consistent (item 2), and held there. lib/ai-deadlines.ts replaces the
  single 25s with three numbers that each measure something real —
  AI_FIRST_TOKEN_TIMEOUT_MS 25s / AI_STALL_TIMEOUT_MS 20s / AI_TOTAL_TIMEOUT_MS 180s — plus
  generationBudgetMs(), the conversion nobody performed when the original pair was chosen.
• Tool results capped (item 3). efe285a, at the point where the result re-enters the prompt,
  with a marker saying what was dropped — so the round most likely to time out is also the round that
  got shorter.
• The message names an action that works (item 4). deadlineMessage() writes a different sentence
  per deadline, which is the substance: a provider that never started is worth retrying, a reply too
  long to finish never is. The total-budget case now says "Sending the same message again will fail
  the same way. Ask for a shorter answer … or narrow the request" — the opposite of the advice the
  reporter took twice.

The third criterion asked for a test, and got one

lib/ai-deadlines.test.ts:25-30 asserts generationBudgetMs(CHAT_MAX_TOKENS) <= AI_TOTAL_TIMEOUT_MS
and fails naming both numbers. That is the criterion literally — "a test that fails if one is changed
without the other" — and it is the right shape, because the pair drifted apart silently once and
nothing noticed.

Live corroboration

The durable error log carries exactly two AI request timed out (25s) rows, at 05:33:55 and
05:40:38 today — the reported incident. None since the deploy. That is consistent rather than
conclusive (nobody has driven a deliberately long reply through production), so I am recording it as
corroboration, not as proof.

Item 5 (resumability) is #442, and is being worked now

"the tool result is already in hand, so a retry should be able to re-run only the generation" was
split out into #442, which is open and actively assigned. Keeping #427 open as a second home for
it would leave two tickets describing one piece of work — the failure this backlog audit exists to
stop. Closing here; #442 carries it.

Closed.