[bug] Concurrent chat turns on one instance produce two contradictory replies — step 1/40 then step 0/40, 1.8s apart, and one message answered twice while another got no reply

[serge-ivo]

Two chat turns run concurrently on one instance, and the user gets two contradictory answers — progress appearing to go backwards

Observed live in bd43f4de-… ("Chess coder 2") while the owner was working, and reproduced in a second instance a month earlier. Not a stale report — the Chess coder transcript below is from 06:11 UTC today.

Evidence 1 — live, today

06:09:33.668  user       "retry now"
06:11:01.749  user       "https://www.youtube.com"          ← accidental send
06:11:07.951  system     **Loop → engine** (step 1)
06:11:14.671  system     ✅ check_work  … run 47bf3703 : running
06:11:14.671  assistant  "Still running — it's at step 1/40 … The executor is actively working
                          through the implementation."
06:11:16.483  system     ✅ github_list_issues / github_read_issue ×2 / repo_tree / repo_read_file
06:11:16.483  assistant  "The run is still at step 0/40 — the executor is initialising and
                          hasn't produced output yet."

Two assistant messages 1.8 seconds apart, each with its own tool block, saying:

• step 1/40, "actively working through the implementation"
• step 0/40, "initialising and hasn't produced output yet"

Progress runs backwards on screen. The later message reports less progress, because the two turns sampled the run at different moments (turn B started at 06:11:01 and read before step 1 was recorded at 06:11:07; turn A started at 06:09:33 and read after). Both are individually truthful about the instant they sampled; interleaved, they read as the agent contradicting itself.

And neither turn answered the message that started the second one. https://www.youtube.com — an obvious mis-send — got a run-status report instead of "that looks like it was sent by accident". The user's actual input vanished into a reply about something else.

Evidence 2 — the same shape, a month earlier

6d3b28f1-… ("Language Buddy"), timestamps monotonic, ordering correct:

06:46:26.845  user       我现在想不Fairview Park往前走。
06:46:30.903  user       你好！今天我们练习什么？        ← sent 4s later, before any reply
06:46:33.899  assistant  听起来很好！法尔维尤公园很美吗？   ← answers Fairview Park
06:46:36.512  assistant  很好，小明！Fairview Park 听起来很美！ ← ALSO answers Fairview Park

One message got two answers; the other got none. 你好！今天我们练习什么？ ("what are we practising today?") was never replied to.

This is the hands-free case specifically: the mic reopens and the user speaks again before the reply lands, so overlapping turns are not an edge case there — they are the normal rhythm of talking to an agent that takes a few seconds to answer.

Verified: concurrent turns are permitted by design, and nothing reconciles them

• agent-do.ts:379 handleChat awaits think() and returns inline. Grepping the DO for blockConcurrency, inFlight-as-a-lock, mutex, or a busy check returns nothing.
• lib/chat-inflight.ts is not a lock. Its purpose is [bug] A chat turn is bound to the request — navigating away can lose the reply while its tool side effects persist #251 — recording that a turn started so an interrupted one can be named. Its own comment states the intent plainly: "one key per in-flight turn, so concurrent turns can't clobber" — i.e. concurrency is expected and accommodated, not prevented.
• Storage is safe: appendMessage (agent-do.ts:752) writes one key per message (msg:${createdAt}:${id}), so there is no lost update — both replies persist. That is why the failure is contradiction, not data loss.
• Each turn builds its context from history at its own start time, so turn B cannot see turn A's not-yet-written reply. Two turns therefore answer the same state, and the second has no idea the first is in flight.

Why this is worth fixing rather than accepting

The marketplace pitch is an agent you can talk to hands-free. In that mode the user cannot see the screen, so the only signal that a turn is in progress is the reply itself — and speaking again before it arrives is not user error, it is the interface working as designed. Today that produces two answers to one question, none to the other, and a progress number that moves backwards.

It also multiplies the cost: two full BYOK turns, five tool calls in the second one, for one answer the user wanted.

What to do

Three options, cheapest first. I would take (2).

1. Serialize per instance. Queue a turn behind an in-flight one in the DO. Correct, and the simplest to state — but it makes the second speaker wait with no feedback, which in hands-free is indistinguishable from the agent ignoring them.

2. Coalesce (recommended). If a turn is already in flight for this instance, append the new user message to it rather than starting a second turn — the marker in chat-inflight.ts already tells you a turn is live, so the detection is in place. The user gets one reply that accounts for everything they said, which is what a human interlocutor does. you 你好… would have been answered.

3. Make it visible. At minimum, if a second turn does start, tell the second one that another is in flight and let it say so. This does not fix the contradiction; it stops it reading as the agent being confused.

Fix (2) needs a decision on the cut-off: a message arriving after the model has begun generating cannot be folded in, so there is still a window where (1) or (3) applies. Worth specifying rather than discovering.

Alternatives considered and rejected

• Client-side lock — disable send while awaiting a reply. Rejected: the console already holds the composer during a turn, and this still happened, because in hands-free voice sends the turn, not the composer. A UI-level guard cannot cover the surface where the problem actually occurs. It would also block the legitimate "wait, ignore that, I meant…" interjection the Loop feature explicitly supports.
• Treat it as a prompt problem (tell the model to check for concurrent turns). Rejected: the model cannot see a turn that has written nothing yet.
• Order replies by turn start time. Rejected: it fixes the display order and none of the substance — the user would still get two answers to one question.

Acceptance criteria

• Two user messages sent 4 seconds apart, with a reply outstanding, produce one reply that addresses both — or an explicit statement that a turn is already running.
• No two assistant messages report contradictory state for the same run.
• A message that receives no reply is impossible: every user message is either answered or explicitly acknowledged as folded into another turn.
• Regression test at the DO level: two concurrent POST /chat calls on one instance.

Regression risk

• Coalescing changes turn boundaries, which chat-inflight.ts ([bug] A chat turn is bound to the request — navigating away can lose the reply while its tool side effects persist #251) and lib/chat-turns.ts (turnSpanFor) both reason about; the interrupted-turn notice must not start firing for a folded message.
• The Loop orchestrator drives chat autonomously and a human can interject by design ([bug] Muting while a turn is transcribing deletes the words — the bubble is cleared and the in-flight transcript is then dropped or sent by an unrelated 800ms echo timer #420's /loop-decide path). Coalescing must not swallow a human interjection into the loop's own turn — that interjection is specifically meant to interrupt.

Related: #251 (in-flight turn markers — the detection this can reuse), #427 (the timeouts in the same live transcript), #406 (assistant messages asserting state that was not verified).

[serge-ivo]

Fixed and verified in production — a7393bb, plus 0b4afd0 which fixes a defect the first commit introduced and production caught. Both deployed.

Option 2, coalesce. handleChat now submits its turn through a ChatTurnGate (workers/api/src/lib/chat-turn-gate.ts, pure + tested):

• free gate → the turn runs now, byte-for-byte the previous behaviour. The common case pays nothing.
• busy gate → the arrival joins the one follow-up turn that starts when the running one finishes. It works because the user message is appended and broadcast before the gate ever sees it, and think() builds its context from that transcript — so the follow-up sees everything said while the agent was talking and answers it in one BYOK turn rather than one per message.

The cut-off you asked to have decided rather than discovered: a message arriving after the running turn has begun generating cannot join that turn, so it joins the next. Nothing waits for a turn it could never have been part of.

What actually serialises a turn today: nothing. Confirmed before changing anything — the DO input gate opens at every non-storage await, lib/chat-inflight.ts is a marker rather than a lock, and the summarizing flag beside it is the same instance-level shape for the same reason. The gate holds no await of its own, drain() runs from the previous turn's finally, and a turn that throws — including synchronously, before its first await — drains identically, so it cannot wedge. If the object restarts the gate opens, which is the safe direction.

The gate is per DO, and every DO is one conversation with one person (instance = owner-scoped, creator = the owner's template, public trial = keyed by agent + hashed IP). No two people queue behind each other.

The production check found a real regression, fixed forward in 0b4afd0

Two POSTs 4s apart on 6d3b28f1-… — the Language Buddy instance from Evidence 2. The gate worked (turn B started only after A finished, no overlap) and then the turn 400'd:

Anthropic (400): This model does not support assistant message prefill.
The conversation must end with a user message.

Serialising created a history that had never existed: the mid-turn arrival is stored when it arrives, the reply when it finishes, so the tail is user A, user B, assistant A-reply — ending on an assistant turn, which Anthropic reads as a prefill. Fixed at normalizeForAnthropic, which has dropped leading assistant turns for the same class of reason since the beginning; endOnUserTurn is the mirror image at the other end. The last user turn moves rather than the assistant turn being dropped — dropping it would delete the reply the agent just gave from its own context, which is exactly the fact this ticket exists to make visible.

After the fix, same test, same instance:

07:16:49.666  user       "…Reply with exactly one word: ALPHA"
07:16:53.401  user       "…Reply with exactly one word: BRAVO"   ← sent 4s later, reply outstanding
07:16:55.004  assistant  …
07:16:58.542  assistant  …

Two turns, sequential (B's turn ran 415.7→419.1s, after A's 409.1→415.7s — no overlap), both persisted, no error. Compare the pre-fix run at 07:08 in the same transcript: user, user, assistant, system Error: 400.

Acceptance criteria

• Two messages 4s apart with a reply outstanding → every message answered. Both got a reply; before, one got two and the other none.
• No two assistant messages report contradictory state for the same run — turns are serialised, so start order equals finish order and sampling is monotone. This is the structural fix; the step-1/step-0 pair could not recur.
• No message goes unanswered — asserted directly against the Language Buddy failure in chat-turn-gate.test.ts.
• "Regression test at the DO level: two concurrent POST /chat" — delivered as a gate-level test, not a DO-level one, because this repo has no harness that instantiates AgentDO (agent-do.test.ts is types + pure helpers). The 11 tests drive the same scheduler the DO uses, with a harness that snapshots the transcript at turn start exactly as think() does, and assert: two submits never overlap; three mid-turn arrivals produce two turns rather than four; every message answered exactly once; the gate reopens after a sync throw, an async throw and a failed follow-up. The end-to-end pair was checked live instead, above. Flagging this rather than ticking it.

Regression risks you named, both checked. A folded message gets no in-flight marker of its own (markers are written inside runTurn), so #251's interrupted-turn notice cannot start firing for one. A human interjection during a Loop turn is not swallowed — it is in the transcript and the next turn answers it.

Known limitation, stated rather than hidden: a queued caller's HTTP request now waits for the running turn as well as its own, so the worst case roughly doubles. The turn is durable (keepAlive) and the reply lands in the transcript regardless, so a caller that gives up loses the delivery, not the answer. If that becomes visible in practice it wants its own ticket — a queued caller could return an acknowledgement and let the reply arrive over the socket.