[bug] Spoken words vanish between end-of-turn and the reply — dictation should land in the thread as a pending message with a transcribing status

[serge-ivo]

The report

It is not capturing everything I'm saying. I want everything I say transcribed into the message as
I speak it, in the thread right away. Currently the transcribed text disappears after I submit and
reappears once the LLM has processed it. I want to see what I said (dictation), then the converted
transcription, and a status on the message in the thread saying it is being transcribed.

Where the text actually goes today

The live transcript lives in the composer, never in the thread, and it is destroyed at
end-of-turn:

1. Speaking → streamed partials land in voice.interim, rendered in the textarea
   (InstanceDetail.tsx:786, value={voice.interim || input} — accent, italic, readOnly).
2. End-of-turn → use-voice.ts:312 does setInterim("Transcribing…"). The words you just spoke
   are overwritten by that literal string. This is the disappearance in the report — it is not a
   race or a dropped update, it is an assignment.
3. Final transcript returns → onSend → doSend → only now is a user message appended
   (InstanceDetail.tsx:405).

So there is a window — the whole upload-and-transcribe round trip — where the user's words exist
nowhere on screen. In hands-free that window is exactly when the user is not looking at the
composer and expects to see what was heard.

What is being asked for

Move the utterance into the thread, as a real message with a lifecycle, instead of a transient
composer value:

State	Shows
dictating	live partials, updating as spoken, in a pending user bubble
transcribing	the same bubble, with a "transcribing" status on it
sent	replaced in place by the final transcript
failed	stays, with the failure and the audio retained so nothing is silently lost

doSend already appends optimistically, so the pattern exists — it just starts too late. The bubble
should be created when speech starts, not when transcription ends.

The capture-loss half, and why the fix above is also the diagnostic

Evidence from the Coder Lead thread (all voice turns, all with audioKey):

• "Coder Lead" — a bare fragment, alone as a whole turn.
• "What do they need to do for this model?" → the agent replied "I don't have context on what
  'this model' refers to." The referent is missing, consistent with words being cut.

Candidate causes, all plausible and not currently distinguishable from the outside:

• the VAD ends the turn mid-sentence (silenceMs / sensitivity) and the tail is never recorded;
• the final Whisper transcript replaces the streamed partials, so anything the partials caught
  but the final pass dropped is lost with no trace;
• isTooShortToTranscribe discards a clip that did contain speech.

Right now a lost tail is invisible: the partials are gone (overwritten by "Transcribing…") and
only the final text is ever recorded. Keeping the dictation in the thread makes the loss
observable — if the bubble showed "…verify the application is built on the platform for the MCP"
during dictation and the final message reads "…for this model", the discrepancy is on screen and
reportable instead of a vague feeling that words go missing.

Worth logging both strings (partial-at-end-of-turn vs final) when they diverge materially, at least
behind a debug flag, so the cause can be identified rather than guessed.

Notes

• Voice transcription lost when agent message arrives mid-dictation #175 (turn lost when an agent message arrives mid-dictation) is fixed and recovers text into the
  composer via onRecoveredText. That is the same instinct as this ticket — do not throw away what
  the user said — applied one step earlier.
• The audio is already persisted per turn (audioKey, replay on double-tap), so a failed or
  suspect transcription always has the original to fall back on.

Verification

• Speaking in hands-free shows a pending user bubble in the thread that updates as words arrive.
• The bubble is never blanked; it carries a transcribing status until the final text replaces it.
• A failed transcription leaves the bubble with its partials and the recording, not an empty gap.
• Where the final transcript materially differs from the last partial, the divergence is recorded.

[serge-ivo]

Fixed in 56ca1ae (on main; the Host Worker deploy for that commit succeeded, so it is live).

Was this new, or classifyResult's recover never surfaced?

Genuinely new, and deliberately not the same mechanism — though you were right that it is the same instinct one step earlier.

They sit at different points in the turn's life:

• recover is about a result's DESTINATION. It only exists on the exceptional path: shouldIgnoreResult is already true, and recover rescues a real turn from being dropped because the conversation moved on (the agent replied mid-dictation, or the mode changed). Its answer is "the composer, not the thread" — precisely because firing it blind into a changed thread would be wrong.
• [bug] Spoken words vanish between end-of-turn and the reply — dictation should land in the thread as a pending message with a transcribing status #281 is about the utterance's VISIBILITY, on the ordinary happy path where nothing went wrong. The verdict here is accept. No guard fires. The words vanished before any STT result exists at all — at the VAD end-of-turn, where setInterim("Transcribing…") assigned over them.

So surfacing recover differently would have fixed nothing: on the flow you reported, recover never fires. Your diagnosis in the ticket was exactly right — it is an assignment, not a race, and not a dropped update.

The two now compose rather than overlap: the recover path clears the pending bubble and routes the text to the composer, so a recovered turn is still visible in exactly one place.

The actual root cause

interim was carrying three unrelated things at once: the live words, the "Transcribing…" sentinel, and error notices. Signalling "transcribing" therefore required destroying the words.

Worse, and the reason this was load-bearing: "are we transcribing?" was re-derived by string-comparing that literal in three places — the SDK's own derivePhase, the console's InstanceDetail, and the Coder's CopilotView. Three copies of one decision, held in agreement only by the destruction they described. That is exactly the drift class machine.ts's header exists to prevent, so the status pill and the thread could not have disagreed more structurally.

What changed

Pure layer (machine.ts) — a Dictation (text / status / startedAt / heard / note) advanced by reduceDictation, whose single invariant is that only an explicit clear removes words; a status change never costs the user their speech. Every one of the ~20 imperative sites that used to poke setInterim now names an event and lets the reducer decide the consequence. transcribing is a real boolean read from that state, and the three string-compares are gone.

Behaviours worth calling out, each with a test naming the failure it prevents:

• an empty partial never blanks a live bubble (recognizers emit them constantly between phrases);
• gpt-4o-transcribe streaming partials arriving after end-of-turn update the text but hold the status at transcribing — they are the transcription landing, not the mic reopening;
• a failed turn is finished: new speech starts a fresh one rather than reviving it with the old failure attached;
• with no live gate (iOS) the bubble still opens and says what is happening, rather than leaving the gap.

Consumers — the utterance renders as a pending bubble in the thread in both the Assistant tab and the Coder Co-pilot: Speaking… while words arrive, Transcribing… with a spinner while the clip uploads, and on failure it stays with its words, the reason, and the saved recording, plus a Dismiss. The final transcript still arrives via onSend, which appends the real message and clears the bubble. It auto-scrolls with the thread, but only when you are already at the bottom.

The capture-loss half

dictationDiverged(heard, final) compares what the live gate heard at end-of-turn against the final transcript by volume, not wording — deliberately, because those are two different engines (browser Web Speech vs Whisper) and they disagree about phrasing constantly. Flagging that would make the signal worthless. A final that dropped more than ~40% of a four-word-or-longer utterance is a different shape: a lost tail, which is what "I don't have context on what 'this model' refers to" looks like from the outside.

It always console.warns both strings; it reports to the durable log only behind localStorage['pags:voice-debug'], since this is a diagnostic, not a platform failure, and should not flood the error log. That should let you distinguish your three candidate causes (VAD cutting early vs Whisper dropping the tail vs isTooShortToTranscribe) rather than guess — and per your point, the bubble makes the discrepancy visible on screen at the moment it happens.

Verification against your list

• ✅ speaking in hands-free shows a pending user bubble in the thread that updates as words arrive
• ✅ the bubble is never blanked; it carries a transcribing status until the final text replaces it
• ✅ a failed transcription leaves the bubble with its partials and the recording, not an empty gap
• ✅ where the final materially differs from the last partial, the divergence is recorded

pnpm typecheck, the full unit suite (3660 tests), and biome check at its pre-existing baseline.