Voice: mute is unreachable while the user is speaking in Whisper STT mode — works in dictation, silently not in OpenAI

[serge-ivo]

Problem

The mute voice command works in five of the six voice phases, and fails in the one where users reach for it most: while they are speaking a turn — but only when STT is set to Whisper/OpenAI, which is the default on real instances (sttMode: "openai", sttModel: "gpt-4o-transcribe").

Same words, same settings, different behaviour depending on a transcription setting the user has no reason to connect to it.

The matrix

Phases from derivePhase (packages/sdk/src/voice/machine.ts:77). Control listener runs when engaged && !micOn (use-voice.ts:1094).

Phase	micOn	Control listener	Main-path interim	Mute works?
idle	false	ON	—	✅
listening	true	OFF	dictation: yes · Whisper: none	✅ dictation · ❌ Whisper
transcribing	false	ON	—	✅
processing	false	ON	—	✅
speaking	false	ON	—	✅ (see 3 below)
muted	false	ON	—	✅ (no-op)

Root cause — an assumption from #153 that holds for only one STT mode

#153 built the always-on control listener and had it yield during active capture, on this stated premise:

"it yields to the main recorder while the main path is actively capturing a turn, which already checks control words"

That premise is true for browser dictation — VoiceStt emits onResult(interim, false) live (stt.ts:176-177), which reaches the interim command check at use-voice.ts:680.

It is false for Whisper/OpenAI. That path records with MediaRecorder, and the streaming SSE only produces deltas after the clip is uploaded (stt.ts:272-305). During capture there is no interim at all. So in Whisper mode, for the entire recording window:

• control listener → stopped (micOn is true)
• main-path interim → does not exist yet

Mute is unreachable until end-of-turn. With maxDictationMs: 60000 that can be a full minute, and it only lands at all if the whole utterance was the command (single-word phrases must match the whole utterance, convo.ts:214) — so "okay stop, mute" never fires.

The predicate has no way to know any of this: shouldRunControlListener({ engaged, mainRecording }) (convo.ts:250) yields on micOn alone. The ensureControlStt comment says it should yield "when the main path is browser dictation" — the STT mode was simply never an input.

The words are already on screen and never checked

In Whisper mode the speech gate (gate.ts:84) runs a continuous browser recognizer during capture — for noise filtering — and emits live text. Its handler does exactly one thing:

// use-voice.ts:233-237
onInterim: (text) => {
    if (mutedRef.current || shouldIgnoreResult(readGuard(), Date.now())) return;
    setInterim(text);          // display only — never command-checked
},

So the user watches "mute" appear in the live transcript while nothing happens.

Fix — two options

A (cheapest, no new recognizer): command-check gate.onInterim. The gate already has live words during exactly the window that's broken, and already carries the echo/paused guard. ~3 lines.

B (principled): make the STT mode an input to the predicate — shouldRunControlListener({ engaged, mainRecording, mainUsesBrowserSpeech }) — and yield only when the main path is browser dictation (a real two-recognizer conflict). In Whisper mode the main path is MediaRecorder, so there is nothing to conflict with.

B matches the documented intent; A is less code and avoids a third recognizer competing with the gate. Recommend A, with shouldRunControlListener's doc corrected to say it yields on capture regardless of mode and why.

Secondary inconsistencies in the same code (fix together)

1. No echo guard on the control path. handleControlResult (use-voice.ts:604) applies no echo/paused guard, while gate.onInterim applies shouldIgnoreResult. During speaking the control listener hears the agent's own TTS through the speakers; a recognized utterance of exactly "mute" self-mutes the session. Interrupting during TTS is the whole point of Voice: mute command only active during active recording — must work at all times (during TTS, agent processing, co-pilot chat) #153, so a blanket echo guard is wrong — but the asymmetry should be a decision, not an accident (e.g. suppress only when the text matches what TTS is currently uttering).
2. Two matching semantics in one function. stopSpeechKeyword matches by substring (text.toLowerCase().includes(stopKw)) while mute requires a whole-utterance match. Inert today (default ""), a footgun the moment anyone sets a common word.
3. derivePhase reports "muted" only in hands-free (machine.ts:89). Say "mute" in tap-to-talk and the status pill shows idle; the Mute button is hands-free-only (InstanceDetail.tsx:681, CopilotView.tsx:198), so there is no visible muted state at all. Not a trap — beginTalk clears muted (use-voice.ts:1018), so the next tap self-heals — but the only feedback the user gets is wrong.

Acceptance criteria

• Saying the mute word while actively speaking a turn mutes, in Whisper/OpenAI mode, within the same latency as dictation mode.
• Behaviour is identical across sttMode: browser and sttMode: openai in all six phases.
• It fires mid-utterance ("okay stop, mute"), not only when the command is the entire turn — or the whole-utterance rule is documented as intentional and applied consistently in both modes.
• No regression in the five phases that work today (Voice: mute command only active during active recording — must work at all times (during TTS, agent processing, co-pilot chat) #153's criteria still pass).
• The echo asymmetry in (1) is resolved deliberately, with the chosen behaviour recorded in a comment.
• shouldRunControlListener unit tests cover the STT-mode dimension; convo.test.ts covers mid-utterance matching.

Related

• Voice: mute command only active during active recording — must work at all times (during TTS, agent processing, co-pilot chat) #153 — built the listener; this is the residual gap in its yield condition (closed)
• Voice: unmute command word — dictation-based wake-to-record with configurable keyword and start-recording signal #152 — unmute; same matcher and call sites, will inherit this bug verbatim if fixed first
• Voice: exit hands-free mode command word — return to text mode #165 — exit-hands-free command word; same functions again