Voice: mute command only active during active recording — must work at all times (during TTS, agent processing, co-pilot chat)

[serge-ivo]

Problem

The mute (and other) voice control words are only checked inside the transcription paths — real-time interim, hands-free finalize, and tap-to-talk final (packages/sdk/src/voice/use-voice.ts handleResult). All three only run when the main STT is actively recording a user turn.

So the mute word does NOT fire when:

1. The co-pilot / agent is speaking (TTS playing) — the mic is stopped, nothing is transcribed.
2. The agent is working / processing between utterances — the mic is paused (pausedForThinking).
3. The mic is muted — obviously nothing is being recorded.

In practice: exactly the moments a user wants to say "mute" (to shut the agent up while it talks or thinks) are the moments the mute word is unreachable. The command is gated on the recording state machine.

Root cause

Control-word detection is coupled to the main recording pipeline. When that pipeline is stopped/paused (TTS, thinking, muted), there is no listener at all, so control words are never checked.

Fix

A continuous, always-on background control-word listener, separate from the main recording pipeline:

1. A lightweight browser-dictation listener dedicated to control words (mute / stop), active whenever voice is engaged — regardless of app state (TTS playing, agent processing, mic paused/muted).
2. Separate from the main STT so it never interferes with normal recording (it yields to the main recorder while the main path is actively capturing a turn, which already checks control words).
3. Checks spoken words against the mute list at any time.
4. On a mute word: immediately mute the mic AND cancel any TTS playing.
5. Wired through useVoice, so it works in both the agent Assistant chat and the Coder Co-pilot (both consume the same hook).

Acceptance Criteria

• Saying the mute word while the agent is speaking stops the mic and cancels TTS.
• Saying the mute word while the agent is processing (between utterances) mutes.
• Works in co-pilot chat and agent chat (via useVoice).
• The background listener does not double-fire or fight the main recorder during active capture.
• Pure decision logic (when to run the listener; control-word match) is unit-tested.

Related

• Voice: move hands-free control words to the global user profile #129 (mute control words → global profile — the words exist, but only checked during recording)
• Configurable voice control: hands-off command mode (next / play / record) #8 (hands-off voice command mode — broader wake-word epic)
• Voice: unmute command word — dictation-based wake-to-record with configurable keyword and start-recording signal #152 (unmute wake-word — the mirror; a background listener is a shared prerequisite)

[serge-ivo]

Implemented in 79f94d6.

Always-on background control-word listener (packages/sdk/src/voice/use-voice.ts): a lightweight browser Web Speech recognizer (ctrlSttRef), separate from the main recording pipeline, whose only job is to catch a control word at any moment.

• Runs whenever voice is engaged but the main recorder is idle — shouldRunControlListener({ engaged, mainRecording }) = engaged (hands-free or ptt) AND NOT the main mic actively capturing. So it's live while the agent is speaking, processing, or the mic is muted. It yields the mic to the main recognizer during active capture (which already checks control words), so the two never fight: startListening stops it synchronously before opening the main mic, a reconcile effect re-arms it when the main mic idles, and it's disposed on unmount.
• Its onResult only checks control words (never feeds the LLM/STT pipeline): a mute word → muteFromCommand; the stop-speech keyword → cancel TTS while speaking.
• muteFromCommand now also cancels in-flight/queued TTS ("mute" = be quiet), so it interrupts the agent mid-sentence.
• Wired through useVoice → covers both the Assistant chat and the Coder Co-pilot. Degrades gracefully to the existing in-turn detection where Web Speech is absent (iOS Safari).

Multi-word phrases (convo.ts): matchVoiceCommand matches a single-word command only as the whole utterance (precision), but a multi-word phrase like mute mute also as a contiguous whole-word run — and mute alone (half the phrase) does NOT trigger. Delimiter parse splits on comma/newline/semicolon only (never space), so mute mute stays one phrase.

AC status: ✅ mute while speaking (stops mic + cancels TTS) · ✅ mute while processing · ✅ works in co-pilot + agent chat (shared hook) · ✅ yields to the main recorder, no double-fire · ✅ pure logic (shouldRunControlListener, multi-word matchVoiceCommand) unit-tested.

Tests: +multi-word phrase matching, +shouldRunControlListener, +delimiter-preservation. Full voice suite green (170); SDK + console build clean.

Caveat: the background recognizer hears the agent's own TTS while it plays, so a mute phrase spoken by the agent could self-trigger — mitigated by using a distinctive phrase (a two-word mute mute is very unlikely in agent speech). Same tradeoff the existing stop-speech keyword already accepts.

[serge-ivo]

Fixed in 79f94d6. Added an always-on background control-word listener (packages/sdk/src/voice/use-voice.ts) — a lightweight browser Web Speech recognizer, separate from the main recording pipeline, that runs whenever voice is engaged but the main recorder is idle (shouldRunControlListener). So the mute word is now interceptable while the agent is speaking, processing, or the mic is muted — it stops the mic AND cancels TTS. Covers both the Assistant chat and the Coder Co-pilot via useVoice. +tests (multi-word phrase matching, shouldRunControlListener); full voice suite green.