[bug] STT transcribes silence in hands-free mode — Whisper hallucinations posted as phantom user turns ("Pottery Barn", "Thank you for watching") — gate fails on iOS Safari and when SpeechRecognition stalls

[serge-ivo]

Context / Why

Two confirmed phantom user messages sent from the "Heartfull (tmux)" instance
(cda75e28-cace-4958-ac3e-6a7528e6b719) during hands-free auto-capture:

• 2026-08-10T09:16:24Z — audioKey: 7c4ea5a2-6d73-4ad0-a5d5-fdef5f419da8

  "Pottery Barn Please visit www.potterybarn.com for more ideas and inspiration. Please visit www.potterybarn.com for more ideas and inspiration."

• 2026-08-07T23:38Z — audioKey: 65ae0a63-…

  "Thank you for watching!"

Both are role:user messages with audioKey set — they came through the Whisper path.
Both are canonical gpt-4o-transcribe silence hallucinations: the model was trained on
captioned web/video audio so on near-silence or steady noise it emits promotional or
sign-off boilerplate.

A phrase blocklist is NOT the fix. SILENCE_HALLUCINATIONS in audio.ts already contains
"thank you for watching" — the second phantom should have been caught by isNoiseTranscript.
That it was NOT means the problem is upstream: the gate that is supposed to discard the clip
before it reaches Whisper at all is either absent or non-functional. The blocklist is
whack-a-mole ("Pottery Barn" could never be anticipated), breaks across 16 supported languages,
and can eat real speech. It is explicitly ruled out as a primary fix here; if it is widened
at all it is only a secondary backstop.

Root-cause analysis

1. The gate — what it does and why it can be bypassed

packages/sdk/src/voice/gate.ts (createSpeechGate) runs Web Speech API alongside the
Whisper MediaRecorder to answer "did real words happen?" at end-of-turn. If heardSpeech()
is false and isAlive() is true, endOfTurnAction (machine.ts:145) returns "discard"
and use-voice.ts:448 calls sttRef.current?.stopDiscard() — the clip is dropped, never
uploaded to Whisper.

The gate is null on iOS Safari (use-voice.ts:310 — speechGateAvailable() returns
false when window.SpeechRecognition is absent). On iOS the entire branch is skipped and
gateRef.current stays null. When gateSnapshot() returns null,
endOfTurnAction(null) (machine.ts:145–147) returns "transcribe" — the turn is uploaded
regardless of whether the user spoke. On iOS, hands-free has no pre-upload speech gate.

The gate can be dead without the caller knowing. createSpeechGate returns a live object
even if the underlying SpeechRecognition never fires a result (e.g. the browser grants
construction but silently fails to start the engine — a known Chrome edge case). In that state
isAlive() returns false and endOfTurnAction again returns "transcribe"
(machine.ts:146 — the condition is gate?.isAlive && !gate.heardSpeech; a non-alive gate
passes through to "transcribe"). This is deliberate — a dead gate that blocks speech would
be worse — but it means a stalled gate reverts the protection to nothing without any log row.

2. The existing hadSpeech / _peakLevel check and its gap

stt.ts:340 contains a speech gate at the MediaRecorder onstop handler:

if (this._peakLevel > 0 && !hadSpeech(this._peakLevel)) {
    this.onEnd();
    return;
}

hadSpeech (vad.ts:64) checks peakLevel > VOICE_FLOOR where VOICE_FLOOR = 0.1.
noteLevel is called every audio frame and accumulates the max. This should catch a
completely silent clip. However:

• The check is conditional on _peakLevel > 0. If noteLevel was never called (the
  audio monitor didn't start, or tore down before the recorder's onstop fired),
  _peakLevel stays 0 and the branch is skipped entirely (use-voice.ts:426 feeds it,
  but the monitor is stopped before onstop fires in some teardown orderings). The comment
  at stt.ts:338 already acknowledges: "Skipped when nothing ever called noteLevel(), so a
  caller with no analyser keeps working rather than going silently deaf."

• The threshold is fixed at 0.1. A steady-noise environment (fan, HVAC, traffic) can
  sustain peakLevel above 0.1 indefinitely — every frame qualifies as "not silent" — so
  the clip uploads. The VAD is adaptive to YOUR VOICE (speakFrac = 0.35 / max(0.4, \ sensitivity)), but hadSpeech is a fixed absolute floor, not adaptive.

• isTooShortToTranscribe (audio.ts:58) only rejects clips under 250ms or 512 bytes.
  A 30-second mostly-silent recording with steady room noise passes both checks.

3. The VAD seen flag and the idle path

vadStep (vad.ts:90–112) tracks s.seen (whether peak > VOICE_FLOOR AND the current
frame is above speakFrac * peak). If s.seen is never set, vadStep returns "idle"
after idleMs (default 15s), which triggers stopDiscard() — the clip is dropped. This
looks correct.

The gap: s.seen is set to true whenever level > VOICE_FLOOR * speakFrac * peak —
i.e. it uses the PEAK to compute the threshold, and the peak accumulates room noise. In a
noisy room, peak rises to 0.12–0.15 from ambient sound. speakFrac = 0.35 / 1.0 = 0.35,
so speakFrac * peak ≈ 0.042–0.053. Then level > 0.042 fires nearly every frame of
ambient noise — so seen = true is set from room tone, the turn is treated as real speech,
and the idle path never fires. The turn reaches "end" when the post-noise silence arrives,
and the clip uploads.

Control-flow summary for hands-free in a noisy environment (not iOS):

tick() → vadStep() sees ambient noise → s.seen = true → returns null (keep listening)
... 15s later → level drops below threshold → vadStep → "end"
→ endOfTurnAction(gateSnapshot()):
     if gate alive + heardSpeech: "transcribe" ← gate may say YES because it also heard noise
     if gate null/dead:            "transcribe" (iOS / stalled recognizer)
→ sttRef.current.stop() → MediaRecorder onstop:
     _peakLevel > 0 (noise fed it): hadSpeech check skipped (0.1 threshold met by noise)
     clip uploaded → gpt-4o-transcribe → hallucination → isNoiseTranscript:
       "thank you for watching" → SHOULD be caught here ← but was not in the second incident
       "Pottery Barn…"         → NOT in the set → posted as user turn

4. Why isNoiseTranscript did not catch "Thank you for watching"

"thank you for watching" is in SILENCE_HALLUCINATIONS (audio.ts:94). The test at
silence-hallucinations.test.ts:44 verifies it. If it still reached the agent as a message,
one of two things happened:

(a) The planSend / isNoiseTranscript call was on a path that didn't fire (e.g. the
message was posted via a non-voice code path that attached an audioKey separately), or
(b) The transcript returned by gpt-4o-transcribe had slightly different casing/punctuation
that survived normalizeSpeech in a different form — but the test covers
\"Thank you for watching!\" so this is unlikely.

The audioKey on both phantom turns means they went through _transcribeWhisper →
_readTranscriptionStream → onAudio(blob) → onResult(text, true) → handleResult →
planNoiseRejection. The client:voice durable log (MCP list_errors) should have a
\"voice turn rejected as noise\" row for the second incident if the blocklist fired, or no
row if it did not. Verify with MCP list_errors on that instance before coding.

What is NOT happening

This is NOT a VAD hang issue (that is #489 — AudioBufferSourceNode.onerror never resolving).
The phantom turns were submitted; the issue is what was submitted, not that the session hung.
Both issues share the same hands-free auto-loop as the context; fixing this one reduces the
surface that #489 is triggered on.

Scope

In scope:

• Pre-upload speech-onset gate that is robust when the Web Speech gate is absent (iOS) or unreliable (noisy room)
• Post-transcription confidence check using the model's own no-speech signals (temperature=0, checking no_speech_prob or avg-logprob via verbose_json mode when available)
• Bias-prompt gating on low-energy input (suppress the prompt when the clip is likely silence)
• Unit tests feeding silent/near-silent audio buffers asserting no transcript is submitted

Out of scope:

• Widening the SILENCE_HALLUCINATIONS phrase blocklist as a primary fix (see above — explicitly rejected; no Pottery Barn entry, no new phrases added as a primary measure)
• Changes to the Whisper or gpt-4o-transcribe models
• Changes to how existing messages are stored or displayed

Acceptance criteria

• A clip recorded during hands-free on iOS Safari (no Web Speech gate) is not uploaded to Whisper unless the _peakLevel exceeds an adaptive threshold that distinguishes speech onset from steady ambient noise
• The adaptive threshold is relative to the ambient noise floor measured at the START of the recording window, not a fixed 0.1 constant — so a user in a quiet room and a user near an HVAC get the same protection
• A clip where noteLevel() was never called (audio monitor not running) is treated as "unknown energy" — if there is no gate AND no energy data, the clip is discarded rather than uploaded
• For gpt-4o-transcribe: the request is made with temperature=0 (reduces hallucination probability per OpenAI docs); the response is checked for no_speech_prob or equivalent low-confidence signal (requires response_format=verbose_json); a result with no-speech confidence above the threshold emits \"no-speech\" instead of calling onResult
• The transcription bias prompt (transcribePrompt) is suppressed — or its content stripped to an empty string — when the input clip is classified as likely-silent/low-energy, so the model cannot continue it on silence
• stt.test.ts or a new stt-silence.test.ts includes a test that feeds a MediaRecorder onstop with a silent buffer (all _peakLevel = 0.02) and asserts onResult is never called and onEnd is called once
• vad.test.ts includes a test where steady ambient noise (constant level = 0.08, above VOICE_FLOOR 0.1 -- no, wait: 0.08 < 0.1, so use level = 0.12) feeds vadStep and asserts that s.seen is NOT set by ambient-only frames below the adaptive onset threshold
• The two saved recordings (audioKey 7c4ea5a2-… and 65ae0a63-…) are confirmed via MCP list_errors on the instance to determine whether planNoiseRejection fired — this informs whether the gap is pre-upload (energy gate) or post-upload (blocklist path). Document finding in the PR.
• After the fix, a 30-second recording of background noise in hands-free mode does NOT post a user turn. This can be verified by running hands-free in a quiet room, letting it idle, and confirming the client:voice log shows either \"idle recycle\" or a \"voice turn rejected\" row, with no new user message.

Technical notes

Files

File	Relevant lines	What to look at
packages/sdk/src/voice/vad.ts	48, 64, 90–111	VOICE_FLOOR = 0.1 fixed threshold; hadSpeech; vadStep seen flag set from ambient noise
packages/sdk/src/voice/stt.ts	304, 338–344	_peakLevel accumulator; hadSpeech gate; conditional skip when _peakLevel === 0
packages/sdk/src/voice/use-voice.ts	226–228, 310, 427, 438–468	gateRef is null on iOS; noteLevel feed; endOfTurnAction branch
packages/sdk/src/voice/machine.ts	145–148	endOfTurnAction — null gate → "transcribe"
packages/sdk/src/voice/audio.ts	56–60, 86–137	isTooShortToTranscribe (250ms); SILENCE_HALLUCINATIONS set; isNoiseTranscript
packages/sdk/src/voice/gate.ts	125–128	speechGateAvailable — returns false on iOS Safari
packages/sdk/src/voice/prompt.ts	157–167	Prompt is already a bare term list (not prose) to reduce fluent hallucination; still supplies candidates on silence
packages/sdk/src/voice/silence-hallucinations.test.ts	whole file	Existing blocklist test invariants to not break

Specific code path that produced the phantom turns (non-iOS, ambient noise)

startAudioMonitor() →
  vadStep(): ambient level 0.12 > VOICE_FLOOR 0.1 → peak = 0.12 → heardVoice = true
  speakFrac = 0.35 → level 0.12 > 0.12 * 0.35 = 0.042 → speaking = true → s.seen = true
  ... noise continues ... 
  then quiet → level drops → now - lastLoud > silenceMs → "end"
→ endOfTurnAction(gateSnapshot()):
  gate may be alive but also triggered by noise → "transcribe"
→ sttRef.stop() → onstop:
  _peakLevel = 0.12 > 0 → hadSpeech(0.12) = true (0.12 > 0.1) → upload proceeds
→ _transcribeWhisper(blob):
  prompt attached (non-empty transcribePrompt) → model continues it → hallucination
→ _readTranscriptionStream → onResult("Pottery Barn…", true)
→ handleResult → planNoiseRejection → isNoiseTranscript("pottery barn…") = false → finalize → emitSend

Proposed layered fix

(a) Adaptive noise floor in the VAD / energy gate (primary fix, pre-upload):

Measure ambient noise during the first N frames of each turn (before seen is ever set) to
establish a per-turn noise floor. Require speech onset to be at least k × noiseFloor above
that floor (e.g. k=3) before setting s.seen = true. This is the same principle the existing
speakFrac uses relative to peak, but applied to the ONSET decision rather than the
relative-to-peak instantaneous level. Without speech onset at this ratio, the clip is treated
as ambient-only and discarded before upload.

This also fixes the iOS case: a clip with no speech onset (as measured by energy alone) is
discarded in stt.ts onstop via hadSpeech, even without a Web Speech gate.

(b) Model-side confidence check (secondary fix, post-upload):

Request response_format=verbose_json from the transcription endpoint (works for both
whisper-1 and gpt-4o-transcribe). The response includes no_speech_prob (0–1) and
avg_logprob. Treat the transcript as "no-speech" when no_speech_prob > 0.6 (OpenAI's
documented threshold for silence) and emit the soft \"no-speech\" sentinel instead of
calling onResult. Note: gpt-4o-transcribe in streaming mode may not return
verbose_json — check the OpenAI API docs for the streaming equivalent or fall back to
non-streaming when the confidence check is desired. This is a network-round-trip-level fix,
not a substitute for the energy gate.

(c) Bias prompt gating on low-energy input:

If the clip is classified as low-energy (no speech onset) but is uploaded anyway (e.g. the
adaptive floor is disabled or unavailable), do NOT attach the transcribePrompt. An empty
prompt means the model has fewer tokens to "continue" on silence. The prompt is already a
bare comma-separated term list (prompt.ts:157–167) rather than a prose sentence (that fix
was in 365475c), but even a bare list supplies candidates. Suppressing it on low-energy
input removes the last scaffold.

Cross-link

• [bug] Hands-free hangs on "Speaking · tap to stop" with no audio — AudioBufferSourceNode onerror never resolves the playback promise, no speaking watchdog #489 — hands-free "Speaking" hang (same auto-loop context; reducing phantom uploads reduces the surface that hang occurs on)

Verification steps

1. MCP list_errors on instance cda75e28-cace-4958-ac3e-6a7528e6b719 — filter source: \"voice\" — check whether a \"voice turn rejected as noise\" row exists near 2026-08-10T09:16:24Z and 2026-08-07T23:38Z. Absence means planNoiseRejection never ran (the gap is pre-send, likely the gate vouched for ambient noise).
2. The two saved recordings (audioKey 7c4ea5a2-6d73-4ad0-a5d5-fdef5f419da8 and 65ae0a63-…) are retrievable via GET /v1/instances/:id/voice-audio/:turnId — replay them to confirm they contain silence/noise only, not actual speech.

Priority

P1 — phantom user messages are trust-destroying; they cause the agent to act on words the user never said, in hands-free mode where the user cannot easily intervene. The apply pipeline and the Coder loop both auto-act on user messages.