Skip to content

(sarvam stt): fix end of speech timing - #6052

Merged
tinalenguyen merged 3 commits into
mainfrom
tina/fix-sarvam-stt-eos
Jun 11, 2026
Merged

tinalenguyen merged 3 commits into
mainfrom
tina/fix-sarvam-stt-eos

Conversation

@tinalenguyen

Copy link
Copy Markdown
Member

Problem

The Sarvam streaming STT plugin tried to manufacture an audio-relative speech-end time from two sources that don't actually provide one:

  • The VAD END_SPEECH event's occured_at field — which logging proved is a wall-clock Unix epoch, not an audio-stream offset, so it was rejected 100% of the time by a range-check heuristic.
  • A local send-clock counter (_audio_position) that counts audio uploaded, not processed — biased and fabricated.

Sarvam's streaming socket genuinely sends no usable word timing (no timestamps array; speech_start/speech_end come back null), so all this machinery produced misleading timestamps.

Change

Aligned Sarvam with how every other STT plugin (Deepgram, AssemblyAI, Google, Azure…) handles this:

  1. END_OF_SPEECH is now emitted bare — no alternatives/end_time. Previously it carried an empty-text SpeechData with a fabricated end_time + unused speech_end_wall_time metadata.
  2. FINAL_TRANSCRIPT timing comes only from the provider — start_time/end_time read from speech_start/speech_end, falling back to 0.0. The voice pipeline then uses wall-clock for EOU timing (standard behavior for providers without streaming word timing).
  3. Deleted the dead machinery: _interpret_signal_time, _resolved_speech_end, the _audio_position send-clock (+ its hot-loop increment), _utterance_server_speech_end, _utterance_speech_end_audio_pos, _utterance_speech_end_wall, and the require_end_time param.

@chenghao-mou
chenghao-mou requested a review from a team June 11, 2026 05:09

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 3 additional findings.

Open in Devin Review

@chenghao-mou chenghao-mou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, one small nitpick.

text=transcript_text,
start_time=start_time,
end_time=end_time,
start_time=self._positive_time(transcript_data.get("speech_start")) or 0.0,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can add the start_time_offset in _positive_time so that all values through that function includes session start -> audio start delay:

    def _positive_time(self, value: object) -> float | None:
        if isinstance(value, bool) or not isinstance(value, (int, float)):
            return None
        if value <= 0:
            return None
        return float(value) + self.start_time_offset

@tinalenguyen
tinalenguyen merged commit 447cff9 into main Jun 11, 2026
23 checks passed
@tinalenguyen
tinalenguyen deleted the tina/fix-sarvam-stt-eos branch June 11, 2026 16:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants