Skip to content

fix(xai): emit one final user transcript per turn - #2238

Merged
longcw merged 5 commits into
mainfrom
reinsert-bastes-whom
Aug 12, 2026
Merged

longcw merged 5 commits into
mainfrom
reinsert-bastes-whom

Conversation

@rosetta-livekit-bot

@rosetta-livekit-bot rosetta-livekit-bot Bot commented Aug 7, 2026 •

Copy link
Copy Markdown
Contributor

Ports livekit/agents#6729 and folds in agents-js#1912.

xAI retranscribes its persistent input item from the beginning at every audio commit. Hold the latest committed transcript until a response produces output, then emit one final transcript for the turn. This also closes silently abandoned response streams, preserves held turns during chat-context reconciliation, handles unknown xAI item anchors, and settles rejected OpenAI chat-context events without waiting for the timeout.

Source diff coverage
  • Adapted: livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/realtime/realtime_model.py -> plugins/openai/src/realtime/realtime_model.ts. Ported event-ID-correlated chat-context waiters, nonfatal rejection handling, settled-waiter guards, reconnect reset ordering, generation-close/subclass hooks, and audio debug suppression using TypeScript primitives. The rebased target now provides native toUpdate reconciliation, which supplies the modified-item behavior required by held turns.
  • Adapted: livekit-plugins/livekit-plugins-xai/livekit/plugins/xai/realtime/realtime_model.py -> plugins/xai/src/realtime/realtime_model.ts. Added the xAI realtime session subclass with deferred final transcripts, silent-response disposal, held-turn reconciliation, unknown-anchor fallback, reconnect/close flushing, and audio/text response boundaries.
  • Adapted: livekit-plugins/livekit-plugins-xai/livekit/plugins/xai/types.py -> plugins/xai/src/realtime/realtime_model.ts, where agents-js defines GrokRealtimeModels. Added grok-voice-think-fast-2.0.
  • Ported: tests/test_realtime/test_openai_realtime_model.py -> plugins/openai/src/realtime/realtime_model.test.ts. Ported all five chat-context rejection/waiter scenarios.
  • Ported: tests/test_realtime/test_xai_realtime_model.py -> plugins/xai/src/realtime/realtime_model.test.ts. Ported the source turn-state, mirror integrity, reconciliation, anchor, and abandoned-response scenarios; the Python-only speech timestamp assertion is represented by the target contract that repeated same-item speech starts keep the held turn pending because agents-js does not track per-item speech onset timestamps.
  • Not applicable: none.

Validation

  • pnpm exec turbo run build --filter=@livekit/agents-plugin-openai... --filter=@livekit/agents-plugin-xai...
  • env -u OPENAI_API_KEY pnpm test -- plugins/openai (106 passed, 7 credential-gated skipped)
  • env -u XAI_API_KEY pnpm test -- plugins/xai (25 passed, 1 credential-gated skipped)
  • ESLint on all changed source and test files
  • Prettier check on all changed source and test files

Package-wide lint was also run, but current main has unrelated Prettier failures in untouched OpenAI and xAI files. cue-cli runtime validation remains unavailable because the environment has LiveKit credentials but no XAI_API_KEY.

@changeset-bot

changeset-bot Bot commented Aug 7, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 0f0dfee

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 39 packages
Name Type
@livekit/agents-plugin-openai Major
@livekit/agents-plugin-xai Major
@livekit/agents-plugin-anam Major
@livekit/agents-plugin-cartesia Major
@livekit/agents-plugin-cerebras Major
@livekit/agents-plugin-elevenlabs Major
@livekit/agents-plugin-fishaudio Major
@livekit/agents-plugin-google Major
@livekit/agents-plugin-hume Major
@livekit/agents-plugin-inworld Major
@livekit/agents-plugin-krisp Major
@livekit/agents-plugin-neuphonic Major
@livekit/agents-plugin-perplexity Major
@livekit/agents-plugin-rime Major
@livekit/agents-plugin-sarvam Major
@livekit/agents Major
@livekit/agents-plugin-anthropic Major
@livekit/agents-plugin-assemblyai Major
@livekit/agents-plugin-azure Major
@livekit/agents-plugin-baseten Major
@livekit/agents-plugin-bey Major
@livekit/agents-plugin-deepgram Major
@livekit/agents-plugin-did Major
@livekit/agents-plugin-hedra Major
@livekit/agents-plugin-lemonslice Major
@livekit/agents-plugin-liveavatar Major
@livekit/agents-plugin-livekit Major
@livekit/agents-plugin-minimax Major
@livekit/agents-plugin-mistral Major
@livekit/agents-plugin-mistralai Major
@livekit/agents-plugin-phonic Major
@livekit/agents-plugin-protoface Major
@livekit/agents-plugin-resemble Major
@livekit/agents-plugin-runway Major
@livekit/agents-plugin-silero Major
@livekit/agents-plugin-soniox Major
@livekit/agents-plugin-tavus Major
@livekit/agents-plugin-trugen Major
@livekit/agents-plugins-test Major

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@rosetta-livekit-bot
rosetta-livekit-bot Bot requested a review from longcw August 7, 2026 04:04

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 3 potential issues.

View 3 additional findings in Devin Review.

Open in Devin Review

Comment thread plugins/openai/src/realtime/realtime_model.ts
Comment thread plugins/openai/src/realtime/realtime_model.ts Outdated
Comment on lines +64 to +66
export class RealtimeSession extends OpenAIRealtimeSession {
private pendingTranscription?: realtime.ConversationItemInputAudioTranscriptionCompletedEvent;
private responseSpoke = false;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Newly added public class and its overrides ship without API documentation

The new publicly exported xAI realtime session class and its methods are added without any TSDoc comments, which the repository's contribution rules require for every new class/method.
Impact: Generated API documentation for the xAI plugin will have an undocumented public class.

Rule reference

CONTRIBUTING.md: "If writing new methods/interfaces/enums/classes, document them. This project uses TypeDoc for automatic API documentation generation, and every new addition has to be properly documented." The new RealtimeSession class (plugins/xai/src/realtime/realtime_model.ts:64) and its public members (close, interrupt, session() on RealtimeModel at plugins/xai/src/realtime/realtime_model.ts:59) carry no doc comments.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

@rosetta-livekit-bot
rosetta-livekit-bot Bot changed the base branch from nonesuch-bowman-befogged to main August 12, 2026 09:19
@rosetta-livekit-bot
rosetta-livekit-bot Bot requested a review from a team as a code owner August 12, 2026 09:19
Comment thread plugins/openai/src/realtime/realtime_model.ts Outdated
@longcw
longcw merged commit 3ff013c into main Aug 12, 2026
5 checks passed
@longcw
longcw deleted the reinsert-bastes-whom branch August 12, 2026 10:06
@github-actions github-actions Bot mentioned this pull request Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant