fix(voice): include tool execution context in replies by rosetta-livekit-bot[bot] · Pull Request #1802 · livekit/agents-js

rosetta-livekit-bot · 2026-06-16T14:27:27Z

Summary

Port fix(voice): include conversation items added during tool execution in reply context agents#6084 to agents-js.
Refresh the reply chat context with conversation items added during tool execution before generating a tool response.
Add a patch changeset for @livekit/agents.

Testing

pnpm --filter @livekit/agents build:types

Original PR description

When a function tool awaits an inline AgentTask, the sub-conversation it runs is merged into the agent's chat_ctx at handoff-return. But the reply generated after tool execution uses a chat_ctx snapshot taken before the tool ran, so it's blind to everything said inside the task — in practice the agent re-asks for fields the task already collected.

This refreshes the snapshot with any chat items added during tool execution before generating the tool reply, mirroring the existing update_instructions() refresh just below it.

Found while benchmarking the hotel receptionist example, where the card-collection AgentTask would complete and the agent would immediately ask for the card number again.

…1517)

…1525) Co-authored-by: rosetta-livekit-bot[bot] <282703043+rosetta-livekit-bot[bot]@users.noreply.github.com> Co-authored-by: u9g <jason.lernerman@livekit.io>

Agent.llmNode now returns ReadableStream<ChatChunk | string | FlushSentinel>, but the agent_v2 hook overrides and AgentHookAdapter still declared the narrower ChatChunk | string union, so passing super.llmNode as the fallback failed to type-check. Widen the override return types and the adapter's fallback/return signatures to include FlushSentinel. Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Brian Yin <brian.yin@livekit.io> Co-authored-by: rosetta-livekit-bot[bot] <282703043+rosetta-livekit-bot[bot]@users.noreply.github.com> Co-authored-by: u9g <jason.lernerman@livekit.io>

Catch end-call close listener errors to avoid unhandled rejections during shutdown, and make public tool type guards return false for null inputs.

Co-authored-by: rosetta-livekit-bot[bot] <282703043+rosetta-livekit-bot[bot]@users.noreply.github.com>

…egment (#1760) Co-authored-by: Cursor <cursoragent@cursor.com>

…#1698) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-authored-by: rosetta-livekit-bot[bot] <282703043+rosetta-livekit-bot[bot]@users.noreply.github.com> Co-authored-by: Toubat <brian.yin@livekit.io>

# Conflicts: # agents/src/voice/agent_activity.test.ts # agents/src/voice/generation.ts # agents/src/voice/generation_tts_timeout.test.ts

The test asserted an exact tick count ([0,1,2]) against real timers with a 5ms margin, which flakes on loaded CI runners. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-authored-by: rosetta-livekit-bot[bot] <282703043+rosetta-livekit-bot[bot]@users.noreply.github.com>

changeset-bot · 2026-06-16T14:27:34Z

🦋 Changeset detected

Latest commit: 5c5a574

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 35 packages

Name	Type
@livekit/agents	Patch
@livekit/agents-plugin-anam	Patch
@livekit/agents-plugin-assemblyai	Patch
@livekit/agents-plugin-baseten	Patch
@livekit/agents-plugin-bey	Patch
@livekit/agents-plugin-cartesia	Patch
@livekit/agents-plugin-cerebras	Patch
@livekit/agents-plugin-deepgram	Patch
@livekit/agents-plugin-did	Patch
@livekit/agents-plugin-elevenlabs	Patch
@livekit/agents-plugin-fishaudio	Patch
@livekit/agents-plugin-google	Patch
@livekit/agents-plugin-hedra	Patch
@livekit/agents-plugin-hume	Patch
@livekit/agents-plugin-inworld	Patch
@livekit/agents-plugin-lemonslice	Patch
@livekit/agents-plugin-liveavatar	Patch
@livekit/agents-plugin-livekit	Patch
@livekit/agents-plugin-minimax	Patch
@livekit/agents-plugin-mistral	Patch
@livekit/agents-plugin-mistralai	Patch
@livekit/agents-plugin-neuphonic	Patch
@livekit/agents-plugin-openai	Patch
@livekit/agents-plugin-perplexity	Patch
@livekit/agents-plugin-phonic	Patch
@livekit/agents-plugin-resemble	Patch
@livekit/agents-plugin-rime	Patch
@livekit/agents-plugin-runway	Patch
@livekit/agents-plugin-sarvam	Patch
@livekit/agents-plugin-silero	Patch
@livekit/agents-plugin-soniox	Patch
@livekit/agents-plugin-tavus	Patch
@livekit/agents-plugins-test	Patch
@livekit/agents-plugin-trugen	Patch
@livekit/agents-plugin-xai	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

devin-ai-integration

Devin Review found 1 potential issue.

devin-ai-integration · 2026-06-16T14:32:17Z

      chatCtx.insert(toolMessages);
+      // Refresh conversation items added during tool execution, such as an inline
+      // AgentTask sub-conversation merged into the agent chat context at handoff return.
+      chatCtx.merge(this.agent._chatCtx, { excludeInstructions: true });


🚩 Realtime path does not have equivalent merge for tool execution context

The pipeline path now merges this.agent._chatCtx into the local chatCtx at agents/src/voice/agent_activity.ts:3020 to capture items added during tool execution (e.g., sub-agent handoff conversations). The realtime path at agents/src/voice/agent_activity.ts:3572 copies from realtimeSession.chatCtx and pushes tool outputs, but does NOT perform a similar merge from this.agent._chatCtx. This may be intentional since realtime models manage their own chat context via realtimeSession.updateChatCtx(), but if the same handoff-return scenario can occur in the realtime path, those sub-conversation items would be missing.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration

Devin Review found 2 new potential issues.

devin-ai-integration · 2026-07-02T13:54:21Z

+    const serverVad = this.#opts.serverVad;
+    const commitStrategy = serverVad === undefined || serverVad === null ? 'manual' : 'vad';


🚩 ElevenLabs STT default commit_strategy changed from 'vad' to 'manual'

The old code at plugins/elevenlabs/src/stt.ts:653 used this.#opts.serverVad === null ? 'manual' : 'vad', meaning undefined (the default when no serverVad option is passed) resulted in 'vad' commit strategy. The new code uses serverVad === undefined || serverVad === null ? 'manual' : 'vad', so the default is now 'manual'. This is a behavioral change for users who don't explicitly set serverVad: their ElevenLabs STT streams will now use manual commit strategy instead of server-side VAD. The test updates at stt.test.ts:283,366,396-407 confirm this is intentional. The changeset description mentions this avoids persisting per-call overrides, but the commit_strategy default flip is the more impactful change for existing users.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-07-02T13:54:23Z

+  if (tools.length > 0 && providerTools.length > 0) {
+    throw new Error('Gemini does not support mixing function tools and provider tools');
+  }
+
+  if (onlySingleType && tools.length > 0) {
+    return tools;
+  }
+
+  tools.push(...providerTools);
+
+  return tools.length > 0 ? tools : undefined;
+}


🚩 Google LLM now enforces single-type tool constraint via toToolsConfig

The new toToolsConfig function at plugins/google/src/utils.ts:163-215 throws an error when both function tools and provider tools (GeminiTool subclasses or legacy geminiTools) are present: 'Gemini does not support mixing function tools and provider tools'. This is a new runtime validation that didn't exist before. The onlySingleType parameter (used by the non-realtime LLMStream.run() at plugins/google/src/llm.ts:363) returns early with just function tools when both types are present, effectively silently dropping provider tools. This inconsistency between the two code paths (throw vs. silently drop) could be confusing. The realtime path at plugins/google/src/realtime/realtime_api.ts:1445 does not pass onlySingleType, so it would throw if both types are present.

Was this helpful? React with 👍 or 👎 to provide feedback.

toubatbrian and others added 20 commits May 28, 2026 14:43

Refactor ToolContext to parity class taking a list of Tool | Toolset (#…

244d8bc

…1517)

feat(agents): add Toolset support to ToolContext and AgentActivity (#…

6b85bfe

…1525) Co-authored-by: rosetta-livekit-bot[bot] <282703043+rosetta-livekit-bot[bot]@users.noreply.github.com> Co-authored-by: u9g <jason.lernerman@livekit.io>

Merge branch 'main' into 1.5.0

57c1634

Merge branch 'main' into 1.5.0

07648b0

feat(agents): add beta end call tool (#1474)

c8b56bd

Co-authored-by: Brian Yin <brian.yin@livekit.io> Co-authored-by: rosetta-livekit-bot[bot] <282703043+rosetta-livekit-bot[bot]@users.noreply.github.com> Co-authored-by: u9g <jason.lernerman@livekit.io>

arden end-call shutdown and tool guards

f089da4

Catch end-call close listener errors to avoid unhandled rejections during shutdown, and make public tool type guards return false for null inputs.

fix(elevenlabs): end server vad turns (#1745)

0daf4ef

Co-authored-by: rosetta-livekit-bot[bot] <282703043+rosetta-livekit-bot[bot]@users.noreply.github.com>

Add Inworld delivery mode inference TTS option (#1749)

d423f3f

Co-authored-by: rosetta-livekit-bot[bot] <282703043+rosetta-livekit-bot[bot]@users.noreply.github.com>

Don't retain recorded events when recording is disabled (#1750)

9d86bc5

ci: speed up type checking (#1742)

03429b8

Co-authored-by: rosetta-livekit-bot[bot] <282703043+rosetta-livekit-bot[bot]@users.noreply.github.com>

fix(voice): scope forwardAudio playback-started listener to its own s…

ec765ea

…egment (#1760) Co-authored-by: Cursor <cursoragent@cursor.com>

feat(barge-in): add default threshold support and drop http transport (…

416871a

…#1698) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(voice): add agent instruction updates (#1783)

eae6074

Co-authored-by: rosetta-livekit-bot[bot] <282703043+rosetta-livekit-bot[bot]@users.noreply.github.com> Co-authored-by: Toubat <brian.yin@livekit.io>

Merge remote-tracking branch 'origin/main' into 1.5.0

882491c

# Conflicts: # agents/src/voice/agent_activity.test.ts # agents/src/voice/generation.ts # agents/src/voice/generation_tts_timeout.test.ts

test: use fake timers in manual-abort Task test to deflake CI

b579216

The test asserted an exact tick count ([0,1,2]) against real timers with a 5ms margin, which flakes on loaded CI runners. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

fix(elevenlabs): stop recognize from mutating instance language (#1789)

2cd85da

Co-authored-by: rosetta-livekit-bot[bot] <282703043+rosetta-livekit-bot[bot]@users.noreply.github.com>

fix(phonic): add client header to conversations (#1781)

d9f56db

Co-authored-by: rosetta-livekit-bot[bot] <282703043+rosetta-livekit-bot[bot]@users.noreply.github.com>

fix(inference): remove stale Cartesia STT model type (#1794)

20fcec7

Co-authored-by: rosetta-livekit-bot[bot] <282703043+rosetta-livekit-bot[bot]@users.noreply.github.com>

fix(voice): include tool execution context in replies

5c5a574

rosetta-livekit-bot Bot requested a review from ShayneP June 16, 2026 14:27

devin-ai-integration Bot reviewed Jun 16, 2026

View reviewed changes

Base automatically changed from 1.5.0 to main July 2, 2026 13:48

An error occurred while trying to automatically change base from 1.5.0 to main July 2, 2026 13:48

devin-ai-integration Bot reviewed Jul 2, 2026

View reviewed changes

Navigation Menu

Repository navigation

fix(voice): include tool execution context in replies - #1802

fix(voice): include tool execution context in replies#1802
rosetta-livekit-bot[bot] wants to merge 20 commits into
mainfrom
fix-voice-tool-reply-context

rosetta-livekit-bot Bot commented Jun 16, 2026 •

edited

Loading

Uh oh!

changeset-bot Bot commented Jun 16, 2026

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

devin-ai-integration Bot Jun 16, 2026 •

edited

Loading

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

devin-ai-integration Bot Jul 2, 2026

Uh oh!

devin-ai-integration Bot Jul 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		const serverVad = this.#opts.serverVad;
		const commitStrategy = serverVad === undefined \|\| serverVad === null ? 'manual' : 'vad';

Conversation

rosetta-livekit-bot Bot commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Uh oh!

changeset-bot Bot commented Jun 16, 2026

🦋 Changeset detected

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

rosetta-livekit-bot Bot commented Jun 16, 2026 •

edited

Loading

devin-ai-integration Bot Jun 16, 2026 •

edited

Loading