diff --git a/docs/issues/truncated-tool-call-recovery/spec.md b/docs/issues/truncated-tool-call-recovery/spec.md new file mode 100644 index 0000000000..9039e2d926 --- /dev/null +++ b/docs/issues/truncated-tool-call-recovery/spec.md @@ -0,0 +1,107 @@ +# Truncated Tool Call Recovery + +## Issue + +When a provider reaches the output token limit after emitting one or more tool calls, DeepChat +preserves the `max_tokens` stop reason but treats the round as a normal terminal completion. The +calls are not executed, which is safe, but they are also not settled with matching tool results. +The current finalizer can consequently mark unresolved tool-call blocks as successful, and neither +the active run nor a rebuilt history reliably tells the model why the calls were skipped. + +## Impact + +- The model cannot re-issue a complete call within the same run. +- Persisted history can omit the truncated call or retain misleading successful UI state. +- A provider can reject a later request when an assistant tool call has no corresponding result. +- Executing a call that merely looks complete would risk running silently truncated arguments. + +## Root Cause + +- The provider adapter correctly maps an AI SDK `length` finish reason to `max_tokens`. +- The DeepChat loop creates a tool batch only for `stopReason === 'tool_use'` with completed calls. +- Tool dispatch models only execution, although assistant-message creation, output fitting, block + updates, notifications, execution-state snapshots, and Tape persistence are batch settlement + responsibilities shared by executed and rejected calls. +- Stream events do not identify whether DeepChat or the provider owns tool execution. ACP and AI + SDK provider-executed calls therefore cannot be excluded by a provider-neutral rule. + +## Fix Design + +1. Add optional per-call execution ownership to core tool-call start events. Missing ownership + remains DeepChat-owned for compatibility; official AI SDK and ACP adapters mark provider-owned + calls explicitly. +2. Refactor tool dispatch into `settleToolBatch`, with an explicit `execute` or + `reject/output_truncated` disposition and one shared result-commit path. +3. On `max_tokens`, collect every DeepChat-owned call from the current provider round in source + order, including a started call whose argument stream did not end. Reject the entire batch and + generate one matching error tool result per call. Never invoke permission checks, reviewers, or + tool executors for this disposition. +4. Count rejected calls as zero requested and zero executed tool calls. Persist their call/result + facts and execution state, but do not increment `metadata.toolCalls` or feed the synthetic batch + to the no-progress guard. +5. Permit one automatic recovery provider round per run. If a second round is truncated while + producing DeepChat-owned calls, settle it and finish with `max_tokens` instead of requesting a + third round. Do not modify the user's `maxTokens` setting. +6. Mark other unresolved tool blocks from a `max_tokens` round as incomplete without fabricating a + client tool result. Provider-owned calls remain display-only and are never locally settled. + +## Compatibility And Failure Semantics + +- Existing stream producers that omit execution ownership keep the current DeepChat-owned behavior. +- Plain-text `max_tokens`, normal `tool_use`, provider-owned tool lifecycles, and stored database + formats remain compatible; no migration or user setting is introduced. +- Explicit provider-round limits, aborts, pending user input, and terminal tool-result fitting + errors retain their existing priority. The current truncated batch is settled before a provider + round limit prevents recovery. +- The rejection result uses the existing tool-result fitting path, so context budget failure remains + a terminal error rather than an unbounded or malformed retry. +- Full provider-executed tool-result round-tripping is outside this issue; execution ownership only + prevents DeepChat from taking over those calls and preserves results already emitted by providers. +- No GitHub issue is created or synchronized for this local change. + +## Acceptance Criteria + +1. No tool, permission precheck, or auto-approve review runs for a `max_tokens` tool batch. +2. Every rejected DeepChat-owned call receives an ordered error result and an error UI block marked + with `toolCallSkippedReason: 'max_tokens'`. +3. Incomplete non-rejected tool blocks are errors marked with + `toolCallIncompleteReason: 'max_tokens'` and do not create fake tool messages. +4. A first truncated tool batch can recover in one additional provider round; a second truncated + batch is settled and terminates without a third request. +5. Rejected calls do not consume the 128-call execution budget or increment persisted tool-call + accounting. +6. Rejected call/result pairs survive context rebuilding and Tape fact persistence. +7. Native, legacy, multiple, pending, provider-owned, abort, pending-input, and explicit-round-limit + cases have deterministic regression coverage. + +## Tasks + +- [x] Refactor the loop and dispatch APIs around tool batch settlement. +- [x] Add per-call execution ownership to adapters and accumulation. +- [x] Implement atomic truncated-batch rejection and one-round recovery. +- [x] Add unit, context-rebuild, and deterministic eval coverage. +- [x] Run formatting, i18n, lint, type checking, targeted tests, and agent evals. +- [x] Review every staged commit for correctness, compatibility, side effects, and maintenance risk. + +## Validation + +```bash +pnpm exec vitest run --config vitest.config.ts \ + test/main/agent/deepchat/loop/deepChatLoopEngine.test.ts \ + test/main/agent/deepchat/runtime/accumulator.test.ts \ + test/main/agent/deepchat/runtime/dispatch.test.ts \ + test/main/agent/deepchat/runtime/process.test.ts \ + test/main/agent/deepchat/runtime/contextBuilder.test.ts \ + test/main/agent/acp/runtime/acpContentMapper.test.ts \ + test/main/provider/aiSdkStreamAdapter.test.ts \ + test/main/evals/nativeAgent/nativeAgentBehavior.eval.test.ts +pnpm run format +pnpm run i18n +pnpm run lint +pnpm run typecheck +pnpm run test:agent:eval +``` + +Validated on 2026-07-24: all eight targeted test files passed (273 tests), formatting and i18n +checks completed cleanly, lint and full type checking passed, and all 15 deterministic native Agent +eval scenarios passed. diff --git a/src/main/agent/acp/runtime/acpContentMapper.ts b/src/main/agent/acp/runtime/acpContentMapper.ts index a11a3cf8db..8e8bf5cc56 100644 --- a/src/main/agent/acp/runtime/acpContentMapper.ts +++ b/src/main/agent/acp/runtime/acpContentMapper.ts @@ -463,7 +463,9 @@ export class AcpContentMapper { private emitToolCallStartIfNeeded(state: ToolCallState, payload: MappedContent) { if (state.started) return state.started = true - payload.events.push(createStreamEvent.toolCallStart(state.toolCallId, state.toolName)) + payload.events.push( + createStreamEvent.toolCallStart(state.toolCallId, state.toolName, undefined, 'provider') + ) } private emitToolCallChunk(state: ToolCallState, chunk: string, payload: MappedContent) { diff --git a/src/main/agent/deepchat/loop/deepChatLoopEngine.ts b/src/main/agent/deepchat/loop/deepChatLoopEngine.ts index 1a3557949d..ada3da4f22 100644 --- a/src/main/agent/deepchat/loop/deepChatLoopEngine.ts +++ b/src/main/agent/deepchat/loop/deepChatLoopEngine.ts @@ -4,11 +4,12 @@ export const MAX_TOOL_CALLS = 128 export type ProviderRoundOutcome = | { type: 'terminal' } - | { type: 'tool_batch'; batch: TToolBatch; toolCallCount: number } + | { type: 'tool_batch'; batch: TToolBatch; requestedToolExecutionCount: number } | { type: 'halted'; result: THalted } export type LoopToolBatchOutcome = | { type: 'continue'; executedToolCount: number } + | { type: 'terminal'; executedToolCount: number } | { type: 'halted'; result: THalted } export type DeepChatLoopOutcome = @@ -25,7 +26,7 @@ export interface DeepChatLoopDependencies { run: LoopRun providerRound: number }): Promise> - executeToolBatch(input: { + settleToolBatch(input: { run: LoopRun providerRound: number batch: TToolBatch @@ -100,7 +101,7 @@ export class DeepChatLoopEngine { return providerOutcome } - const attemptedToolCount = executedToolCount + providerOutcome.toolCallCount + const attemptedToolCount = executedToolCount + providerOutcome.requestedToolExecutionCount if (attemptedToolCount > MAX_TOOL_CALLS) { return { type: 'max_tool_calls', @@ -109,7 +110,7 @@ export class DeepChatLoopEngine { } } - const toolOutcome = await dependencies.executeToolBatch({ + const toolOutcome = await dependencies.settleToolBatch({ run, providerRound, batch: providerOutcome.batch @@ -123,6 +124,9 @@ export class DeepChatLoopEngine { return toolOutcome } executedToolCount += toolOutcome.executedToolCount + if (toolOutcome.type === 'terminal') { + return { type: 'terminal' } + } } } } diff --git a/src/main/agent/deepchat/runtime/accumulator.ts b/src/main/agent/deepchat/runtime/accumulator.ts index b4772eae2b..bd9309fc9b 100644 --- a/src/main/agent/deepchat/runtime/accumulator.ts +++ b/src/main/agent/deepchat/runtime/accumulator.ts @@ -164,6 +164,7 @@ export function accumulate(state: StreamState, event: LLMCoreStreamEvent): void name: event.tool_call_name, arguments: '', blockIndex: state.blocks.length - 1, + executionOwner: event.tool_call_execution_owner ?? 'deepchat', providerOptions: event.provider_options }) state.dirty = true @@ -213,12 +214,14 @@ export function accumulate(state: StreamState, event: LLMCoreStreamEvent): void : {}) } } - state.completedToolCalls.push({ - id: event.tool_call_id, - name: pending.name, - arguments: finalArgs, - ...(providerOptions ? { providerOptions } : {}) - }) + if (pending.executionOwner === 'deepchat') { + state.completedToolCalls.push({ + id: event.tool_call_id, + name: pending.name, + arguments: finalArgs, + ...(providerOptions ? { providerOptions } : {}) + }) + } state.pendingToolCalls.delete(event.tool_call_id) state.dirty = true } diff --git a/src/main/agent/deepchat/runtime/dispatch.ts b/src/main/agent/deepchat/runtime/dispatch.ts index b5994b5c79..3e4c5ab759 100644 --- a/src/main/agent/deepchat/runtime/dispatch.ts +++ b/src/main/agent/deepchat/runtime/dispatch.ts @@ -22,6 +22,7 @@ import type { ProcessControlCollaborators, StreamState, ToolBatchInteraction, + ToolCallResult, ToolDispatchCollaborators } from './types' import type { ChatMessage, ChatMessageProviderOptions } from '@shared/types/core/chat-message' @@ -85,6 +86,7 @@ type StagedToolResult = { imagePreviews?: ToolCallImagePreview[] skillDraftPrompt?: SkillDraftPromptPayload postHookKind: 'success' | 'failure' + skippedReason?: 'max_tokens' } type SkillDraftPromptPayload = { @@ -142,15 +144,111 @@ type MutableToolBatchState = { const PARALLEL_READ_ONLY_AGENT_TOOLS = new Set(['read']) const USER_CANCELED_GENERATION_ERROR = 'common.error.userCanceledGeneration' +export const TRUNCATED_TOOL_CALL_ERROR = + 'Tool call was not executed because the model response reached the output token limit, so its arguments may be incomplete. Retry the tool call with complete arguments.' -function createToolBatchState(state: StreamState): MutableToolBatchState { +export type ToolBatchDisposition = + | { kind: 'execute' } + | { kind: 'reject'; reason: 'output_truncated' } + +function createToolBatchState(toolCalls: readonly ToolCallResult[]): MutableToolBatchState { return { - callOrder: state.completedToolCalls.map((toolCall) => toolCall.id), + callOrder: toolCalls.map((toolCall) => toolCall.id), invokedCallIds: new Set(), committedResultCallIds: new Set() } } +interface CommitStagedToolResultsParams { + stagedResults: StagedToolResult[] + pendingInteractions: ToolBatchInteraction[] + batchState: MutableToolBatchState + executed: number + toolsChanged: boolean + conversation: ChatMessage[] + state: StreamState + batchToolCallBlocks: AssistantMessageBlock[] + toolBlockStartIndex: number + io: IoParams + notificationObserver?: DeepChatLoopNotificationObserver + takeInteractionOrder: () => number + toolResults: ToolResultPort + tools: MCPToolDefinition[] + contextLength: number + maxTokens: number + rendererFlushHandle: RendererFlushHandle +} + +async function commitStagedToolResults( + params: CommitStagedToolResultsParams +): Promise> { + const { + stagedResults, + pendingInteractions, + batchState, + executed, + toolsChanged, + conversation, + state, + batchToolCallBlocks, + toolBlockStartIndex, + io, + notificationObserver, + takeInteractionOrder, + toolResults, + tools, + contextLength, + maxTokens, + rendererFlushHandle + } = params + + if (stagedResults.length > 0) { + const fittedResults = await toolResults.fitBatch({ + conversationMessages: conversation, + results: stagedResults.map((result) => ({ + toolCallId: result.toolCallId, + toolName: result.toolName, + responseText: result.responseText, + isError: result.isError, + offloadPath: result.offloadPath + })), + toolDefinitions: tools, + contextLength, + maxTokens + }) + const finalizedInteractions = applyFinalizedToolResults({ + stagedResults, + fittedResults: fittedResults.results, + conversation, + state, + batchToolCallBlocks, + toolBlockStartIndex, + io, + notificationObserver, + appendToConversation: fittedResults.kind === 'ok', + takeInteractionOrder + }) + pendingInteractions.push(...finalizedInteractions) + for (const result of stagedResults) { + batchState.committedResultCallIds.add(result.toolCallId) + } + persistToolExecutionState(io, state, rendererFlushHandle) + + if (fittedResults.kind === 'terminal_error') { + return buildToolBatchOutcome( + batchState, + pendingInteractions, + executed, + toolsChanged, + fittedResults.message + ) + } + } + + persistToolExecutionState(io, state, rendererFlushHandle) + return buildToolBatchOutcome(batchState, pendingInteractions, executed, toolsChanged) +} + function snapshotToolBatchState( state: MutableToolBatchState, interactions: readonly ToolBatchInteraction[] @@ -416,6 +514,21 @@ function updateToolCallBlock( } } +function markToolCallSkipped( + blocks: AssistantMessageBlock[], + toolCallId: string, + reason: NonNullable +): void { + const block = blocks.find((candidate) => candidate.tool_call?.id === toolCallId) + if (!block) return + + block.extra = { + ...block.extra, + toolCallSkippedReason: reason + } + delete block.extra.toolCallIncompleteReason +} + function setToolCallAutoApproveReviewing( blocks: AssistantMessageBlock[], toolCallId: string, @@ -710,7 +823,7 @@ function persistToolExecutionState( scheduleRendererFlush(state, rendererFlushHandle) } -function finalizePendingNarrativeBeforeToolExecution(state: StreamState): void { +function finalizePendingNarrativeBeforeToolSettlement(state: StreamState): void { const last = state.blocks[state.blocks.length - 1] if ( !last || @@ -729,6 +842,8 @@ function applyFinalizedToolResults(params: { fittedResults: ToolBatchOutputFitItem[] conversation: ChatMessage[] state: StreamState + batchToolCallBlocks: AssistantMessageBlock[] + toolBlockStartIndex: number io: IoParams notificationObserver?: DeepChatLoopNotificationObserver appendToConversation: boolean @@ -739,6 +854,8 @@ function applyFinalizedToolResults(params: { fittedResults, conversation, state, + batchToolCallBlocks, + toolBlockStartIndex, io, notificationObserver, appendToConversation, @@ -784,7 +901,7 @@ function applyFinalizedToolResults(params: { imagePreviews: stagedResult.imagePreviews }) updateToolCallBlock( - state.blocks, + batchToolCallBlocks, fittedResult.toolCallId, fittedResult.responseText, fittedResult.isError, @@ -797,10 +914,14 @@ function applyFinalizedToolResults(params: { }, imagePresentation.toolBlockImagePreviews ) + if (stagedResult.skippedReason) { + markToolCallSkipped(batchToolCallBlocks, stagedResult.toolCallId, stagedResult.skippedReason) + } insertBlocksAfterToolCall( state.blocks, fittedResult.toolCallId, - imagePresentation.promotedBlocks + imagePresentation.promotedBlocks, + toolBlockStartIndex ) if (stagedResult.skillDraftPrompt && !fittedResult.isError && !fittedResult.downgraded) { @@ -1030,18 +1151,28 @@ async function reviewAutoApproveAction(params: { controls: ProcessControlCollaborators | undefined io: IoParams state: StreamState + batchToolCallBlocks: AssistantMessageBlock[] rendererFlushHandle: RendererFlushHandle execution: ToolExecutionContext permission: NonNullable reason: 'tool_call' | 'precheck' | 'requires_permission' }): Promise<'auto_allow' | 'ask_user'> { - const { controls, io, state, rendererFlushHandle, execution, permission, reason } = params + const { + controls, + io, + state, + batchToolCallBlocks, + rendererFlushHandle, + execution, + permission, + reason + } = params const reviewToolPermission = controls?.reviewToolPermission if (!reviewToolPermission) { return 'ask_user' } - if (setToolCallAutoApproveReviewing(state.blocks, execution.completedToolCall.id, true)) { + if (setToolCallAutoApproveReviewing(batchToolCallBlocks, execution.completedToolCall.id, true)) { state.dirty = true rendererFlushHandle.flush() } @@ -1071,7 +1202,9 @@ async function reviewAutoApproveAction(params: { } return 'ask_user' } finally { - if (setToolCallAutoApproveReviewing(state.blocks, execution.completedToolCall.id, false)) { + if ( + setToolCallAutoApproveReviewing(batchToolCallBlocks, execution.completedToolCall.id, false) + ) { state.dirty = true rendererFlushHandle.flush() } @@ -1276,6 +1409,7 @@ async function runToolCall(params: { controls?: ProcessControlCollaborators io: IoParams state: StreamState + batchToolCallBlocks: AssistantMessageBlock[] rendererFlushHandle: RendererFlushHandle allowProgressUpdates: boolean onToolCallStarted?: (toolCallId: string) => void @@ -1289,6 +1423,7 @@ async function runToolCall(params: { controls, io, state, + batchToolCallBlocks, rendererFlushHandle, allowProgressUpdates, onToolCallStarted @@ -1304,7 +1439,7 @@ async function runToolCall(params: { update.toolCallId === completedToolCall.id && allowProgressUpdates ) { - markInternalPlanToolCallBlock(state.blocks, completedToolCall.id) + markInternalPlanToolCallBlock(batchToolCallBlocks, completedToolCall.id) const snapshot: AgentPlanSnapshot = { ...update.snapshot, sessionId: io.sessionId, @@ -1327,7 +1462,7 @@ async function runToolCall(params: { } updateSubagentToolCallBlock( - state.blocks, + batchToolCallBlocks, completedToolCall.id, update.responseMarkdown, update.progressJson @@ -1382,6 +1517,7 @@ async function runToolCall(params: { controls, io, state, + batchToolCallBlocks, rendererFlushHandle, execution, permission: pendingPermission, @@ -1440,7 +1576,7 @@ async function runToolCall(params: { const subagentState = extractSubagentToolState(toolRawData) if (allowProgressUpdates && (subagentState.subagentProgress || subagentState.subagentFinal)) { updateSubagentToolCallBlock( - state.blocks, + batchToolCallBlocks, completedToolCall.id, typeof toolRawData.content === 'string' ? toolRawData.content @@ -1528,36 +1664,65 @@ async function runToolCall(params: { } } -export async function executeTools( - state: StreamState, - conversation: ChatMessage[], - prevBlockCount: number, - tools: MCPToolDefinition[], - toolExecution: ToolExecutionPort, - modelId: string, - interleavedReasoning: InterleavedReasoningConfig, - io: IoParams, - permissionMode: PermissionMode, - toolResults: ToolResultPort, - contextLength: number, - maxTokens: number, - rendererFlushHandle: RendererFlushHandle, - collaborators?: ToolDispatchCollaborators, +export interface SettleToolBatchParams { + state: StreamState + conversation: ChatMessage[] + prevBlockCount: number + toolCalls: ToolCallResult[] + disposition: ToolBatchDisposition + tools: MCPToolDefinition[] + toolExecution: ToolExecutionPort + modelId: string + interleavedReasoning: InterleavedReasoningConfig + io: IoParams + permissionMode: PermissionMode + toolResults: ToolResultPort + contextLength: number + maxTokens: number + rendererFlushHandle: RendererFlushHandle + collaborators?: ToolDispatchCollaborators providerId?: string +} + +export async function settleToolBatch( + params: SettleToolBatchParams ): Promise> { + const { + state, + conversation, + prevBlockCount, + toolCalls, + disposition, + tools, + toolExecution, + modelId, + interleavedReasoning, + io, + permissionMode, + toolResults, + contextLength, + maxTokens, + rendererFlushHandle, + collaborators, + providerId + } = params const { notificationObserver, controls, diagnostics, onToolCallStarted } = collaborators ?? {} - io.abortSignal.throwIfAborted() - finalizePendingNarrativeBeforeToolExecution(state) + if (disposition.kind === 'execute') { + io.abortSignal.throwIfAborted() + } + finalizePendingNarrativeBeforeToolSettlement(state) persistToolExecutionState(io, state, rendererFlushHandle) - const toolPermissionMode = getToolCapabilityPermissionMode(permissionMode) - const batchState = createToolBatchState(state) + const batchToolCallBlocks = state.blocks + .slice(prevBlockCount) + .filter((block) => block.type === 'tool_call') + const batchState = createToolBatchState(toolCalls) let nextInteractionOrder = 0 const takeInteractionOrder = () => nextInteractionOrder++ - for (const tc of state.completedToolCalls) { + for (const tc of toolCalls) { const toolDef = tools.find((t) => t.function.name === tc.name) if (!toolDef) continue - const block = state.blocks.find((b) => b.type === 'tool_call' && b.tool_call?.id === tc.id) + const block = batchToolCallBlocks.find((candidate) => candidate.tool_call?.id === tc.id) if (!block?.tool_call) continue block.tool_call.server_name = toolDef.server.name block.tool_call.server_icons = toolDef.server.icons @@ -1570,7 +1735,7 @@ export async function executeTools( const assistantMessage: ChatMessage = { role: 'assistant', content: assistantContent, - tool_calls: state.completedToolCalls.map((tc) => ({ + tool_calls: toolCalls.map((tc) => ({ id: tc.id, type: 'function' as const, function: { name: tc.name, arguments: tc.arguments }, @@ -1599,7 +1764,7 @@ export async function executeTools( modelId, providerDbSourceUrl: interleavedReasoning.providerDbSourceUrl, reasoningContentLength: reasoning.length, - toolCallCount: state.completedToolCalls.length + toolCallCount: toolCalls.length } diagnostics?.onInterleavedReasoningGap?.(gapPayload) if (!diagnostics?.onInterleavedReasoningGap) { @@ -1614,13 +1779,53 @@ export async function executeTools( const pendingInteractions: ToolBatchInteraction[] = [] const stagedResults: StagedToolResult[] = [] + if (disposition.kind === 'reject') { + for (const toolCall of toolCalls) { + const toolDef = tools.find((candidate) => candidate.function.name === toolCall.name) + stagedResults.push({ + toolCallId: toolCall.id, + toolName: toolCall.name, + toolSource: toolDef?.source, + serverName: toolDef?.server.name, + toolArgs: toolCall.arguments, + responseText: TRUNCATED_TOOL_CALL_ERROR, + isError: true, + searchPayload: null, + postHookKind: 'failure', + skippedReason: 'max_tokens' + }) + } + + return await commitStagedToolResults({ + stagedResults, + pendingInteractions, + batchState, + executed, + toolsChanged, + conversation, + state, + batchToolCallBlocks, + toolBlockStartIndex: prevBlockCount, + io, + notificationObserver, + takeInteractionOrder, + toolResults, + tools, + contextLength, + maxTokens, + rendererFlushHandle + }) + } + + const toolPermissionMode = getToolCapabilityPermissionMode(permissionMode) + const canRunReadOnlyBatchInParallel = permissionMode === 'full_access' && - state.completedToolCalls.length > 1 && - state.completedToolCalls.every((tc) => isParallelReadOnlyToolCall(tc, tools)) + toolCalls.length > 1 && + toolCalls.every((tc) => isParallelReadOnlyToolCall(tc, tools)) if (canRunReadOnlyBatchInParallel) { - const executions = state.completedToolCalls.map((tc) => + const executions = toolCalls.map((tc) => buildToolExecutionContext(tc, tools, io.sessionId, providerId) ) @@ -1664,6 +1869,7 @@ export async function executeTools( controls, io, state, + batchToolCallBlocks, rendererFlushHandle, allowProgressUpdates: false, onToolCallStarted @@ -1709,7 +1915,7 @@ export async function executeTools( takeInteractionOrder() ) pendingInteractions.push(interaction) - updateToolCallBlock(state.blocks, outcome.toolContext.id, '', false) + updateToolCallBlock(batchToolCallBlocks, outcome.toolContext.id, '', false) rescheduleRendererFlush(state, rendererFlushHandle) continue } @@ -1723,52 +1929,28 @@ export async function executeTools( throw cancellationError } - if (stagedResults.length > 0) { - const fittedResults = await toolResults.fitBatch({ - conversationMessages: conversation, - results: stagedResults.map((result) => ({ - toolCallId: result.toolCallId, - toolName: result.toolName, - responseText: result.responseText, - isError: result.isError, - offloadPath: result.offloadPath - })), - toolDefinitions: tools, - contextLength, - maxTokens - }) - const finalizedInteractions = applyFinalizedToolResults({ - stagedResults, - fittedResults: fittedResults.results, - conversation, - state, - io, - notificationObserver, - appendToConversation: fittedResults.kind === 'ok', - takeInteractionOrder - }) - pendingInteractions.push(...finalizedInteractions) - for (const result of stagedResults) { - batchState.committedResultCallIds.add(result.toolCallId) - } - persistToolExecutionState(io, state, rendererFlushHandle) - - if (fittedResults.kind === 'terminal_error') { - return buildToolBatchOutcome( - batchState, - pendingInteractions, - executed, - toolsChanged, - fittedResults.message - ) - } - } - - persistToolExecutionState(io, state, rendererFlushHandle) - return buildToolBatchOutcome(batchState, pendingInteractions, executed, toolsChanged) + return await commitStagedToolResults({ + stagedResults, + pendingInteractions, + batchState, + executed, + toolsChanged, + conversation, + state, + batchToolCallBlocks, + toolBlockStartIndex: prevBlockCount, + io, + notificationObserver, + takeInteractionOrder, + toolResults, + tools, + contextLength, + maxTokens, + rendererFlushHandle + }) } - for (const tc of state.completedToolCalls) { + for (const tc of toolCalls) { if (io.abortSignal.aborted && stagedResults.length > 0) { break } @@ -1801,7 +1983,7 @@ export async function executeTools( tool_call_id: tc.id, content: errorText }) - updateToolCallBlock(state.blocks, tc.id, errorText, true) + updateToolCallBlock(batchToolCallBlocks, tc.id, errorText, true) state.dirty = true batchState.committedResultCallIds.add(tc.id) executed += 1 @@ -1824,7 +2006,7 @@ export async function executeTools( takeInteractionOrder() ) pendingInteractions.push(interaction) - updateToolCallBlock(state.blocks, tc.id, '', false) + updateToolCallBlock(batchToolCallBlocks, tc.id, '', false) rescheduleRendererFlush(state, rendererFlushHandle) continue } @@ -1854,6 +2036,7 @@ export async function executeTools( controls, io, state, + batchToolCallBlocks, rendererFlushHandle, execution, permission: preCheckedPermission, @@ -1881,7 +2064,7 @@ export async function executeTools( takeInteractionOrder() ) pendingInteractions.push(interaction) - updateToolCallBlock(state.blocks, tc.id, '', false) + updateToolCallBlock(batchToolCallBlocks, tc.id, '', false) rescheduleRendererFlush(state, rendererFlushHandle) continue } @@ -1904,7 +2087,7 @@ export async function executeTools( takeInteractionOrder() ) pendingInteractions.push(interaction) - updateToolCallBlock(state.blocks, tc.id, '', false) + updateToolCallBlock(batchToolCallBlocks, tc.id, '', false) rescheduleRendererFlush(state, rendererFlushHandle) continue } @@ -1920,6 +2103,7 @@ export async function executeTools( controls, io, state, + batchToolCallBlocks, rendererFlushHandle, execution, permission: reviewPermission, @@ -1944,7 +2128,7 @@ export async function executeTools( takeInteractionOrder() ) pendingInteractions.push(interaction) - updateToolCallBlock(state.blocks, tc.id, '', false) + updateToolCallBlock(batchToolCallBlocks, tc.id, '', false) rescheduleRendererFlush(state, rendererFlushHandle) continue } @@ -1968,6 +2152,7 @@ export async function executeTools( controls, io, state, + batchToolCallBlocks, rendererFlushHandle, allowProgressUpdates: true, onToolCallStarted @@ -1993,7 +2178,7 @@ export async function executeTools( takeInteractionOrder() ) pendingInteractions.push(interaction) - updateToolCallBlock(state.blocks, tc.id, '', false) + updateToolCallBlock(batchToolCallBlocks, tc.id, '', false) rescheduleRendererFlush(state, rendererFlushHandle) continue } @@ -2022,49 +2207,25 @@ export async function executeTools( } } - if (stagedResults.length > 0) { - const fittedResults = await toolResults.fitBatch({ - conversationMessages: conversation, - results: stagedResults.map((result) => ({ - toolCallId: result.toolCallId, - toolName: result.toolName, - responseText: result.responseText, - isError: result.isError, - offloadPath: result.offloadPath - })), - toolDefinitions: tools, - contextLength, - maxTokens - }) - const finalizedInteractions = applyFinalizedToolResults({ - stagedResults, - fittedResults: fittedResults.results, - conversation, - state, - io, - notificationObserver, - appendToConversation: fittedResults.kind === 'ok', - takeInteractionOrder - }) - pendingInteractions.push(...finalizedInteractions) - for (const result of stagedResults) { - batchState.committedResultCallIds.add(result.toolCallId) - } - persistToolExecutionState(io, state, rendererFlushHandle) - - if (fittedResults.kind === 'terminal_error') { - return buildToolBatchOutcome( - batchState, - pendingInteractions, - executed, - toolsChanged, - fittedResults.message - ) - } - } - - persistToolExecutionState(io, state, rendererFlushHandle) - return buildToolBatchOutcome(batchState, pendingInteractions, executed, toolsChanged) + return await commitStagedToolResults({ + stagedResults, + pendingInteractions, + batchState, + executed, + toolsChanged, + conversation, + state, + batchToolCallBlocks, + toolBlockStartIndex: prevBlockCount, + io, + notificationObserver, + takeInteractionOrder, + toolResults, + tools, + contextLength, + maxTokens, + rendererFlushHandle + }) } function stampGenerationTiming(state: StreamState): void { diff --git a/src/main/agent/deepchat/runtime/imageGenerationBlocks.ts b/src/main/agent/deepchat/runtime/imageGenerationBlocks.ts index 5e034a9422..b6502ce663 100644 --- a/src/main/agent/deepchat/runtime/imageGenerationBlocks.ts +++ b/src/main/agent/deepchat/runtime/imageGenerationBlocks.ts @@ -86,14 +86,16 @@ export function prepareToolImagePreviewPresentation(params: { export function insertBlocksAfterToolCall( blocks: AssistantMessageBlock[], toolCallId: string, - newBlocks: AssistantMessageBlock[] + newBlocks: AssistantMessageBlock[], + startIndex = 0 ): void { if (newBlocks.length === 0) { return } const toolBlockIndex = blocks.findIndex( - (block) => block.type === 'tool_call' && block.tool_call?.id === toolCallId + (block, index) => + index >= startIndex && block.type === 'tool_call' && block.tool_call?.id === toolCallId ) if (toolBlockIndex === -1) { blocks.push(...newBlocks) diff --git a/src/main/agent/deepchat/runtime/process.ts b/src/main/agent/deepchat/runtime/process.ts index b75acefbc0..63b67fe49c 100644 --- a/src/main/agent/deepchat/runtime/process.ts +++ b/src/main/agent/deepchat/runtime/process.ts @@ -7,16 +7,18 @@ import type { PendingToolInteraction, ProcessParams, ProcessResult, - StreamState + StreamState, + ToolCallResult } from './types' import { accumulate, commitRoundUsage, finalizeTrailingPendingNarrativeBlocks } from './accumulator' import { startEcho } from './echo' import { - executeTools, finalize, finalizeError, finalizePaused, - publishPlanUpdated + publishPlanUpdated, + settleToolBatch, + type ToolBatchDisposition } from './dispatch' import { isContextWindowErrorLike } from './contextWindowError' import { @@ -34,6 +36,7 @@ import type { OutputSink } from '@/agent/deepchat/loop/ports' import { buildTapeToolFactInputs } from '@/tape/application/factPersistence' const UNKNOWN_CONTEXT_LIMIT = Number.MAX_SAFE_INTEGER +const MAX_TRUNCATED_TOOL_RECOVERY_ATTEMPTS = 1 const USER_CANCELED_GENERATION_ERROR = 'common.error.userCanceledGeneration' export const NO_MODEL_RESPONSE_ERROR = 'common.error.noModelResponse' export const INCOMPLETE_PROVIDER_STREAM_ERROR = @@ -43,7 +46,12 @@ export const INCOMPLETE_TOOL_USE_ERROR = const deepChatLoopEngine = new DeepChatLoopEngine() type PendingPermissionPayload = NonNullable type PendingPermissionCommandInfo = NonNullable -type ToolRoundBatch = { prevBlockCount: number } +type ToolRoundBatch = { + prevBlockCount: number + toolCalls: ToolCallResult[] + disposition: ToolBatchDisposition + nextAction: 'continue' | 'terminal' +} class MaxProviderRoundsError extends Error { constructor(limit: number) { @@ -120,6 +128,115 @@ function markUnexecutedToolCallsForLimit(state: StreamState): void { } } +function countVisibleToolCallIds(blocks: readonly AssistantMessageBlock[]): Map { + const counts = new Map() + for (const block of blocks) { + const toolCallId = block.type === 'tool_call' ? block.tool_call?.id : undefined + if (toolCallId?.trim()) { + counts.set(toolCallId, (counts.get(toolCallId) ?? 0) + 1) + } + } + return counts +} + +function collectOrderedTruncatedDeepChatToolCalls( + state: StreamState, + prevBlockCount: number +): ToolCallResult[] { + const roundBlocks = state.blocks.slice(prevBlockCount) + const visibleCallIdCounts = countVisibleToolCallIds(roundBlocks) + const completedById = new Map(state.completedToolCalls.map((toolCall) => [toolCall.id, toolCall])) + const seenCallIds = new Set() + const orderedCalls: ToolCallResult[] = [] + + for (const block of roundBlocks) { + if (block.type !== 'tool_call') { + continue + } + const toolCallId = block.tool_call?.id + const toolCallName = block.tool_call?.name + if ( + !toolCallId?.trim() || + !toolCallName?.trim() || + visibleCallIdCounts.get(toolCallId) !== 1 || + seenCallIds.has(toolCallId) + ) { + continue + } + + const completed = completedById.get(toolCallId) + if (completed?.name.trim()) { + orderedCalls.push({ ...completed }) + seenCallIds.add(toolCallId) + continue + } + + const pending = state.pendingToolCalls.get(toolCallId) + if (!pending || pending.executionOwner !== 'deepchat') { + continue + } + + orderedCalls.push({ + id: toolCallId, + name: pending.name, + arguments: pending.arguments, + ...(pending.providerOptions ? { providerOptions: pending.providerOptions } : {}) + }) + seenCallIds.add(toolCallId) + } + + // Keep compatibility with custom stream producers that complete a call without a visible block. + for (const completed of state.completedToolCalls) { + if ( + completed.id.trim() && + completed.name.trim() && + !visibleCallIdCounts.has(completed.id) && + !seenCallIds.has(completed.id) + ) { + orderedCalls.push({ ...completed }) + seenCallIds.add(completed.id) + } + } + + return orderedCalls +} + +function markOtherTruncatedToolCallsIncomplete( + state: StreamState, + prevBlockCount: number, + rejectedToolCalls: readonly ToolCallResult[] +): void { + const rejectedCallIds = new Set(rejectedToolCalls.map((toolCall) => toolCall.id)) + const roundBlocks = state.blocks.slice(prevBlockCount) + const visibleCallIdCounts = countVisibleToolCallIds(roundBlocks) + + for (const block of roundBlocks) { + if (block.type !== 'tool_call') { + continue + } + const toolCallId = block.tool_call?.id + if ( + toolCallId?.trim() && + block.tool_call?.name?.trim() && + visibleCallIdCounts.get(toolCallId) === 1 + ) { + if ( + rejectedCallIds.has(toolCallId) || + (block.status !== 'pending' && block.status !== 'loading') + ) { + continue + } + } + + block.status = 'error' + block.extra = { + ...block.extra, + toolCallIncompleteReason: 'max_tokens' + } + state.dirty = true + } +} + export type ProviderTerminalDecision = | { type: 'complete' @@ -702,6 +819,7 @@ export async function processStream(params: ProcessParams): Promise echo.flush(), @@ -898,7 +1016,17 @@ export async function processStream(params: ProcessParams): Promise 0) { + const nextAction = + truncatedToolRecoveryAttempts < MAX_TRUNCATED_TOOL_RECOVERY_ATTEMPTS + ? 'continue' + : 'terminal' + if (nextAction === 'continue') { + truncatedToolRecoveryAttempts += 1 + } + + return { + type: 'tool_batch', + batch: { + prevBlockCount, + toolCalls: truncatedToolCalls, + disposition: { kind: 'reject', reason: 'output_truncated' }, + nextAction + }, + requestedToolExecutionCount: 0 + } + } + + const completedToolCalls = + state.stopReason === 'tool_use' + ? state.completedToolCalls.map((toolCall) => ({ ...toolCall })) + : [] + if (completedToolCalls.length === 0) { return { type: 'terminal' } } return { type: 'tool_batch', - batch: { prevBlockCount }, - toolCallCount: state.completedToolCalls.length + batch: { + prevBlockCount, + toolCalls: completedToolCalls, + disposition: { kind: 'execute' }, + nextAction: 'continue' + }, + requestedToolExecutionCount: completedToolCalls.length } }, - executeToolBatch: async ({ batch }) => { - // A completed tool call implies that the tool presenter and definitions were available. - const completedToolBatch = state.completedToolCalls.map((toolCall) => ({ ...toolCall })) + settleToolBatch: async ({ batch }) => { + const completedToolBatch = batch.toolCalls.map((toolCall) => ({ ...toolCall })) const toolBatchMessageStart = conversationMessages.length let startedToolCallCount = 0 - let executed: Awaited> + let executed: Awaited> try { - executed = await executeTools( + executed = await settleToolBatch({ state, - conversationMessages, - batch.prevBlockCount, - currentTools, + conversation: conversationMessages, + prevBlockCount: batch.prevBlockCount, + toolCalls: batch.toolCalls, + disposition: batch.disposition, + tools: currentTools, toolExecution, modelId, interleavedReasoning, io, permissionMode, toolResults, - providerId === 'acp' - ? Number.MAX_SAFE_INTEGER - : modelConfig.contextLength > 0 - ? modelConfig.contextLength - : UNKNOWN_CONTEXT_LIMIT, + contextLength: + providerId === 'acp' + ? Number.MAX_SAFE_INTEGER + : modelConfig.contextLength > 0 + ? modelConfig.contextLength + : UNKNOWN_CONTEXT_LIMIT, maxTokens, - echo, - { + rendererFlushHandle: echo, + collaborators: { notificationObserver, controls, diagnostics, @@ -954,7 +1114,7 @@ export async function processStream(params: ProcessParams): Promise diff --git a/src/main/provider/aiSdk/streamAdapter.ts b/src/main/provider/aiSdk/streamAdapter.ts index e8349c675e..7a30ad0282 100644 --- a/src/main/provider/aiSdk/streamAdapter.ts +++ b/src/main/provider/aiSdk/streamAdapter.ts @@ -149,7 +149,8 @@ export async function* adaptAiSdkStream( yield createStreamEvent.toolCallStart( part.id, part.toolName, - toProviderOptions((part as any).providerMetadata) + toProviderOptions((part as any).providerMetadata), + part.providerExecuted === true ? 'provider' : undefined ) break @@ -175,7 +176,12 @@ export async function* adaptAiSdkStream( if (!endedToolCalls.has(part.toolCallId)) { const serializedInput = JSON.stringify(part.input ?? {}) const providerOptions = toProviderOptions((part as any).providerMetadata) - yield createStreamEvent.toolCallStart(part.toolCallId, part.toolName, providerOptions) + yield createStreamEvent.toolCallStart( + part.toolCallId, + part.toolName, + providerOptions, + part.providerExecuted === true ? 'provider' : undefined + ) yield createStreamEvent.toolCallChunk(part.toolCallId, serializedInput, providerOptions) yield createStreamEvent.toolCallEnd(part.toolCallId, serializedInput, providerOptions) endedToolCalls.add(part.toolCallId) diff --git a/src/shared/chat.d.ts b/src/shared/chat.d.ts index 240d1390cd..c87e9a65a5 100644 --- a/src/shared/chat.d.ts +++ b/src/shared/chat.d.ts @@ -223,6 +223,8 @@ export type AssistantMessageExtra = Record ({ type: 'tool_call_start', tool_call_id, tool_call_name, + ...(tool_call_execution_owner ? { tool_call_execution_owner } : {}), ...(provider_options ? { provider_options } : {}) }), toolCallChunk: ( diff --git a/test/main/agent/acp/runtime/acpContentMapper.test.ts b/test/main/agent/acp/runtime/acpContentMapper.test.ts index 3f3eb6a6eb..5a30bea68f 100644 --- a/test/main/agent/acp/runtime/acpContentMapper.test.ts +++ b/test/main/agent/acp/runtime/acpContentMapper.test.ts @@ -70,7 +70,8 @@ describe('AcpContentMapper tool call handling', () => { expect(startEvent).toMatchObject({ type: 'tool_call_start', tool_call_id: toolCallId, - tool_call_name: 'write_file' + tool_call_name: 'write_file', + tool_call_execution_owner: 'provider' }) const chunkEvent = start.events.find((event) => event.type === 'tool_call_chunk') diff --git a/test/main/agent/deepchat/loop/deepChatLoopEngine.test.ts b/test/main/agent/deepchat/loop/deepChatLoopEngine.test.ts index ab637a1389..11ccbb5759 100644 --- a/test/main/agent/deepchat/loop/deepChatLoopEngine.test.ts +++ b/test/main/agent/deepchat/loop/deepChatLoopEngine.test.ts @@ -40,7 +40,7 @@ describe('DeepChatLoopEngine', () => { it('settles a simple provider round without executing tools', async () => { const run = createRun() const consumeProviderRound = vi.fn(async () => ({ type: 'terminal' as const })) - const executeToolBatch = vi.fn(async () => ({ + const settleToolBatch = vi.fn(async () => ({ type: 'continue' as const, executedToolCount: 0 })) @@ -49,7 +49,7 @@ describe('DeepChatLoopEngine', () => { run, { consumeProviderRound, - executeToolBatch + settleToolBatch }, createCommitCallbacks() ) @@ -57,7 +57,7 @@ describe('DeepChatLoopEngine', () => { expect(outcome).toEqual({ type: 'terminal' }) expect(run.providerRoundCount).toBe(1) expect(consumeProviderRound).toHaveBeenCalledTimes(1) - expect(executeToolBatch).not.toHaveBeenCalled() + expect(settleToolBatch).not.toHaveBeenCalled() }) it('owns provider and tool-batch alternation across multiple rounds', async () => { @@ -73,11 +73,11 @@ describe('DeepChatLoopEngine', () => { ? { type: 'tool_batch', batch: { id: providerRound }, - toolCallCount: 1 + requestedToolExecutionCount: 1 } : { type: 'terminal' } }, - executeToolBatch: async ({ batch }) => { + settleToolBatch: async ({ batch }) => { order.push(`tools:${batch.id}`) return { type: 'continue', executedToolCount: 1 } } @@ -107,9 +107,9 @@ describe('DeepChatLoopEngine', () => { const consumeProviderRound = vi.fn(async ({ providerRound }: { providerRound: number }) => ({ type: 'tool_batch' as const, batch: providerRound, - toolCallCount: 1 + requestedToolExecutionCount: 1 })) - const executeToolBatch = vi.fn(async () => ({ + const settleToolBatch = vi.fn(async () => ({ type: 'continue' as const, executedToolCount: 1 })) @@ -119,7 +119,7 @@ describe('DeepChatLoopEngine', () => { { maxProviderRounds: 2, consumeProviderRound, - executeToolBatch + settleToolBatch }, createCommitCallbacks() ) @@ -127,12 +127,12 @@ describe('DeepChatLoopEngine', () => { expect(outcome).toEqual({ type: 'max_provider_rounds', limit: 2 }) expect(run.providerRoundCount).toBe(3) expect(consumeProviderRound).toHaveBeenCalledTimes(2) - expect(executeToolBatch).toHaveBeenCalledTimes(2) + expect(settleToolBatch).toHaveBeenCalledTimes(2) }) it('stops before a tool batch would exceed the global tool-call limit', async () => { const run = createRun() - const executeToolBatch = vi.fn(async () => ({ + const settleToolBatch = vi.fn(async () => ({ type: 'continue' as const, executedToolCount: 129 })) @@ -143,9 +143,9 @@ describe('DeepChatLoopEngine', () => { consumeProviderRound: async () => ({ type: 'tool_batch' as const, batch: 'oversized', - toolCallCount: 129 + requestedToolExecutionCount: 129 }), - executeToolBatch + settleToolBatch }, createCommitCallbacks() ) @@ -155,12 +155,12 @@ describe('DeepChatLoopEngine', () => { attemptedToolCount: 129, limit: 128 }) - expect(executeToolBatch).not.toHaveBeenCalled() + expect(settleToolBatch).not.toHaveBeenCalled() }) it('includes previously executed tools when enforcing the global tool-call limit', async () => { const run = createRun() - const executeToolBatch = vi.fn(async () => ({ + const settleToolBatch = vi.fn(async () => ({ type: 'continue' as const, executedToolCount: 1 })) @@ -172,9 +172,9 @@ describe('DeepChatLoopEngine', () => { consumeProviderRound: async () => ({ type: 'tool_batch' as const, batch: 'resumed-overflow', - toolCallCount: 1 + requestedToolExecutionCount: 1 }), - executeToolBatch + settleToolBatch }, createCommitCallbacks() ) @@ -184,7 +184,64 @@ describe('DeepChatLoopEngine', () => { attemptedToolCount: 129, limit: 128 }) - expect(executeToolBatch).not.toHaveBeenCalled() + expect(settleToolBatch).not.toHaveBeenCalled() + }) + + it('settles a zero-request batch without consuming the tool-call budget', async () => { + const run = createRun() + const consumeProviderRound = vi.fn(async ({ providerRound }: { providerRound: number }) => + providerRound === 1 + ? { + type: 'tool_batch' as const, + batch: 'rejected', + requestedToolExecutionCount: 0 + } + : { type: 'terminal' as const } + ) + const settleToolBatch = vi.fn(async () => ({ + type: 'continue' as const, + executedToolCount: 0 + })) + + const outcome = await new DeepChatLoopEngine().run( + run, + { + initialExecutedToolCount: 128, + consumeProviderRound, + settleToolBatch + }, + createCommitCallbacks() + ) + + expect(outcome).toEqual({ type: 'terminal' }) + expect(consumeProviderRound).toHaveBeenCalledTimes(2) + expect(settleToolBatch).toHaveBeenCalledTimes(1) + }) + + it('terminates normally after persisting a terminal tool settlement', async () => { + const run = createRun() + const order: string[] = [] + const consumeProviderRound = vi.fn(async () => ({ + type: 'tool_batch' as const, + batch: 'terminal-batch', + requestedToolExecutionCount: 0 + })) + + const outcome = await new DeepChatLoopEngine().run( + run, + { + consumeProviderRound, + settleToolBatch: async () => ({ + type: 'terminal' as const, + executedToolCount: 0 + }) + }, + createCommitCallbacks(order) + ) + + expect(outcome).toEqual({ type: 'terminal' }) + expect(order).toEqual(['output:1:tool_batch', 'persisted:1:terminal', 'settled']) + expect(consumeProviderRound).toHaveBeenCalledTimes(1) }) it('propagates a paused tool batch without entering another provider round', async () => { @@ -193,14 +250,14 @@ describe('DeepChatLoopEngine', () => { const consumeProviderRound = vi.fn(async () => ({ type: 'tool_batch' as const, batch: 'permission', - toolCallCount: 1 + requestedToolExecutionCount: 1 })) const outcome = await new DeepChatLoopEngine().run( run, { consumeProviderRound, - executeToolBatch: async () => ({ + settleToolBatch: async () => ({ type: 'halted' as const, result: { status: 'paused' as const } }) @@ -225,7 +282,7 @@ describe('DeepChatLoopEngine', () => { consumeProviderRound: async () => { throw error }, - executeToolBatch: async () => ({ + settleToolBatch: async () => ({ type: 'continue' as const, executedToolCount: 0 }) @@ -249,7 +306,7 @@ describe('DeepChatLoopEngine', () => { run, { consumeProviderRound: async () => ({ type: 'terminal' as const }), - executeToolBatch: async () => ({ + settleToolBatch: async () => ({ type: 'continue' as const, executedToolCount: 0 }) @@ -279,7 +336,7 @@ describe('DeepChatLoopEngine', () => { consumeProviderRound: async () => { throw providerError }, - executeToolBatch: async () => ({ + settleToolBatch: async () => ({ type: 'continue' as const, executedToolCount: 0 }) @@ -298,7 +355,7 @@ describe('DeepChatLoopEngine', () => { 'propagates a %s provider outcome without executing tools', async (status) => { const run = createRun() - const executeToolBatch = vi.fn(async () => ({ + const settleToolBatch = vi.fn(async () => ({ type: 'continue' as const, executedToolCount: 0 })) @@ -310,13 +367,13 @@ describe('DeepChatLoopEngine', () => { type: 'halted' as const, result: { status } }), - executeToolBatch + settleToolBatch }, createCommitCallbacks() ) expect(outcome).toEqual({ type: 'halted', result: { status } }) - expect(executeToolBatch).not.toHaveBeenCalled() + expect(settleToolBatch).not.toHaveBeenCalled() } ) @@ -330,10 +387,10 @@ describe('DeepChatLoopEngine', () => { consumeProviderRound: async ({ providerRound, run: currentRun }) => { observedSkills.push([...currentRun.resources.activeSkillNames]) return providerRound === 1 - ? { type: 'tool_batch', batch: 'skill_view', toolCallCount: 1 } + ? { type: 'tool_batch', batch: 'skill_view', requestedToolExecutionCount: 1 } : { type: 'terminal' } }, - executeToolBatch: async ({ run: currentRun }) => { + settleToolBatch: async ({ run: currentRun }) => { currentRun.resources.activeSkillNames = ['deepchat-settings'] return { type: 'continue', executedToolCount: 1 } } diff --git a/test/main/agent/deepchat/runtime/accumulator.test.ts b/test/main/agent/deepchat/runtime/accumulator.test.ts index 52b6722e28..732710530b 100644 --- a/test/main/agent/deepchat/runtime/accumulator.test.ts +++ b/test/main/agent/deepchat/runtime/accumulator.test.ts @@ -213,6 +213,40 @@ describe('accumulate', () => { }) }) + it('keeps provider-owned calls out of the local execution queue', () => { + accumulate(state, { + type: 'tool_call_start', + tool_call_id: 'provider-tc1', + tool_call_name: 'web_search', + tool_call_execution_owner: 'provider' + }) + accumulate(state, { + type: 'tool_call_chunk', + tool_call_id: 'provider-tc1', + tool_call_arguments_chunk: '{"query":"deepchat"}' + }) + accumulate(state, { + type: 'tool_call_end', + tool_call_id: 'provider-tc1', + tool_call_response: 'provider result', + tool_call_status: 'success' + }) + + expect(state.pendingToolCalls.size).toBe(0) + expect(state.completedToolCalls).toHaveLength(0) + expect(state.blocks[0]).toMatchObject({ + type: 'tool_call', + status: 'success', + extra: { toolCallArgsComplete: true }, + tool_call: { + id: 'provider-tc1', + name: 'web_search', + params: '{"query":"deepchat"}', + response: 'provider result' + } + }) + }) + it('tool_call_end with complete args overrides accumulated chunks', () => { accumulate(state, { type: 'tool_call_start', diff --git a/test/main/agent/deepchat/runtime/contextBuilder.test.ts b/test/main/agent/deepchat/runtime/contextBuilder.test.ts index eb660ddc0c..20a6f951a6 100644 --- a/test/main/agent/deepchat/runtime/contextBuilder.test.ts +++ b/test/main/agent/deepchat/runtime/contextBuilder.test.ts @@ -12,6 +12,7 @@ import { truncateContext } from '@/agent/deepchat/runtime/contextBuilder' import { buildContextCheckpoint } from '@/agent/deepchat/runtime/contextContributions' +import { TRUNCATED_TOOL_CALL_ERROR } from '@/agent/deepchat/runtime/dispatch' vi.mock('tokenx', () => ({ approximateTokenSize: vi.fn((text: string) => { @@ -911,6 +912,54 @@ describe('buildContext', () => { ]) }) + it('rebuilds a rejected truncated call with its matching error result', () => { + const rejectedRecord = { + id: 'asst-2', + sessionId: 's1', + orderSeq: 2, + role: 'assistant' as const, + content: JSON.stringify([ + { + type: 'tool_call', + status: 'error', + timestamp: Date.now(), + extra: { toolCallSkippedReason: 'max_tokens' }, + tool_call: { + id: 'tc-truncated', + name: 'read', + params: '{"path":"', + response: TRUNCATED_TOOL_CALL_ERROR + } + } + ]), + status: 'sent' as const, + isContextEdge: 0, + metadata: '{}', + createdAt: Date.now(), + updatedAt: Date.now() + } + const store = createMockMessageStore([makeUserRecord(1, 'read a file'), rejectedRecord]) + + const result = buildContext('s1', { text: 'continue', files: [] }, '', 10000, 4096, store) + + expect(result).toEqual([ + { role: 'user', content: 'read a file' }, + { + role: 'assistant', + content: '', + tool_calls: [ + { + id: 'tc-truncated', + type: 'function', + function: { name: 'read', arguments: '{"path":"' } + } + ] + }, + { role: 'tool', tool_call_id: 'tc-truncated', content: TRUNCATED_TOOL_CALL_ERROR }, + { role: 'user', content: 'continue' } + ]) + }) + it('replays settled tool call provider options for follow-up turns', () => { const messages = [ makeUserRecord(1, 'check this'), diff --git a/test/main/agent/deepchat/runtime/dispatch.test.ts b/test/main/agent/deepchat/runtime/dispatch.test.ts index e8b666182f..bdc8052920 100644 --- a/test/main/agent/deepchat/runtime/dispatch.test.ts +++ b/test/main/agent/deepchat/runtime/dispatch.test.ts @@ -14,7 +14,7 @@ import { createState } from '@/agent/deepchat/runtime/types' import { estimateMessagesTokens } from '@/agent/deepchat/runtime/contextBuilder' import type { MCPToolDefinition } from '@shared/types/mcp' import type { ToolServicePort } from '@shared/types/tool' -import type { PermissionMode } from '@shared/types/agent-interface' +import type { AssistantMessageBlock, PermissionMode } from '@shared/types/agent-interface' import { ToolOutputGuard } from '@/agent/deepchat/runtime/toolOutputGuard' import { createToolExecutionPort, @@ -47,10 +47,12 @@ vi.mock('@/events', () => ({ })) import { - executeTools as executeToolsInternal, finalize, finalizeError, - persistAbortExceptionPlanState + persistAbortExceptionPlanState, + settleToolBatch as settleToolBatchInternal, + TRUNCATED_TOOL_CALL_ERROR, + type ToolBatchDisposition } from '@/agent/deepchat/runtime/dispatch' import type { EchoHandle } from '@/agent/deepchat/runtime/echo' import { accumulate } from '@/agent/deepchat/runtime/accumulator' @@ -151,7 +153,7 @@ function expectDeepchatEvent(eventName: string, payload: Record expect(publishDeepchatEventMock).toHaveBeenCalledWith(eventName, expect.objectContaining(payload)) } -async function executeTools( +async function settleToolBatch( state: StreamState, conversation: any[], prevBlockCount: number, @@ -166,7 +168,8 @@ async function executeTools( hooks?: TestHooks, providerId?: string, interleavedReasoning: InterleavedReasoningConfig = DEFAULT_INTERLEAVED_REASONING, - rendererFlushHandle?: Pick + rendererFlushHandle?: Pick, + disposition: ToolBatchDisposition = { kind: 'execute' } ) { const toolExecution = createToolExecutionPort(toolService)! const toolResults = createToolResultPort({ @@ -211,10 +214,12 @@ async function executeTools( }) } satisfies Pick) - return executeToolsInternal( + return settleToolBatchInternal({ state, conversation, prevBlockCount, + toolCalls: state.completedToolCalls, + disposition, tools, toolExecution, modelId, @@ -224,8 +229,8 @@ async function executeTools( toolResults, contextLength, maxTokens, - flushHandle, - { + rendererFlushHandle: flushHandle, + collaborators: { notificationObserver: hooks ? { notify: (notification) => { @@ -245,7 +250,7 @@ async function executeTools( diagnostics: hooks }, providerId - ) + }) } describe('dispatch', () => { @@ -269,7 +274,7 @@ describe('dispatch', () => { } }) - describe('executeTools', () => { + describe('settleToolBatch', () => { it('builds assistant message, calls tools, updates blocks', async () => { const tools = [makeTool('get_weather')] const toolService = createMockToolService({ get_weather: 'Sunny, 72F' }) @@ -291,7 +296,7 @@ describe('dispatch', () => { }) state.completedToolCalls = [{ id: 'tc1', name: 'get_weather', arguments: '{}' }] - const executed = await executeTools( + const executed = await settleToolBatch( state, conversation, 0, @@ -333,6 +338,252 @@ describe('dispatch', () => { expect(toolBlock!.status).toBe('success') }) + it('rejects an output-truncated batch atomically without tool side effects', async () => { + const calls = [ + { + id: 'tc-question', + name: QUESTION_TOOL_NAME, + arguments: '{"question":"', + providerOptions: { openai: { itemId: 'item-question' } } + }, + { id: 'tc-skill', name: 'skill_view', arguments: '{"skill":"draft"}' } + ] + const tools = calls.map((call) => makeAgentTool(call.name)) + const toolService = createMockToolService() + const conversation: any[] = [{ role: 'user', content: 'Continue' }] + const hooks = { + autoGrantPermission: vi.fn(), + reviewToolPermission: vi.fn(), + activateSkill: vi.fn(), + onPreToolUse: vi.fn(), + onPostToolUse: vi.fn(), + onPostToolUseFailure: vi.fn(), + onPermissionRequest: vi.fn() + } + state.completedToolCalls = calls + state.blocks.push( + ...calls.map((call) => ({ + type: 'tool_call' as const, + content: '', + status: 'pending' as const, + timestamp: Date.now(), + tool_call: { + id: call.id, + name: call.name, + params: call.arguments, + response: '' + }, + ...(call.providerOptions + ? { extra: { providerOptionsJson: JSON.stringify(call.providerOptions) } } + : {}) + })) + ) + + const result = await settleToolBatch( + state, + conversation, + 0, + tools, + toolService, + 'gpt-4', + io, + 'auto_approve', + new ToolOutputGuard(), + 32000, + 1024, + hooks, + 'openai', + DEFAULT_INTERLEAVED_REASONING, + undefined, + { kind: 'reject', reason: 'output_truncated' } + ) + + expect(result).toMatchObject({ + type: 'completed', + executed: 0, + toolsChanged: false, + executionState: { + callOrder: ['tc-question', 'tc-skill'], + invokedCallIds: [], + committedResultCallIds: ['tc-question', 'tc-skill'], + pendingInteractionCallIds: [] + } + }) + expect(conversation).toEqual([ + { role: 'user', content: 'Continue' }, + { + role: 'assistant', + content: '', + tool_calls: [ + { + id: 'tc-question', + type: 'function', + function: { name: QUESTION_TOOL_NAME, arguments: '{"question":"' }, + provider_options: { openai: { itemId: 'item-question' } } + }, + { + id: 'tc-skill', + type: 'function', + function: { name: 'skill_view', arguments: '{"skill":"draft"}' } + } + ] + }, + { role: 'tool', tool_call_id: 'tc-question', content: TRUNCATED_TOOL_CALL_ERROR }, + { role: 'tool', tool_call_id: 'tc-skill', content: TRUNCATED_TOOL_CALL_ERROR } + ]) + expect(state.blocks).toEqual( + calls.map((call) => + expect.objectContaining({ + type: 'tool_call', + status: 'error', + extra: expect.objectContaining({ toolCallSkippedReason: 'max_tokens' }), + tool_call: expect.objectContaining({ + id: call.id, + name: call.name, + params: call.arguments, + response: TRUNCATED_TOOL_CALL_ERROR + }) + }) + ) + ) + expect(toolService.preCheckToolPermission).not.toHaveBeenCalled() + expect(toolService.callTool).not.toHaveBeenCalled() + expect(hooks.autoGrantPermission).not.toHaveBeenCalled() + expect(hooks.reviewToolPermission).not.toHaveBeenCalled() + expect(hooks.activateSkill).not.toHaveBeenCalled() + expect(hooks.onPreToolUse).not.toHaveBeenCalled() + expect(hooks.onPostToolUse).not.toHaveBeenCalled() + expect(hooks.onPermissionRequest).not.toHaveBeenCalled() + expect(hooks.onPostToolUseFailure.mock.calls.map(([tool]) => tool.callId)).toEqual([ + 'tc-question', + 'tc-skill' + ]) + }) + + it('surfaces a terminal fitting error after rejecting a truncated batch', async () => { + const toolService = createMockToolService() + const hooks = { + onPreToolUse: vi.fn(), + onPostToolUseFailure: vi.fn() + } + state.blocks.push({ + type: 'tool_call', + content: '', + status: 'pending', + timestamp: Date.now(), + tool_call: { id: 'tc1', name: 'read', params: '{"path":"', response: '' } + }) + state.completedToolCalls = [{ id: 'tc1', name: 'read', arguments: '{"path":"' }] + + const result = await settleToolBatch( + state, + [], + 0, + [makeAgentTool('read')], + toolService, + 'gpt-4', + io, + 'full_access', + new ToolOutputGuard(), + 1, + 1, + hooks, + 'openai', + DEFAULT_INTERLEAVED_REASONING, + undefined, + { kind: 'reject', reason: 'output_truncated' } + ) + + expect(result.terminalError).toContain('remaining context window is too small') + expect(result.executionState).toEqual({ + callOrder: ['tc1'], + invokedCallIds: [], + committedResultCallIds: ['tc1'], + pendingInteractionCallIds: [] + }) + expect(toolService.preCheckToolPermission).not.toHaveBeenCalled() + expect(toolService.callTool).not.toHaveBeenCalled() + expect(hooks.onPreToolUse).not.toHaveBeenCalled() + expect(hooks.onPostToolUseFailure).toHaveBeenCalledTimes(1) + expect(state.blocks[0]).toMatchObject({ + status: 'error', + extra: { toolCallSkippedReason: 'max_tokens' }, + tool_call: { response: expect.stringContaining('remaining context window is too small') } + }) + }) + + it('settles a reused call id against the current provider round', async () => { + const previousRoundBlock: AssistantMessageBlock = { + type: 'tool_call', + content: '', + status: 'success', + timestamp: Date.now(), + tool_call: { + id: 'reused-call-id', + name: 'read', + params: '{"path":"complete.txt"}', + response: 'previous result', + server_name: 'previous-server' + } + } + state.blocks.push(previousRoundBlock) + const prevBlockCount = state.blocks.length + state.blocks.push({ + type: 'tool_call', + content: '', + status: 'pending', + timestamp: Date.now(), + tool_call: { + id: 'reused-call-id', + name: 'read', + params: '{"path":"', + response: '' + } + }) + state.completedToolCalls = [ + { id: 'reused-call-id', name: 'read', arguments: '{"path":"' } + ] + + await settleToolBatch( + state, + [], + prevBlockCount, + [makeAgentTool('read')], + createMockToolService(), + 'gpt-4', + io, + 'full_access', + new ToolOutputGuard(), + 32000, + 1024, + undefined, + 'openai', + DEFAULT_INTERLEAVED_REASONING, + undefined, + { kind: 'reject', reason: 'output_truncated' } + ) + + expect(state.blocks[0]).toMatchObject({ + status: 'success', + tool_call: { + id: 'reused-call-id', + params: '{"path":"complete.txt"}', + response: 'previous result', + server_name: 'previous-server' + } + }) + expect(state.blocks[0].extra).toBeUndefined() + expect(state.blocks[1]).toMatchObject({ + status: 'error', + extra: { toolCallSkippedReason: 'max_tokens' }, + tool_call: { + id: 'reused-call-id', + params: '{"path":"', + response: TRUNCATED_TOOL_CALL_ERROR + } + }) + }) + it('rejects calls missing from the current session tool definitions', async () => { const tools = [makeAgentTool('read')] const toolService = createMockToolService() @@ -346,7 +597,7 @@ describe('dispatch', () => { }) state.completedToolCalls = [{ id: 'tc1', name: 'exec', arguments: '{}' }] - const outcome = await executeTools( + const outcome = await settleToolBatch( state, conversation, 0, @@ -410,7 +661,7 @@ describe('dispatch', () => { }) state.completedToolCalls = [{ id: 'tc-plan', name: 'update_plan', arguments: '{}' }] - await executeTools( + await settleToolBatch( state, conversation, 0, @@ -506,7 +757,7 @@ describe('dispatch', () => { }) state.completedToolCalls = [{ id: 'tc-plan', name: 'update_plan', arguments: '{}' }] - await executeTools( + await settleToolBatch( state, conversation, 0, @@ -584,7 +835,7 @@ describe('dispatch', () => { { id: 'tc-read-b', name: 'read', arguments: '{}' } ] - await executeTools( + await settleToolBatch( state, conversation, 0, @@ -665,7 +916,7 @@ describe('dispatch', () => { { id: 'tc-read-b', name: 'read', arguments: '{"path":"b.txt"}' } ] - const execution = executeTools( + const execution = settleToolBatch( state, conversation, 0, @@ -731,7 +982,7 @@ describe('dispatch', () => { { id: 'tc-read-b', name: 'read', arguments: '{"path":"b.txt"}' } ] - const executed = await executeTools( + const executed = await settleToolBatch( state, [], 0, @@ -808,7 +1059,7 @@ describe('dispatch', () => { { id: 'tc-read', name: 'read', arguments: '{}' } ] - const execution = executeTools( + const execution = settleToolBatch( state, [], 0, @@ -866,7 +1117,7 @@ describe('dispatch', () => { }) state.completedToolCalls = [{ id: 'tc1', name: 'subagent_orchestrator', arguments: '{}' }] - await executeTools( + await settleToolBatch( state, [], 0, @@ -944,7 +1195,7 @@ describe('dispatch', () => { } }) - await executeTools( + await settleToolBatch( state, conversation, 0, @@ -1010,7 +1261,7 @@ describe('dispatch', () => { { id: 'tc1', name: 'skill_manage', arguments: '{"action":"create"}' } ] - const result = await executeTools( + const result = await settleToolBatch( state, conversation, 0, @@ -1135,7 +1386,7 @@ describe('dispatch', () => { })) ) - const result = await executeTools( + const result = await settleToolBatch( state, [], 0, @@ -1209,7 +1460,7 @@ describe('dispatch', () => { } ] - const result = await executeTools( + const result = await settleToolBatch( state, [], 0, @@ -1268,7 +1519,7 @@ describe('dispatch', () => { }) state.completedToolCalls = [{ id: 'tc1', name: 'write_file', arguments: '{"path":"a.txt"}' }] - const result = await executeTools( + const result = await settleToolBatch( state, [], 0, @@ -1324,7 +1575,7 @@ describe('dispatch', () => { { id: 'tc-read', name: 'read', arguments: '{"path":"/tmp/outside.txt"}' } ] - const result = await executeTools( + const result = await settleToolBatch( state, [], 0, @@ -1400,7 +1651,7 @@ describe('dispatch', () => { { id: 'tc-write', name: 'write', arguments: '{"path":"/tmp/secret.txt"}' } ] - const result = await executeTools( + const result = await settleToolBatch( state, [], 0, @@ -1457,7 +1708,7 @@ describe('dispatch', () => { { id: 'tc-exec', name: 'exec', arguments: '{"command":"rm -rf /tmp/project"}' } ] - const result = await executeTools( + const result = await settleToolBatch( state, [], 0, @@ -1519,7 +1770,7 @@ describe('dispatch', () => { { id: 'tc-read', name: 'read', arguments: '{"path":"/tmp/outside.txt"}' } ] - const executePromise = executeTools( + const executePromise = settleToolBatch( state, [], 0, @@ -1582,7 +1833,7 @@ describe('dispatch', () => { { id: 'tc-read', name: 'read', arguments: '{"path":"/tmp/outside.txt"}' } ] - const result = await executeTools( + const result = await settleToolBatch( state, [], 0, @@ -1641,7 +1892,7 @@ describe('dispatch', () => { } ] - const result = await executeTools( + const result = await settleToolBatch( state, [], 0, @@ -1717,7 +1968,7 @@ describe('dispatch', () => { } ] - const result = await executeTools( + const result = await settleToolBatch( state, [], 0, @@ -1771,7 +2022,7 @@ describe('dispatch', () => { } ] - const result = await executeTools( + const result = await settleToolBatch( state, [], 0, @@ -1834,7 +2085,7 @@ describe('dispatch', () => { } ] - const result = await executeTools( + const result = await settleToolBatch( state, [], 0, @@ -1889,7 +2140,7 @@ describe('dispatch', () => { }) state.completedToolCalls = [{ id: 'tc1', name: 'get_weather', arguments: '{}' }] - await executeTools( + await settleToolBatch( state, [], 0, @@ -1945,7 +2196,7 @@ describe('dispatch', () => { { id: 'tc1', name: 'skill_view', arguments: '{"name":"deepchat-settings"}' } ] - const result = await executeTools( + const result = await settleToolBatch( state, [], 0, @@ -1982,7 +2233,7 @@ describe('dispatch', () => { }) state.completedToolCalls = [{ id: 'tc1', name: 'search', arguments: '{}' }] - await executeTools( + await settleToolBatch( state, conversation, 0, @@ -2021,7 +2272,7 @@ describe('dispatch', () => { }) state.completedToolCalls = [{ id: 'tc1', name: 'search', arguments: '{}' }] - await executeTools( + await settleToolBatch( state, conversation, 0, @@ -2062,7 +2313,7 @@ describe('dispatch', () => { }) state.completedToolCalls = [{ id: 'tc1', name: 'search', arguments: '{}' }] - await executeTools( + await settleToolBatch( state, conversation, 0, @@ -2126,7 +2377,7 @@ describe('dispatch', () => { } ] - await executeTools( + await settleToolBatch( state, conversation, 0, @@ -2175,7 +2426,7 @@ describe('dispatch', () => { }) state.completedToolCalls = [{ id: 'tc1', name: 'search', arguments: '{}' }] - await executeTools( + await settleToolBatch( state, conversation, 0, @@ -2216,7 +2467,7 @@ describe('dispatch', () => { }) state.completedToolCalls = [{ id: 'tc1', name: 'search', arguments: '{}' }] - await executeTools( + await settleToolBatch( state, conversation, 0, @@ -2265,7 +2516,7 @@ describe('dispatch', () => { }) state.completedToolCalls = [{ id: 'tc1', name: 'bad_tool', arguments: '{}' }] - await executeTools( + await settleToolBatch( state, conversation, 0, @@ -2309,7 +2560,7 @@ describe('dispatch', () => { }) state.completedToolCalls = [{ id: 'tc1', name: 'bad_tool', arguments: '{}' }] - await executeTools( + await settleToolBatch( state, conversation, 0, @@ -2383,7 +2634,7 @@ describe('dispatch', () => { { id: 'tc1', name: 'cdp_send', arguments: '{"method":"Page.captureScreenshot"}' } ] - await executeTools( + await settleToolBatch( state, conversation, 0, @@ -2437,7 +2688,7 @@ describe('dispatch', () => { ] const conversation: any[] = [] - const executing = executeTools( + const executing = settleToolBatch( state, conversation, 0, @@ -2503,7 +2754,7 @@ describe('dispatch', () => { const conversation: any[] = [] await expect( - executeTools( + settleToolBatch( state, conversation, 0, @@ -2558,7 +2809,7 @@ describe('dispatch', () => { state.completedToolCalls = [{ id: 'tc1', name: 'tool_a', arguments: '{}' }] await expect( - executeTools( + settleToolBatch( state, conversation, 0, @@ -2598,7 +2849,7 @@ describe('dispatch', () => { }) state.completedToolCalls = [{ id: 'tc1', name: 'tool_a', arguments: '{}' }] - await executeTools( + await settleToolBatch( state, [], 0, @@ -2625,7 +2876,7 @@ describe('dispatch', () => { expect(io.messageStore.updateAssistantContent).toHaveBeenCalled() }) - it('promotes image previews from structured tool output into assistant image blocks', async () => { + it('promotes image previews after the current-round block when call ids repeat', async () => { const tools = [makeTool('tool_image')] const toolService = { getAllToolDefinitions: vi.fn().mockResolvedValue([]), @@ -2654,6 +2905,19 @@ describe('dispatch', () => { buildToolSystemPrompt: vi.fn().mockReturnValue('') } as unknown as ToolServicePort + state.blocks.push({ + type: 'tool_call', + content: '', + status: 'success', + timestamp: Date.now(), + tool_call: { + id: 'tc1', + name: 'tool_image', + params: '{"previous":true}', + response: 'previous result' + } + }) + const prevBlockCount = state.blocks.length state.blocks.push({ type: 'tool_call', content: '', @@ -2663,10 +2927,10 @@ describe('dispatch', () => { }) state.completedToolCalls = [{ id: 'tc1', name: 'tool_image', arguments: '{}' }] - await executeTools( + await settleToolBatch( state, [], - 0, + prevBlockCount, tools, toolService, 'gpt-4', @@ -2677,15 +2941,23 @@ describe('dispatch', () => { 1024 ) - expect(state.blocks[0].tool_call?.imagePreviews).toEqual([ + expect(state.blocks[0]).toMatchObject({ + status: 'success', + tool_call: { + params: '{"previous":true}', + response: 'previous result' + } + }) + expect(state.blocks[0].tool_call?.imagePreviews).toBeUndefined() + expect(state.blocks[1].tool_call?.imagePreviews).toEqual([ { id: 'metadata-only', mimeType: 'image/png', source: 'mcp_image' } ]) - expect(state.blocks).toHaveLength(2) - expect(state.blocks[1]).toEqual( + expect(state.blocks).toHaveLength(3) + expect(state.blocks[2]).toEqual( expect.objectContaining({ type: 'image', status: 'success', @@ -2737,7 +3009,7 @@ describe('dispatch', () => { }) state.completedToolCalls = [{ id: 'tc1', name: IMAGE_GENERATE_TOOL_NAME, arguments: '{}' }] - await executeTools( + await settleToolBatch( state, [], 0, @@ -2811,7 +3083,7 @@ describe('dispatch', () => { }) state.completedToolCalls = [{ id: 'tc1', name: IMAGE_GENERATE_TOOL_NAME, arguments: '{}' }] - await executeTools( + await settleToolBatch( state, [], 0, @@ -2884,7 +3156,7 @@ describe('dispatch', () => { }) state.completedToolCalls = [{ id: 'tc1', name: IMAGE_GENERATE_TOOL_NAME, arguments: '{}' }] - await executeTools( + await settleToolBatch( state, [], 0, @@ -2939,7 +3211,7 @@ describe('dispatch', () => { } ] - const executed = await executeTools( + const executed = await settleToolBatch( state, conversation, 0, @@ -2993,7 +3265,7 @@ describe('dispatch', () => { } ] - const executed = await executeTools( + const executed = await settleToolBatch( state, conversation, 0, @@ -3056,7 +3328,7 @@ describe('dispatch', () => { } ] - await executeTools( + await settleToolBatch( state, conversation, 0, @@ -3117,7 +3389,7 @@ describe('dispatch', () => { { id: 'tc2', name: 'read', arguments: '{"path":"b.txt"}' } ] - const executed = await executeTools( + const executed = await settleToolBatch( state, conversation, 0, @@ -3211,7 +3483,7 @@ describe('dispatch', () => { ) const contextLength = estimateMessagesTokens(fittingPrefixMessages) + toolDefinitionTokens + 1 - const executed = await executeTools( + const executed = await settleToolBatch( state, conversation, 0, @@ -3274,7 +3546,7 @@ describe('dispatch', () => { { id: 'tc2', name: 'exec', arguments: '{"command":"ls"}' } ] - const executed = await executeTools( + const executed = await settleToolBatch( state, conversation, 0, @@ -3356,7 +3628,7 @@ describe('dispatch', () => { { id: 'tc2', name: 'search_docs', arguments: '{"q":"x"}' } ] - const executed = await executeTools( + const executed = await settleToolBatch( state, conversation, 0, @@ -3405,7 +3677,7 @@ describe('dispatch', () => { } ] - await executeTools( + await settleToolBatch( state, conversation, 0, @@ -3460,7 +3732,7 @@ describe('dispatch', () => { } ] - const executed = await executeTools( + const executed = await settleToolBatch( state, conversation, 0, diff --git a/test/main/agent/deepchat/runtime/process.test.ts b/test/main/agent/deepchat/runtime/process.test.ts index 75de6fe02b..4e1cf3eec0 100644 --- a/test/main/agent/deepchat/runtime/process.test.ts +++ b/test/main/agent/deepchat/runtime/process.test.ts @@ -35,6 +35,7 @@ import { processStream, resolveProviderTerminalDecision } from '@/agent/deepchat/runtime/process' +import { TRUNCATED_TOOL_CALL_ERROR } from '@/agent/deepchat/runtime/dispatch' function expectDeepchatEvent(eventName: string, payload: Record): void { expect(publishDeepchatEventMock).toHaveBeenCalledWith(eventName, expect.objectContaining(payload)) @@ -314,6 +315,28 @@ describe('processStream', () => { }) as unknown as ProcessParams['coreStream'] } + function createScriptedCoreStream( + rounds: readonly (readonly LLMCoreStreamEvent[])[], + providerInputs: ChatMessage[][] = [], + onRoundComplete?: (roundIndex: number) => void + ): ProcessParams['coreStream'] { + let roundIndex = 0 + return vi.fn((messages: ChatMessage[]) => { + const currentRoundIndex = roundIndex++ + const events = rounds[currentRoundIndex] + if (!events) { + throw new Error(`Missing scripted provider round ${currentRoundIndex + 1}`) + } + providerInputs.push(structuredClone(messages)) + return (async function* () { + for (const event of events) { + yield event + } + onRoundComplete?.(currentRoundIndex) + })() + }) as unknown as ProcessParams['coreStream'] + } + describe('fixed lifecycle commits', () => { const TOOL_ROUND_COMMIT_ORDER = [ 'renderer:update', @@ -957,6 +980,476 @@ describe('processStream', () => { }) }) + describe('truncated tool call recovery', () => { + it('rejects the local batch in source order and recovers on the next provider round', async () => { + const providerInputs: ChatMessage[][] = [] + const notifications: DeepChatLoopNotification[] = [] + const coreStream = createScriptedCoreStream( + [ + [ + { + type: 'usage', + usage: { prompt_tokens: 2, completion_tokens: 3, total_tokens: 5 } + }, + { + type: 'tool_call_start', + tool_call_id: 'local-complete', + tool_call_name: 'read', + provider_options: { openai: { itemId: 'item-local' } } + }, + { + type: 'tool_call_end', + tool_call_id: 'local-complete', + tool_call_arguments_complete: '{"path":"README.md"}' + }, + { + type: 'tool_call_start', + tool_call_id: 'provider-pending', + tool_call_name: 'web_search', + tool_call_execution_owner: 'provider' + }, + { + type: 'tool_call_chunk', + tool_call_id: 'provider-pending', + tool_call_arguments_chunk: '{"query":"deepchat"}' + }, + { + type: 'tool_call_start', + tool_call_id: 'local-pending', + tool_call_name: 'exec', + provider_options: { openai: { itemId: 'item-pending' } } + }, + { + type: 'tool_call_chunk', + tool_call_id: 'local-pending', + tool_call_arguments_chunk: '{"command":"pnpm test' + }, + { type: 'stop', stop_reason: 'max_tokens' } + ], + [ + { + type: 'usage', + usage: { prompt_tokens: 4, completion_tokens: 1, total_tokens: 5 } + }, + { type: 'text', content: 'Recovered safely' }, + { type: 'stop', stop_reason: 'complete' } + ] + ], + providerInputs + ) + const toolService = createMockToolService() + ;(toolService.preCheckToolPermission as ReturnType).mockResolvedValue({ + needsPermission: true, + permissionType: 'write', + description: 'Should never be requested' + }) + const controls = { + autoGrantPermission: vi.fn(), + reviewToolPermission: vi.fn(), + activateSkill: vi.fn().mockResolvedValue([]) + } + const params = createParams({ + coreStream, + toolExecution: createToolExecutionPort(toolService), + tools: [makeTool('read'), makeTool('exec')], + permissionMode: 'auto_approve', + controls, + notificationObserver: { + notify: (notification) => notifications.push(structuredClone(notification)) + } + }) + + const result = await processStream(params) + + expect(result).toMatchObject({ + status: 'completed', + stopReason: 'complete', + usage: { inputTokens: 6, outputTokens: 4, totalTokens: 10 } + }) + expect(coreStream).toHaveBeenCalledTimes(2) + expect(providerInputs).toHaveLength(2) + const recoveryAssistant = providerInputs[1].find( + (message) => message.role === 'assistant' && message.tool_calls?.length + ) + expect(recoveryAssistant?.tool_calls).toEqual([ + { + id: 'local-complete', + type: 'function', + function: { name: 'read', arguments: '{"path":"README.md"}' }, + provider_options: { openai: { itemId: 'item-local' } } + }, + { + id: 'local-pending', + type: 'function', + function: { name: 'exec', arguments: '{"command":"pnpm test' }, + provider_options: { openai: { itemId: 'item-pending' } } + } + ]) + expect( + providerInputs[1] + .filter((message) => message.role === 'tool') + .map((message) => [message.tool_call_id, message.content]) + ).toEqual([ + ['local-complete', TRUNCATED_TOOL_CALL_ERROR], + ['local-pending', TRUNCATED_TOOL_CALL_ERROR] + ]) + const toolBlocks = params.run.streamState.blocks.filter((block) => block.type === 'tool_call') + expect(toolBlocks).toEqual([ + expect.objectContaining({ + status: 'error', + extra: expect.objectContaining({ toolCallSkippedReason: 'max_tokens' }), + tool_call: expect.objectContaining({ + id: 'local-complete', + response: TRUNCATED_TOOL_CALL_ERROR + }) + }), + expect.objectContaining({ + status: 'error', + extra: expect.objectContaining({ toolCallIncompleteReason: 'max_tokens' }), + tool_call: expect.objectContaining({ id: 'provider-pending', response: '' }) + }), + expect.objectContaining({ + status: 'error', + extra: expect.objectContaining({ toolCallSkippedReason: 'max_tokens' }), + tool_call: expect.objectContaining({ + id: 'local-pending', + response: TRUNCATED_TOOL_CALL_ERROR + }) + }) + ]) + expect(toolService.preCheckToolPermission).not.toHaveBeenCalled() + expect(toolService.callTool).not.toHaveBeenCalled() + expect(controls.autoGrantPermission).not.toHaveBeenCalled() + expect(controls.reviewToolPermission).not.toHaveBeenCalled() + expect(controls.activateSkill).not.toHaveBeenCalled() + expect(notifications.map((notification) => notification.event)).toEqual([ + 'PostToolUseFailure', + 'PostToolUseFailure' + ]) + expect( + notifications.map((notification) => + notification.event === 'PostToolUseFailure' ? notification.tool.callId : null + ) + ).toEqual(['local-complete', 'local-pending']) + expect(params.run.streamState.metadata).toMatchObject({ + providerRounds: 2, + toolCalls: 0, + inputTokens: 6, + outputTokens: 4, + totalTokens: 10 + }) + expect(params.run.streamState.metadata.noProgressToolLoop).toBeUndefined() + const tapeFacts = tapeToolFactWriter.appendToolFact.mock.calls.map(([input]) => ({ + source: input.provenance.source, + sourceId: input.provenance.sourceId + })) + for (const callId of ['local-complete', 'local-pending']) { + expect(tapeFacts).toEqual( + expect.arrayContaining([ + { source: 'tool_call', sourceId: `m1:${callId}` }, + { source: 'tool_result', sourceId: `m1:${callId}` } + ]) + ) + } + expect(tapeFacts).not.toContainEqual({ + source: 'tool_result', + sourceId: 'm1:provider-pending' + }) + }) + + it('settles a second truncated batch and stops without a third provider request', async () => { + const providerInputs: ChatMessage[][] = [] + const coreStream = createScriptedCoreStream( + [ + [ + { + type: 'tool_call_start', + tool_call_id: 'tc-first', + tool_call_name: 'action' + }, + { + type: 'tool_call_chunk', + tool_call_id: 'tc-first', + tool_call_arguments_chunk: '{"round":1' + }, + { type: 'stop', stop_reason: 'max_tokens' } + ], + [ + { + type: 'tool_call_start', + tool_call_id: 'tc-first', + tool_call_name: 'action' + }, + { + type: 'tool_call_chunk', + tool_call_id: 'tc-first', + tool_call_arguments_chunk: '{"round":2' + }, + { type: 'stop', stop_reason: 'max_tokens' } + ] + ], + providerInputs + ) + const toolService = createMockToolService() + const params = createParams({ + coreStream, + toolExecution: createToolExecutionPort(toolService), + tools: [makeTool('action')] + }) + + const result = await processStream(params) + + expect(result).toMatchObject({ status: 'completed', stopReason: 'max_tokens' }) + expect(coreStream).toHaveBeenCalledTimes(2) + expect(providerInputs).toHaveLength(2) + expect( + providerInputs[1].find((message) => message.role === 'tool' && message.tool_call_id === 'tc-first') + ).toMatchObject({ content: TRUNCATED_TOOL_CALL_ERROR }) + expect(params.run.messages.filter((message) => message.role === 'tool')).toEqual([ + { role: 'tool', tool_call_id: 'tc-first', content: TRUNCATED_TOOL_CALL_ERROR }, + { role: 'tool', tool_call_id: 'tc-first', content: TRUNCATED_TOOL_CALL_ERROR } + ]) + expect( + params.run.streamState.blocks.filter( + (block) => block.type === 'tool_call' && block.tool_call?.id === 'tc-first' + ) + ).toEqual([ + expect.objectContaining({ + status: 'error', + extra: expect.objectContaining({ toolCallSkippedReason: 'max_tokens' }), + tool_call: expect.objectContaining({ params: '{"round":1' }) + }), + expect.objectContaining({ + status: 'error', + extra: expect.objectContaining({ toolCallSkippedReason: 'max_tokens' }), + tool_call: expect.objectContaining({ params: '{"round":2' }) + }) + ]) + expect(toolService.callTool).not.toHaveBeenCalled() + expect(params.run.streamState.metadata).toMatchObject({ providerRounds: 2, toolCalls: 0 }) + expect(params.run.streamState.metadata.noProgressToolLoop).toBeUndefined() + }) + + it('keeps plain-text max_tokens terminal without starting recovery', async () => { + const coreStream = createScriptedCoreStream([ + [ + { type: 'text', content: 'Partial answer' }, + { type: 'stop', stop_reason: 'max_tokens' } + ] + ]) + const params = createParams({ coreStream }) + + const result = await processStream(params) + + expect(result).toMatchObject({ status: 'completed', stopReason: 'max_tokens' }) + expect(coreStream).toHaveBeenCalledTimes(1) + expect(params.run.messages).toEqual([{ role: 'user', content: 'Hello' }]) + expect(params.run.streamState.metadata).toMatchObject({ providerRounds: 1, toolCalls: 0 }) + }) + + it('does not recover or synthesize results for a provider-owned truncated call', async () => { + const coreStream = createScriptedCoreStream([ + [ + { + type: 'tool_call_start', + tool_call_id: 'provider-only', + tool_call_name: 'web_search', + tool_call_execution_owner: 'provider' + }, + { + type: 'tool_call_chunk', + tool_call_id: 'provider-only', + tool_call_arguments_chunk: '{"query":"deepchat"}' + }, + { type: 'stop', stop_reason: 'max_tokens' } + ] + ]) + const toolService = createMockToolService() + const params = createParams({ + coreStream, + toolExecution: createToolExecutionPort(toolService), + tools: [makeTool('web_search')] + }) + + const result = await processStream(params) + + expect(result).toMatchObject({ status: 'completed', stopReason: 'max_tokens' }) + expect(coreStream).toHaveBeenCalledTimes(1) + expect(toolService.callTool).not.toHaveBeenCalled() + expect(params.run.messages).toEqual([{ role: 'user', content: 'Hello' }]) + expect(params.run.streamState.blocks[0]).toMatchObject({ + status: 'error', + extra: { toolCallIncompleteReason: 'max_tokens' }, + tool_call: { id: 'provider-only', response: '' } + }) + expect(params.run.streamState.pendingToolCalls.size).toBe(0) + }) + + it('does not synthesize results for structurally unclosable calls', async () => { + const coreStream = createScriptedCoreStream([ + [ + { type: 'tool_call_start', tool_call_id: ' ', tool_call_name: 'action' }, + { type: 'tool_call_start', tool_call_id: 'duplicate', tool_call_name: 'action' }, + { type: 'tool_call_start', tool_call_id: 'duplicate', tool_call_name: 'action' }, + { type: 'stop', stop_reason: 'max_tokens' } + ] + ]) + const params = createParams({ coreStream, tools: [makeTool('action')] }) + + const result = await processStream(params) + + expect(result).toMatchObject({ status: 'completed', stopReason: 'max_tokens' }) + expect(coreStream).toHaveBeenCalledTimes(1) + expect(params.run.messages).toEqual([{ role: 'user', content: 'Hello' }]) + expect(params.run.streamState.pendingToolCalls.size).toBe(0) + expect(params.run.streamState.blocks).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + status: 'error', + extra: { toolCallIncompleteReason: 'max_tokens' } + }) + ]) + ) + expect( + params.run.streamState.blocks.filter( + (block) => block.type === 'tool_call' && block.extra?.toolCallIncompleteReason === 'max_tokens' + ) + ).toHaveLength(3) + }) + + it('settles the rejection before yielding to pending input', async () => { + const coreStream = createScriptedCoreStream([ + [ + { type: 'tool_call_start', tool_call_id: 'tc1', tool_call_name: 'action' }, + { + type: 'tool_call_end', + tool_call_id: 'tc1', + tool_call_arguments_complete: '{}' + }, + { type: 'stop', stop_reason: 'max_tokens' } + ], + [{ type: 'stop', stop_reason: 'complete' }] + ]) + const shouldYieldForPendingInput = vi.fn(() => true) + const toolService = createMockToolService() + const params = createParams({ + coreStream, + toolExecution: createToolExecutionPort(toolService), + tools: [makeTool('action')], + shouldYieldForPendingInput + }) + + const result = await processStream(params) + + expect(result).toMatchObject({ status: 'completed', stopReason: 'pending_input' }) + expect(coreStream).toHaveBeenCalledTimes(1) + expect(shouldYieldForPendingInput).toHaveBeenCalledTimes(1) + expect(params.run.messages.at(-1)).toEqual({ + role: 'tool', + tool_call_id: 'tc1', + content: TRUNCATED_TOOL_CALL_ERROR + }) + expect(tapeToolFactWriter.appendToolFact).toHaveBeenCalledTimes(2) + }) + + it('settles the rejection before honoring a post-stream abort', async () => { + const abortController = new AbortController() + const coreStream = createScriptedCoreStream( + [ + [ + { type: 'tool_call_start', tool_call_id: 'tc1', tool_call_name: 'action' }, + { + type: 'tool_call_chunk', + tool_call_id: 'tc1', + tool_call_arguments_chunk: '{"value":' + }, + { type: 'stop', stop_reason: 'max_tokens' } + ] + ], + [], + () => abortController.abort() + ) + const toolService = createMockToolService() + const params = createParams({ + coreStream, + toolExecution: createToolExecutionPort(toolService), + tools: [makeTool('action')], + abortController + }) + + const result = await processStream(params) + + expect(result).toMatchObject({ status: 'aborted', stopReason: 'user_stop' }) + expect(toolService.callTool).not.toHaveBeenCalled() + expect(params.run.messages.at(-1)).toEqual({ + role: 'tool', + tool_call_id: 'tc1', + content: TRUNCATED_TOOL_CALL_ERROR + }) + expect(tapeToolFactWriter.appendToolFact).toHaveBeenCalledTimes(2) + expect(messageStore.setMessageError).toHaveBeenCalled() + }) + + it('settles the rejection before a provider-round cap prevents recovery', async () => { + const coreStream = createScriptedCoreStream([ + [ + { type: 'tool_call_start', tool_call_id: 'tc1', tool_call_name: 'action' }, + { + type: 'tool_call_end', + tool_call_id: 'tc1', + tool_call_arguments_complete: '{}' + }, + { type: 'stop', stop_reason: 'max_tokens' } + ] + ]) + const params = createParams({ + coreStream, + tools: [makeTool('action')], + maxProviderRounds: 1 + }) + + const result = await processStream(params) + + expect(result).toMatchObject({ status: 'error', stopReason: 'max_turns' }) + expect(coreStream).toHaveBeenCalledTimes(1) + expect(params.run.messages.at(-1)).toEqual({ + role: 'tool', + tool_call_id: 'tc1', + content: TRUNCATED_TOOL_CALL_ERROR + }) + expect(tapeToolFactWriter.appendToolFact).toHaveBeenCalledTimes(2) + }) + + it('stops on a terminal fitting failure without requesting recovery', async () => { + const coreStream = createScriptedCoreStream([ + [ + { type: 'tool_call_start', tool_call_id: 'tc1', tool_call_name: 'read' }, + { + type: 'tool_call_chunk', + tool_call_id: 'tc1', + tool_call_arguments_chunk: '{"path":"' + }, + { type: 'stop', stop_reason: 'max_tokens' } + ] + ]) + const toolService = createMockToolService() + const params = createParams({ + coreStream, + toolExecution: createToolExecutionPort(toolService), + tools: [makeTool('read')], + modelConfig: { contextLength: 1 } as any, + maxTokens: 1 + }) + + const result = await processStream(params) + + expect(result).toMatchObject({ status: 'error', stopReason: 'tool_error' }) + expect(result.terminalError).toContain('remaining context window is too small') + expect(coreStream).toHaveBeenCalledTimes(1) + expect(toolService.callTool).not.toHaveBeenCalled() + }) + }) + it('flushes ACP provider permission blocks immediately and keeps live permission updates mutable', async () => { let releaseStream: (() => void) | null = null let commitDecision: ((granted: boolean) => void) | null = null @@ -1910,7 +2403,7 @@ describe('processStream', () => { await expect(fs.readFile(offloadPath!, 'utf-8')).resolves.toBe(longScreenshot) }) - it('multiple tool calls in one turn', async () => { + it('preserves completed-call order for multiple tools in one turn', async () => { let callCount = 0 const toolService = createMockToolService({ get_weather: 'Sunny', @@ -1926,11 +2419,6 @@ describe('processStream', () => { tool_call_id: 'tc1', tool_call_name: 'get_weather' } as LLMCoreStreamEvent - yield { - type: 'tool_call_end', - tool_call_id: 'tc1', - tool_call_arguments_complete: '{}' - } as LLMCoreStreamEvent yield { type: 'tool_call_start', tool_call_id: 'tc2', @@ -1941,6 +2429,11 @@ describe('processStream', () => { tool_call_id: 'tc2', tool_call_arguments_complete: '{}' } as LLMCoreStreamEvent + yield { + type: 'tool_call_end', + tool_call_id: 'tc1', + tool_call_arguments_complete: '{}' + } as LLMCoreStreamEvent yield { type: 'stop', stop_reason: 'tool_use' } as LLMCoreStreamEvent })() } else { @@ -1962,6 +2455,11 @@ describe('processStream', () => { await promise expect(toolService.callTool).toHaveBeenCalledTimes(2) + expect( + (toolService.callTool as ReturnType).mock.calls.map( + ([request]) => request.function.name + ) + ).toEqual(['get_time', 'get_weather']) expect(coreStream).toHaveBeenCalledTimes(2) }) diff --git a/test/main/evals/nativeAgent/nativeAgentBehavior.eval.test.ts b/test/main/evals/nativeAgent/nativeAgentBehavior.eval.test.ts index 50b4aedf07..20320c17b2 100644 --- a/test/main/evals/nativeAgent/nativeAgentBehavior.eval.test.ts +++ b/test/main/evals/nativeAgent/nativeAgentBehavior.eval.test.ts @@ -9,6 +9,7 @@ import { NATIVE_AGENT_EVAL_SCENARIOS } from './scenarios' const EXPECTED_SCENARIO_IDS = [ 'direct-completion', 'max-tokens', + 'truncated-tool-call-recovery', 'single-tool-round', 'multiple-tool-rounds', 'tool-failure-recovery', @@ -98,7 +99,7 @@ describe('native Agent deterministic behavior eval', () => { passed: EXPECTED_SCENARIO_IDS.length, passRate: 1, totalProviderRounds: expectedProviderRounds, - providerRoundBudget: 150, + providerRoundBudget: 152, totalToolCalls: expectedToolCalls, toolCallBudget: 139, totalTokens: expectedTotalTokens, diff --git a/test/main/evals/nativeAgent/scenarios.ts b/test/main/evals/nativeAgent/scenarios.ts index 5aa50d6287..dd3189f43b 100644 --- a/test/main/evals/nativeAgent/scenarios.ts +++ b/test/main/evals/nativeAgent/scenarios.ts @@ -8,6 +8,11 @@ import { completionRound, toolRound } from './harness' const DIRECT_USAGE = { inputTokens: 10, outputTokens: 4, totalTokens: 14 } const MAX_TOKENS_USAGE = { inputTokens: 11, outputTokens: 4, totalTokens: 15 } +const TRUNCATED_TOOL_RECOVERY_ROUND_USAGES = [ + { inputTokens: 3, outputTokens: 2, totalTokens: 5 }, + { inputTokens: 4, outputTokens: 1, totalTokens: 5 } +] satisfies ScriptedProviderUsage[] +const TRUNCATED_TOOL_RECOVERY_USAGE = { inputTokens: 7, outputTokens: 3, totalTokens: 10 } const SINGLE_TOOL_USAGE = { inputTokens: 18, outputTokens: 6, totalTokens: 24 } const MULTI_TOOL_ROUND_USAGES = [ { @@ -126,6 +131,60 @@ export const NATIVE_AGENT_EVAL_SCENARIOS: NativeAgentEvalScenario[] = [ usage: expectedUsage(MAX_TOKENS_USAGE) } }, + { + id: 'truncated-tool-call-recovery', + rounds: [ + { + events: [ + { + type: 'usage', + usage: { + prompt_tokens: TRUNCATED_TOOL_RECOVERY_ROUND_USAGES[0].inputTokens, + completion_tokens: TRUNCATED_TOOL_RECOVERY_ROUND_USAGES[0].outputTokens, + total_tokens: TRUNCATED_TOOL_RECOVERY_ROUND_USAGES[0].totalTokens + } + }, + { + type: 'tool_call_start', + tool_call_id: 'call-truncated-action', + tool_call_name: 'action' + }, + { + type: 'tool_call_chunk', + tool_call_id: 'call-truncated-action', + tool_call_arguments_chunk: '{"path":"result.txt"' + }, + { type: 'stop', stop_reason: 'max_tokens' } + ] + }, + completionRound('Recovered truncated tool call', TRUNCATED_TOOL_RECOVERY_ROUND_USAGES[1]) + ], + tools: { + action: { + response: 'must not execute', + permission: { + permissionType: 'write', + description: 'Must not request permission for a truncated call' + } + } + }, + permissionMode: 'ask_user', + budget: { maxProviderRounds: 2, maxToolCalls: 0 }, + expected: { + status: 'completed', + stopReason: 'complete', + persistedStatus: 'sent', + persistedRunOutcome: 'completed', + persistedRunStopReason: 'complete', + providerRounds: 2, + toolCalls: 0, + finalTextIncludes: 'Recovered truncated tool call', + toolMessageIncludes: ['model response reached the output token limit'], + failedToolCalls: 0, + permissionRequests: 0, + usage: expectedUsage(TRUNCATED_TOOL_RECOVERY_USAGE) + } + }, { id: 'single-tool-round', rounds: [ diff --git a/test/main/provider/aiSdkStreamAdapter.test.ts b/test/main/provider/aiSdkStreamAdapter.test.ts index e4f76b80db..f3960c2452 100644 --- a/test/main/provider/aiSdkStreamAdapter.test.ts +++ b/test/main/provider/aiSdkStreamAdapter.test.ts @@ -105,6 +105,48 @@ describe('AI SDK stream adapter', () => { ]) }) + it('marks streaming and atomic provider-executed calls as provider-owned', async () => { + const events = await collectEvents( + [ + { + type: 'tool-input-start', + id: 'streamed-provider-call', + toolName: 'web_search', + providerExecuted: true + }, + { + type: 'tool-input-delta', + id: 'streamed-provider-call', + delta: '{"query":"deepchat"}' + }, + { type: 'tool-input-end', id: 'streamed-provider-call' }, + { + type: 'tool-call', + toolCallId: 'atomic-provider-call', + toolName: 'code_execution', + input: { code: '1 + 1' }, + providerExecuted: true + } + ], + { supportsNativeTools: true } + ) + + expect(events.filter((event) => event.type === 'tool_call_start')).toEqual([ + { + type: 'tool_call_start', + tool_call_id: 'streamed-provider-call', + tool_call_name: 'web_search', + tool_call_execution_owner: 'provider' + }, + { + type: 'tool_call_start', + tool_call_id: 'atomic-provider-call', + tool_call_name: 'code_execution', + tool_call_execution_owner: 'provider' + } + ]) + }) + it('preserves explicit zero cache usage reported by the provider', async () => { const events = await collectEvents( [