Log per-action tool calls in CuaAgent templates - #241
Open
dprevoznik wants to merge 5 commits into
Open
Conversation
The anthropic/openai/gemini computer-use templates never subscribed to CuaAgent's tool_execution_start/end events, so they only printed the final answer. Wire up agent.subscribe() with a small logger so each click/type/screenshot action streams to stdout, matching the yutori and tzafon templates.
AgentToolResult.content carries the image bytes sent back to the model, so logging the whole result dumped truncated base64 on every screenshot turn. Log result.details — the structured, human-readable summary — to match the browser-loop reference logger.
The first pass logged raw tool_execution_start/end pairs with screenshot byte counts — hard to follow. Subscribe to message_end too and print the model's narration (`agent>`) between steps, then each action as a concise line (`→ click (85, 267)`, `→ drag (780,385) → (1100,385)`), unwrapping Anthropic's computer_batch and OpenAI's computer_use_extra so every provider reads the same. Failed actions are marked so retries are visible.
Replace the per-tool formatters (point/dragPath/quote/keypress/etc.) with a generic key=value renderer, keeping only the two unwraps that materially help readability: Anthropic's computer_batch and OpenAI's computer_use_extra. Collapse narration whitespace so each agent> line stays on one line. ~99 → 45 lines, same clean output across all three providers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
anthropic-computer-use,openai-computer-use, andgemini-computer-usetemplates build onCuaAgentbut never subscribed to its events, so they only printed the final answer — no visibility into the steps the agent took.This adds a shared
logAgentEventhelper per template (logging.ts/lib/logging.ts, matching each template's file layout) wired up withagent.subscribe(logAgentEvent). It logs:agent> ...) frommessage_end→ click (85, 267),→ drag (780,385) → (1100,385),→ goto https://...) fromtool_execution_start✗ click failed) so retries are visibleAnthropic batches actions under
computer_batchand OpenAI wraps navigation incomputer_use_extra; both are unwrapped so every provider's logs read the same. Byte-count/base64 noise from the raw tool results is not logged.This brings the CuaAgent templates in line with the readable per-step logs the hand-rolled
yutori/tzafontemplates and the Pythonopenai-computer-usetemplate already produce.Test plan
Deployed and invoked all three end-to-end against live Kernel browsers (multi-step task: navigate, filter, drag cards). Sample output:
bun install && bunx tsc --noEmitpasses in all three template dirscomputer_batch), openai (computer_use_extra), and gemini (top-level tools) all verified liveNote
Low Risk
Template-only console logging with no changes to agent execution, auth, or data handling.
Overview
The anthropic, gemini, and openai TypeScript computer-use templates now subscribe to
CuaAgentevents so runs print step-by-step output instead of only the final answer.Each template adds a
logAgentEventhelper (underlogging.tsoropenai-computer-use/lib/logging.ts) and registers it withagent.subscribe(logAgentEvent)beforeprompt. Logs include assistant narration (agent> …onmessage_end), each browser action at start (→ …ontool_execution_start), and failures (✗ …whentool_execution_endis an error).describeunwrapscomputer_batchandcomputer_use_extraso Anthropic/OpenAI nested actions read like flat click/drag/goto lines, with long argument values truncated.Reviewed by Cursor Bugbot for commit a489ac7. Bugbot is set up for automated code reviews on this repo. Configure here.