Skip to content

Log per-action tool calls in CuaAgent templates - #241

Open
dprevoznik wants to merge 5 commits into
mainfrom
hypeship/cua-template-action-logging
Open

Log per-action tool calls in CuaAgent templates#241
dprevoznik wants to merge 5 commits into
mainfrom
hypeship/cua-template-action-logging

Conversation

@dprevoznik

@dprevoznik dprevoznik commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Summary

The anthropic-computer-use, openai-computer-use, and gemini-computer-use templates build on CuaAgent but never subscribed to its events, so they only printed the final answer — no visibility into the steps the agent took.

This adds a shared logAgentEvent helper per template (logging.ts / lib/logging.ts, matching each template's file layout) wired up with agent.subscribe(logAgentEvent). It logs:

  • the model's narration between steps (agent> ...) from message_end
  • each concrete browser action (→ click (85, 267), → drag (780,385) → (1100,385), → goto https://...) from tool_execution_start
  • failed actions (✗ click failed) so retries are visible

Anthropic batches actions under computer_batch and OpenAI wraps navigation in computer_use_extra; both are unwrapped so every provider's logs read the same. Byte-count/base64 noise from the raw tool results is not logged.

This brings the CuaAgent templates in line with the readable per-step logs the hand-rolled yutori/tzafon templates and the Python openai-computer-use template already produce.

Test plan

Deployed and invoked all three end-to-end against live Kernel browsers (multi-step task: navigate, filter, drag cards). Sample output:

agent> I need to filter by Alice Johnson. Let me click the Filters button.
  → click (638)
  ✗ click failed
  → click (638, 125)
agent> The filter dropdown appeared. Clicking Alice Johnson.
  → click (706, 271)
agent> Now dragging her in-progress tasks to Done.
  → drag (780, 385) → (1100, 385)
  • bun install && bunx tsc --noEmit passes in all three template dirs
  • anthropic (computer_batch), openai (computer_use_extra), and gemini (top-level tools) all verified live

Note

Low Risk
Template-only console logging with no changes to agent execution, auth, or data handling.

Overview
The anthropic, gemini, and openai TypeScript computer-use templates now subscribe to CuaAgent events so runs print step-by-step output instead of only the final answer.

Each template adds a logAgentEvent helper (under logging.ts or openai-computer-use/lib/logging.ts) and registers it with agent.subscribe(logAgentEvent) before prompt. Logs include assistant narration (agent> … on message_end), each browser action at start (→ … on tool_execution_start), and failures (✗ … when tool_execution_end is an error). describe unwraps computer_batch and computer_use_extra so Anthropic/OpenAI nested actions read like flat click/drag/goto lines, with long argument values truncated.

Reviewed by Cursor Bugbot for commit a489ac7. Bugbot is set up for automated code reviews on this repo. Configure here.

dprevoznik and others added 5 commits September 1, 2026 14:44
The anthropic/openai/gemini computer-use templates never subscribed to
CuaAgent's tool_execution_start/end events, so they only printed the
final answer. Wire up agent.subscribe() with a small logger so each
click/type/screenshot action streams to stdout, matching the yutori
and tzafon templates.
AgentToolResult.content carries the image bytes sent back to the model,
so logging the whole result dumped truncated base64 on every screenshot
turn. Log result.details — the structured, human-readable summary — to
match the browser-loop reference logger.
The first pass logged raw tool_execution_start/end pairs with screenshot
byte counts — hard to follow. Subscribe to message_end too and print the
model's narration (`agent>`) between steps, then each action as a concise
line (`→ click (85, 267)`, `→ drag (780,385) → (1100,385)`), unwrapping
Anthropic's computer_batch and OpenAI's computer_use_extra so every
provider reads the same. Failed actions are marked so retries are visible.
Replace the per-tool formatters (point/dragPath/quote/keypress/etc.) with a
generic key=value renderer, keeping only the two unwraps that materially help
readability: Anthropic's computer_batch and OpenAI's computer_use_extra.
Collapse narration whitespace so each agent> line stays on one line. ~99 → 45
lines, same clean output across all three providers.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant