Skip to content

fix(local_browser): drop final_script.py workflow and bound ARIA history#16

Merged
adamlu123 merged 1 commit into
microsoft:mainfrom
adamlu123:fix/local-browser-workflow
May 28, 2026
Merged

fix(local_browser): drop final_script.py workflow and bound ARIA history#16
adamlu123 merged 1 commit into
microsoft:mainfrom
adamlu123:fix/local-browser-workflow

Conversation

@adamlu123

Copy link
Copy Markdown
Contributor

What

local_browser.yaml's prompts were copy-pasted from base.yaml's bash_command-driven workspace flow, which doesn't fit a live persistent Playwright session. Two concrete bugs:

  1. The agent was told to launch its own browser inside final_script.py while also being told never to import or launch Playwright.
  2. It was told to shell out via run_command(...) — a helper that doesn't exist anywhere in the codebase. Any agent step calling it would die with NameError.

This PR reframes the live-browser mode around its actual capabilities and adds a context-bounding knob so ARIA snapshots don't dominate token usage as the loop runs.

Changes

local_browser.yaml (−221 / +127)

  • Drop final_script.py, plan.md, self_reflect_config.json, final_runs/, image_qa, and self_reflection from the prompts. There is no workspace directory in this mode; the agent drives the live page and reports its answer in final_response when done.
  • Override agent.require_self_reflection_success: false so the completion gate (which checks for final_runs/run_<id>/self_reflect_result.json) does not block done: true.
  • Override agent.summary_user_prompt to talk about live browser state (URL, controls, selectors) instead of workspace artifacts (plan.md, final_script.py, final_runs/) that don't exist in this mode.
  • Document in the yaml header what each step's observation contains (URL / title / stdout / ARIA snapshot / screenshot path) and note that the screenshot is NOT visually attached by default — flipping model.attach_observation_screenshot: true enables multimodal image input.
  • Opt into ARIA-history pruning with agent.keep_last_n_observations: 1.

AgentConfig.keep_last_n_observations (+33 / -1)

  • New knob in agents/default.py. Default -1 (disabled) — opt in per config.
  • When > 0: strips the ARIA snapshot payload from observation messages older than the most recent N, both in the rendered text content and in the extra.observation dict. URL / title / printed stdout are preserved so the agent still knows what it did.
  • Implementation: walks self.messages after each add_messages call, finds observation messages by extra.observation, replaces the saved aria_snapshot string in the text content with (ARIA snapshot pruned; see most recent observation).

Validation

Live end-to-end run against task: "Search for flights from SEA to JFK on 2026-08-15 to 2026-08-20" via base.yaml + local_browser.yaml + model_openai.yaml.

Metric Without pruning (N disabled) N=1 (this PR's default for local_browser)
Steps 9 7
Model calls 10 8
Exit Submitted Submitted
Final URL google.com/travel/flights/search?tfs=… (same shape)
Final response "Google Flights results … SEA → JFK, round trip, Aug 15, 2026 to Aug 20, 2026." "Flights searched successfully on Google Flights for SEA to JFK … Results page shows 11 results."
Trajectory size on disk 7.5 MB 2.2 MB

Trajectory inspection of the N=1 run confirms: observation #1–6 have ARIA stripped from both the text content and extra.observation.aria_snapshot; only observation #7 retains its full 15,282-char ARIA snapshot.

Test plan

  • Existing unit tests pass (7/7).
  • Live e2e against Google Flights with model_openai.yaml succeeds and reaches the results page in 7 steps with N=1.
  • Trajectory inspection confirms ARIA pruning fires as designed.
  • Suggested follow-up: apply the same prompt cleanup to persistent_browser.yaml, persistent_browser_always.yaml, crafted_cli.yaml, and task_showcase.yaml (same run_command / final_script.py issues are copy-pasted there).

local_browser.yaml's prompts were copy-pasted from base.yaml's
bash_command-driven workspace flow, which doesn't fit a live persistent
Playwright session. The agent was told to launch its own browser inside
final_script.py while also being told never to import or launch
Playwright; and it was told to shell out via `run_command(...)`, a
helper that does not exist anywhere in the codebase.

Reframe the live-browser mode around its actual capabilities:

- Drop final_script.py, plan.md, self_reflect_config.json, final_runs/,
  image_qa, and self_reflection from the prompts. There is no workspace
  directory in this mode; the agent drives the live `page` and reports
  its answer in `final_response` when done.
- Override `agent.require_self_reflection_success: false` so the
  completion gate (which checks for `final_runs/run_<id>/self_reflect_result.json`)
  does not block `done: true` in this mode.
- Override `agent.summary_user_prompt` to talk about live browser
  state (URL, controls, selectors) instead of workspace artifacts
  (`plan.md`, `final_script.py`, `final_runs/`) that don't exist.
- Document in the yaml header what each step's observation contains
  (URL / title / stdout / ARIA snapshot / screenshot path) and note
  that the screenshot is NOT visually attached by default — flipping
  `model.attach_observation_screenshot: true` enables multimodal image
  input.

Add a context-bounding knob in `AgentConfig`:

- `keep_last_n_observations` (default -1, disabled) strips the ARIA
  snapshot payload from observation messages older than the most
  recent N, both in the rendered text content and the `extra`
  observation dict. URL / title / printed stdout are preserved so the
  agent still knows what it did. local_browser.yaml opts in with N=1.

Validated end-to-end against "Search for flights from SEA to JFK on
2026-08-15 to 2026-08-20" via base.yaml + local_browser.yaml + model_openai.yaml:
the agent reached the Google Flights results page (round trip,
SEA -> JFK, Aug 15 -> Aug 20) in 7 steps with N=1 pruning (vs 9 steps
without), and the saved trajectory shows ARIA stripped from the first
6 observations and preserved on the 7th.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@adamlu123 adamlu123 merged commit 7dc7442 into microsoft:main May 28, 2026
1 check was pending
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant