QA Automation Agent: a separate QA-mindset agent that writes and runs app flow tests, with its own UI page

[serge-ivo]

Motivation

PAS Agent Teams builds apps (PO/BA/Dev/QA loop) — but flow-level test automation must NOT be written by the dev team that builds the feature ("the dev team doesn't grade its own homework"). We want a dedicated agent with a QA mindset, delivered the PAGS way: an agent as a product, with its own UI page that has nothing to do with the development console.

The developer experience: open the QA agent's page, point it at your app, and watch tests run live in the browser — no local installs, no repo changes, ever.

What the agent does

1. Explores the app like a tester — reads the app's live UI (and optionally its knowledge base), identifies the user journeys that matter (sign in, core actions, error paths), and proposes a flow list. QA mindset: edge cases, wrong inputs, empty states — not happy-path-only.
2. Writes flow specs in the platform test format (JSON steps, Playwright-shaped — see Observable in-browser test execution: platform-run app automation, specs stored off-repo, Playwright-translatable proappstore-online/platform#38) and saves them via the PAS test-flows API. Specs live in platform storage, never in the product repo.
3. Runs them and watches results: triggers in-browser runs (embedded observable runner) and headless platform runs; on failures, produces a human-readable diagnosis (which step, screenshot, likely cause).
4. Maintains them: after app deploys, re-runs, notices selector drift/broken flows, and repairs specs — filing findings the owner can forward to the dev side (one-way integration; the QA agent never edits app code).

The UI (a separate page, not a dev tool)

A PAGS agent instance page (standard agent board pattern):

• App picker (owner's PAS apps) → flow list with pass/fail status and last-run times
• "Watch" mode: embeds the PAS observable runner — the app in an iframe, steps executing visibly, step list with live pass/fail
• Run history + failure screenshots; a chat panel to instruct the agent ("cover the puzzle wrong-attempt cap", "why did flow 3 fail?")
• Zero code shown by default — this is a QA surface, not an IDE

Dependencies / integration

• Substrate: Observable in-browser test execution: platform-run app automation, specs stored off-repo, Playwright-translatable proappstore-online/platform#38 (spec format, off-repo storage, observable in-browser runner, Browser Rendering headless runs, PW transpilation, test-flows API). This issue is the agent + UI consuming it.
• Auth for flows: PAS credential accounts (QA test org) — sign-in happens through the real UI inside the iframe.
• Agent runtime: standard PAGS agent (Workers AI / BYO key), instance knowledge holds the app's flow inventory.

Acceptance criteria

• The agent, pointed at chess-academy, proposes a sensible flow list and writes ≥3 runnable specs without human-authored steps
• Flows run visibly in the embedded runner and headlessly on the platform, from the agent's own page
• A deliberately broken flow produces a diagnosis with step + screenshot
• The chess-academy repo remains untouched by all of the above
• The page is fully separate from any development/build UI

🤖 Generated with Claude Code

[serge-ivo]

Shipped & verified live ✅

The QA Automation agent is built, deployed, and published on ProAgentStore — a dedicated QA-mindset agent, separate from the dev/Agent-Teams loop, delivered the PAGS way (its own product page, nothing to do with the development console).

• Repo: ProAgentStore/qa-automation · live: https://qa-automation.proagentstore.online
• Consumes the PAS platform substrate from Observable in-browser test execution: platform-run app automation, specs stored off-repo, Playwright-translatable proappstore-online/platform#38 (specs live in the platform, never in the app repo).

What it does (all verified against chess-academy in prod)

• connect <app-id> <qa-key> — links an app via a scoped QA key (never an owner session token). Verified by listing flows.
• flows [app] — lists flows with last-run status + live "watch" links to /__qa/.
• run [flow] [app] — queues headless runs on the platform's Browser Rendering; returns watch links. Verified: agent queued runs → all passed (incl. student-credential-signin 8/8, a real authenticated student sign-in).
• status [app] — recent results with failure step/error + an LLM diagnosis (guarded against garbled model output).
• AI flow authoring — describe a flow in plain language → the agent (QA mindset: prefers assertions over waits, adds a negative/edge check) writes a validated spec and PUTs it to the platform. Verified live: authored signed-out-visitor-signin (11 steps incl. a wrong-password negative check) using caller-provided Workers AI, then ran it headlessly.
• BYOK billing — Workers AI credentials are caller-provided (X-CF-Account-ID/X-CF-AI-Token); deterministic commands (connect/flows/run/status) need no AI key.

Design honored

• QA mindset, not dev team ✅ · separate agent + own page ✅ · watch live in-browser ✅ (/__qa/) · specs off-repo ✅ · translatable to Playwright ✅ · no local installs ✅.

Published (visibility=published). Closing as delivered.