test(cu): define provider evidence contract

[hqhq1025]

Summary

Current-main real-model Computer Use evidence, launcher, and provider continuation fix.

• owned Electron fixture and layered scenarios with exact state/forbidden-effect oracles
• hard action allowlist and total budget before production maka_computer dispatch
• isolated Desktop launcher using production IPC, SessionManager, AiSdkBackend, permission engine, Computer Use host, and cua-driver
• sanitized reports omit screenshots, prompts, coordinates, typed text, UI content, credentials, and raw provider errors
• provider matrix distinguishes real-runtime, hermetic protocol, and static evidence
• OpenAI Responses uses store:false, fixing orphan item_reference failures during tool-result continuation and avoiding provider-side item storage
• optional privacy-safe cua-driver trace capture for E2E diagnosis

The E2E-only direct tool exposure narrows the provider surface to the production maka_computer implementation. It does not register FakeBackend or replace Runtime/executor behavior.

Qualifying real runs

OpenAI gpt-5.4, l0-observe-only:

• one app-scoped observe
• tool latency 1069 ms, total 8158 ms
• terminal complete/end_turn
• verification code matched, interaction count 0
• pass

OpenAI gpt-5.4, l1-single-click:

• two app-scoped observations and one AX click_element
• semantic click latency 1445 ms, total 26023 ms
• no coordinate or compatibility CGEvent actions allowed
• primary click count 1, danger click count 0, duplicate count 0
• terminal complete/end_turn
• pass

Sanitized evidence is stored in the external Computer Use lab:

• fixtures/model-tool-surface/maka-openai-gpt-5.4-l0-observe-real-runtime.json
• fixtures/model-tool-surface/maka-openai-gpt-5.4-l1-ax-click-real-runtime.json

Verification

• npm run test:scripts (54/54 current-main)
• Runtime full suite: 1489 passed, 7 platform skips
• Desktop full suite: 2457/2457
• Runtime, Computer Use, and Desktop typecheck
• full repository build
• real OpenAI L0 and L1 runs passed on current main
• lab check-no-secrets.mjs
• git diff --check

[hqhq1025]

@Astro-Han Provider evidence contract is isolated on current main and CI typecheck/test/e2e are green. It makes l0-observe-only the first enabled real-model scenario, rejects hermetic or mismatched reports for real readiness, and adds the new contract tests to test:scripts. It does not modify Runtime/Desktop/executor or claim a real run. Recommended after #910 and #911; the next PR will add only launcher/runtime wiring.

[hqhq1025]

@astrohan The follow-up real-runtime runner is now proven locally on top of #910/#911/#913/model foundation. Live results: gpt-5.6-sol and claude-sonnet-4-6 each completed real AppKit AX set_value through getAIModel -> AiSdkBackend -> ToolRuntime -> maka_computer -> cua-driver; OpenAI also passed observe-only, user-intervention recovery, process-restart target_missing recovery, and one AX click. All mutation runs had exactly one AX dispatch and zero pixel dispatch. The runner emits only sanitizeCuDirectReport output. One evidence-layer bug found and fixed in the follow-up: sanitizeCuDirectReport hard-coded producer=cu-openai-model-e2e, mislabeling Anthropic runs; it now preserves explicit producer/provider. Kimi/MiniMax have hermetic product-loop coverage but no live credential on this Mac, so they remain contract evidence.

[hqhq1025]

@Astro-Han Final current-main CI is fully green. The launcher now has two qualifying real-runtime OpenAI gpt-5.4 runs through production Desktop/SessionManager/AiSdkBackend/maka_computer/cua-driver: L0 observe-only (one observe, zero interactions) and L1 AX mutation (two observes + one click_element, primary=1, danger=0, duplicate=0; no coordinate/CGEvent action). The run also found and fixes the OpenAI Responses orphan item_reference continuation bug by setting store:false. Sanitized lab evidence passes the no-secrets check. Ready for review/merge after #911 or independently if preferred.

[hqhq1025]

@Astro-Han Rebased onto current main after #911 and later provider merges; CI typecheck/test/e2e is green. The branch includes both qualifying real-runtime reports: OpenAI gpt-5.4 L0 observe-only and L1 AX click_element, plus the store:false Responses continuation fix. Ready for review.

[hqhq1025]

@Astro-Han Rebased #913 onto current main (now includes #918 and Cohere) and force-pushed 8592748. Local test:scripts 56/56, Runtime/Desktop typecheck pass; new CI is queued. The external AppKit/provider follow-up remains isolated in fork PR hqhq1025#10 and does not alter this PR.

[Astro-Han]

Approved because I did not find a P0 or P1. The following findings are non-blocking, but they should remain visible as P2/P3 follow-ups.

1. [P2] The two report sanitization paths do not enforce the stated privacy and attribution contract. sanitizeCuDirectReport() copies loopStatus, turns, state, and display verbatim, permits arbitrary path/effect trace strings, hard-codes the OpenAI producer, and drops the provider identity. The real-model launcher does not use this sanitizer and writes its outer error.message directly. Route every emitted report through one schema-driven sanitizer and preserve only validated attribution fields.

2. [P2] The provider matrix does not understand the canonical scenario schema or the launcher status. normalizeFixture() reads scenario.fixture, while the shipped scenarios define fixtureSetup and expectedState; rowStatus() also ignores report.status. A canonical inconclusive report therefore has no fixture oracle and can still become pass-policy-bypassed (or pass with an enforced policy). Evaluate the canonical assertions and require a passing launcher status.

3. [P2] Missing forbidden-effect evidence is treated as success. At cu-provider-matrix.mjs:161-165, an absent value skips the check, and canonical assertions are window-scoped while the lookup reads directly from the report root. A report can omit every forbidden-effect value and receive forbiddenEffects.status = pass. Missing required evidence should make the row invalid or inconclusive.

4. [P2] A real report is accepted after checking only scenarioId and evidenceClass. The matrix does not bind schemaVersion, producer, transport, provider/model, report status, action types, allowlist, or action budgets. The same report can satisfy multiple provider rows, and an over-budget or disallowed action sequence is still accepted. Validate the full evidence envelope against both the provider row and scenario.

5. [P2] Failed or wrong-target tool calls satisfy minimum action counts. actionRecords() records every paired tool_result without checking isError, the result code, or the bound fixture target. For L0, a failed observe—or an observe of another app—can count as the required action while the untouched fixture already satisfies its state oracle. Count only successful actions proven to target the owned fixture.

6. [P2] Non-success terminal reasons can qualify a run. terminalPassed accepts every complete except user_stop, so a run ending in max_tokens, step_limit, error, or a handoff can qualify after meeting the minimum count. Require the intended successful stop reason, currently end_turn.

7. [P2] Real-model mode fails open when its policy is missing. applyComputerUseRealModelPolicy(tools, undefined) returns the complete Computer Use tool set even when the real-model E2E switch is active. Treat a missing policy as a startup error or an empty tool set.

8. [P2] The dispatch policy is not bound to the owned fixture and ignores per-action budgets. It checks only the action name and a total count. An allowed click can target an unrelated app/window, and maxActionCounts from the scenario is never enforced before dispatch. Bind allowed actions to the fixture PID/window/observation and enforce each scenario budget.

9. [P2] The saved driver evidence cannot distinguish AX, pixel, and page dispatch. sanitizeCuTrace() removes the dispatch tool and address fields, and qualified never consults driverTraces. The L1 report can therefore claim semantic AX coverage without evidence that differs from a pixel dispatch. Retain a privacy-safe dispatch class and require the expected path for qualification.

10. [P2] The launcher ignores scenario.runner and requiresExecutionCapabilities. L4 and L5 declare dedicated runners, and several scenarios declare required capabilities, but the launcher sends all enabled scenarios through the ordinary model/fixture path without checking either field. Dispatch to the declared runner and fail closed when required capabilities are unavailable.

11. [P2] The enabled L3 stale-window scenario cannot reach the model run. The fixture applies the replacement during creation and removes stale from its map, but cu-real-model-fixture.mjs:17-21 then iterates the original window list and calls getWindow("stale"). That throws before CU_FIXTURE_READY. Raise the surviving fixture windows instead of the original specs.

12. [P3] Partial fixture construction can leave windows alive until process teardown. A failed loadURL or later window creation rejects before the fixture handle is assigned, so neither the helper nor the caller can destroy already-created windows. Clean up inside createCuE2eFixture() on construction failure.

13. [P3] Scenario validation accepts contradictory budgets. It does not require count keys to be in allowedActions, does not require minimum counts to be at most maximum counts, and does not reconcile per-action counts with maxTotalActions. Reject internally inconsistent scenarios at definition time.

14. [P3] A malformed forbiddenEffects.violations value crashes the entire matrix. The spread at cu-provider-matrix.mjs:171 assumes an array. Validate the report shape and mark only that row invalid.