fix(cu): harden merged executor boundaries - #910
Conversation
94a0305 to
158474f
Compare
|
@Astro-Han Updated on current main after #895/#896/#897. CI typecheck/test/e2e are green. This PR now also absorbs the two actionable executor P2s from #905: verified Electron CDP semantic click/text remains reachable while CGEvent fallback stays disabled, and native semantic AX dispatch emits a privacy-safe trace for restart oracles. Recommended merge queue: #910, #911, then #913. |
|
@astrohan Real-model AX integration found a remaining semantic/refetch boundary on current #910 tip. Scenario: real gpt-5.6-sol observes one |
Astro-Han
left a comment
There was a problem hiding this comment.
Approved on dae38967. I found no P0 or P1. CI is green, the full workspace rebuild and stale check passed locally, and the Computer Use suite passed 119/119. The findings below are non-blocking.
P2
-
The window fingerprint now includes every AX label and value (
packages/computer-use/src/cua-driver-backend.ts:350-364). An unrelated clock, progress label, or status counter changing between observation and action changes the whole fingerprint, so a bound coordinate action or Electron semantic pointer action returnstarget_changedeven when its target is unchanged. The tests replace the production fingerprint with a constant and do not cover this case. -
Several failures after delivery still lose the delivered state. Native and Electron text writes return
capture_failedwhen readback cannot confirm the write (cua-driver-backend.ts:1551-1563,:1652-1668), and a successful native semantic action can becomecapture_failedorsensitivity_blockedwhen its fresh screenshot fails (:1965-1980). These should preserveoutcome_unknownso the model is not encouraged to repeat a write or click that may already have happened. -
Keyboard ownership validates only the turn and PID/window ID (
cua-driver-backend.ts:1468-1482). If that window moves or its fields reflow after the click,typereuses the old screen point and can select a different empty field in the same window. A successful readback then reports success for the wrong field. -
When the original observation could not establish Electron page identity, a later bound action can resolve a current CDP target and dispatch without a document identity check (
cua-driver-backend.ts:929-939,:1716-1726,:2075-2101). A reload or navigation to a structurally identical page can therefore operate on a different document.
P3
-
observeResolvedWindow()stores the observation before screenshot normalization. A missing or oversized screenshot throws without returning the ID, but the unreachable observation still consumes one of the 16 FIFO slots and can evict an earlier usable observation. -
The model-facing tool description in
packages/runtime/src/computer-use-tools.ts:1009-1011still says Electron text targets are refused, although this change keeps the CDP click and text path reachable. Models following the description will not use the new path.
Summary
Post-merge hardening for the actionable findings recorded on #893 and the remaining executor findings from #905 review.
outcome_unknownafter delivered but unverifiable actionsDeliberate boundaries
See
docs/computer-use-executor-hardening.mdfor the finding-by-finding rationale and remaining interface limitation.Verification
npm --workspace @maka/core run buildnpm --workspace @maka/runtime run buildnpm --workspace @maka/computer-use run typecheckgit diff --checkFollow-up to #893. Supersedes the two actionable P2 executor findings on #905.