Skip to content

fix(cu): harden merged executor boundaries - #910

Merged
Astro-Han merged 2 commits into
apache:mainfrom
hqhq1025:codex/cu-executor-hardening
Jul 13, 2026
Merged

fix(cu): harden merged executor boundaries#910
Astro-Han merged 2 commits into
apache:mainfrom
hqhq1025:codex/cu-executor-hardening

Conversation

@hqhq1025

@hqhq1025 hqhq1025 commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Post-merge hardening for the actionable findings recorded on #893 and the remaining executor findings from #905 review.

  • fail closed on ambiguous or replaced semantic elements
  • include label/value in native content fingerprints
  • compress and cap window screenshots like desktop screenshots
  • bound unconsumed observations per session
  • bind keyboard ownership to the current PID/window
  • preserve outcome_unknown after delivered but unverifiable actions
  • reject semantic actions absent from the pinned driver registry
  • keep verified Electron CDP semantic click/text reachable while compatibility CGEvent input remains disabled
  • emit privacy-safe native semantic dispatch traces so restart tests can prove whether AX dispatch occurred

Deliberate boundaries

  • compatibility coordinate/scroll/drag/key dispatch remains disabled
  • the shared operation queue remains global because the current backend owns one action-child connection and must preserve fresh-snapshot/action ordering; cross-session concurrency needs isolated connections, not removal of the transaction boundary
  • Desktop supplies the physical-input guard before native semantic dispatch

See docs/computer-use-executor-hardening.md for the finding-by-finding rationale and remaining interface limitation.

Verification

  • npm --workspace @maka/core run build
  • npm --workspace @maka/runtime run build
  • npm --workspace @maka/computer-use run typecheck
  • complete Computer Use suite serialized: 119/119
  • git diff --check

Follow-up to #893. Supersedes the two actionable P2 executor findings on #905.

@hqhq1025
hqhq1025 force-pushed the codex/cu-executor-hardening branch from 94a0305 to 158474f Compare July 13, 2026 18:59
@hqhq1025

Copy link
Copy Markdown
Contributor Author

@Astro-Han Updated on current main after #895/#896/#897. CI typecheck/test/e2e are green. This PR now also absorbs the two actionable executor P2s from #905: verified Electron CDP semantic click/text remains reachable while CGEvent fallback stays disabled, and native semantic AX dispatch emits a privacy-safe trace for restart oracles. Recommended merge queue: #910, #911, then #913.

@hqhq1025

Copy link
Copy Markdown
Contributor Author

@astrohan Real-model AX integration found a remaining semantic/refetch boundary on current #910 tip. Scenario: real gpt-5.6-sol observes one CUA Lab Stale Target; an independent AX setup action inserts a second same-role/label/value/identifier target; the model then calls click_element with the old observation. Result is fail-closed target_changed, zero model dispatch and zero mutation, because validateStoredWindow(..., mode=semantic) rejects the changed content fingerprint before refetchSemanticElement can classify ambiguity. This is safe, but stricter than recovered Codex behavior (AX structure may change, then strict role/title/identifier/value refetch decides unique/missing/ambiguous). Please assess whether semantic mode should allow content-fingerprint change through to identity refetch while coordinate mode retains the current fingerprint gate. I did not modify cua-driver-backend.ts. Evidence lives in local integration branch codex/cu-model-real-ax-integration; exact outcome: click_element -> target_changed, setup AX dispatch 1, model dispatch 0, stale/wrong target mutations 0.

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved on dae38967. I found no P0 or P1. CI is green, the full workspace rebuild and stale check passed locally, and the Computer Use suite passed 119/119. The findings below are non-blocking.

P2

  1. The window fingerprint now includes every AX label and value (packages/computer-use/src/cua-driver-backend.ts:350-364). An unrelated clock, progress label, or status counter changing between observation and action changes the whole fingerprint, so a bound coordinate action or Electron semantic pointer action returns target_changed even when its target is unchanged. The tests replace the production fingerprint with a constant and do not cover this case.

  2. Several failures after delivery still lose the delivered state. Native and Electron text writes return capture_failed when readback cannot confirm the write (cua-driver-backend.ts:1551-1563, :1652-1668), and a successful native semantic action can become capture_failed or sensitivity_blocked when its fresh screenshot fails (:1965-1980). These should preserve outcome_unknown so the model is not encouraged to repeat a write or click that may already have happened.

  3. Keyboard ownership validates only the turn and PID/window ID (cua-driver-backend.ts:1468-1482). If that window moves or its fields reflow after the click, type reuses the old screen point and can select a different empty field in the same window. A successful readback then reports success for the wrong field.

  4. When the original observation could not establish Electron page identity, a later bound action can resolve a current CDP target and dispatch without a document identity check (cua-driver-backend.ts:929-939, :1716-1726, :2075-2101). A reload or navigation to a structurally identical page can therefore operate on a different document.

P3

  1. observeResolvedWindow() stores the observation before screenshot normalization. A missing or oversized screenshot throws without returning the ID, but the unreachable observation still consumes one of the 16 FIFO slots and can evict an earlier usable observation.

  2. The model-facing tool description in packages/runtime/src/computer-use-tools.ts:1009-1011 still says Electron text targets are refused, although this change keeps the CDP click and text path reachable. Models following the description will not use the new path.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants