fix(cu): coordinate execution with cursor presentation - #777
Closed
hqhq1025 wants to merge 73 commits into
Closed
Conversation
Zero-dependency type foundation for host-level computer use, locking the Path 18 (smoke.md S12-S18) invariants into shared vocabulary before any runner/overlay lands: - ComputerUseErrorCode: the closed S17 fail-closed error enum (7 codes) - CuAction: normalized action union adapting Anthropic computer_20251124 (coordinate/text-modifier/scroll/zoom), leaving room for other adapters - ComputerUseScreenFrame + 2MB cap: the S15b typed provider-frame boundary - ComputerUseDispatchTier (ax | coordinate-background | foreground-visible): the capability-probed ladder so degradation is reported, never silent - ComputerUseActionOutcome: typed success(tier,verified)/failure(S17 code) Pins the model tool contract (computer_20251124 + computer-use-2025-11-24) for Opus 4.8 via coproxy. Pure additive types; 9 unit tests; core 694/694.
… wip) Native NDJSON-over-stdio helper the main process spawns for Tier-1, public-API, background computer use on macOS: Accessibility action dispatch (AXPress / AXSetValue) + capture. No private SkyLight SPI, no global CGEventPost HID-tap — never moves the real cursor or steals focus (measured: frontmost + cursor unchanged across every op). Ops: preflight (live TCC) / screenshot (fail-closed on missing Screen Recording, 2MB S15b cap) / click / type / key. Findings baked in from on-device validation (macOS 26.5): - AXPress can return success while doing nothing → every mutating op reports honest `verified` via readback; window-controls re-read state. - Background coordinate clicks must be pid-scoped hit-tests (occlusion- independent, app-scoped refs dispatch reliably); global element-at- position hits the occluding window and its ref can silently no-op. - Background AX traversal is transiently flaky → op-level retries. Typed S17 error enum on every failure; no swallowed catch, no faked success. README documents the signing/notarization + ScreenCaptureKit productionization TODO (the biggest new-infra item).
…ispatch seam (PR-RUNTIME-CU) - permission: add `computer_use` ToolCategory (block in explore, prompt in ask+execute like browser — host control of real apps is irreversible, always prompt; the overlay + per-turn approval are the safety net; bypass allows). Single turn-scope so one approval carries the screenshot→click→type loop. New 'computer_use' permission reason (permission.ts + events.ts). - @maka/runtime `computer` MakaTool (the name Anthropic's model emits) + CuDispatchBackend seam (desktop spawns the signed helper behind it) + adaptToCuAction mapping the flat computer_20251124 grammar → CuAction. Owns the OS-independent Path 18 duties: per-action TCC re-check (S12, fail-closed), coordinate authority stays runtime-side (S15), typed errors (S17), AbortSignal short-circuit (S18). Honest summaries: verified=false tells the model to re-screenshot — never a silent success. Tests: runtime computer-use-tools 14/14; full runtime typecheck clean; core 694/694. (2 unrelated Bash-streaming/shell-exec suites are pre-existing sandbox timing flakiness, not touched by this change.) Next increment (provider-tool wiring, spec'd): register anthropic.tools.computer_20251124 under map key 'computer' in ai-sdk-backend, execute=wrapToolExecute(computerTool), toModelOutput→image-data for screenshots.
…JSON (PR-RUNTIME-CU)
Concrete Tier-1 backend injected into buildComputerUseTools({backend}). Per-
request spawn of maka-cu-helper (stateless; inherits Electron's TCC grants, no
2nd prompt), one NDJSON request line → one response line → typed
ComputerUseActionOutcome. Maps CuAction → helper op (screenshot writes a temp
PNG read back to base64 with the 2MB S15b cap; click/type/key; wait sleeps;
unimplemented actions fail closed honestly, never faked). AbortSignal + timeout
kill the child.
Verified against the real helper binary: preflight maps
{accessibility:true,screenRecording:false}; screenshot fail-closes to
permission_missing (no capture — privacy-safe); invalid coordinate round-trips
to a typed invalid_coordinate. Desktop main typecheck: 0 errors.
…ed_action code
Wire the model-facing `computer` tool's vision + error surface, and add the
S17 code the cua-driver backend needs to fail closed on keyboard:
- tool-runtime: optional MakaTool.toModelOutput (input/output typed unknown so
MakaTool stays covariant/assignable); ai-sdk-backend forwards it. Lets a tool
return a native image block — a screenshot the vision model can SEE — while
session history keeps only the text summary (coerceResultContent drops the
screenshot field, so the <=2MB frame never bloats history).
- computer-use-tools: structured { text, screenshot } result feeding
toModelOutput; redact backend-supplied outcome.message via redactSecrets at
the runtime chokepoint (S16 — cua-driver does not redact upstream).
- core: add 'unsupported_action' to the closed S17 enum (+ smoke.md S17 +
enum lock test) — a backend refusing an action it cannot do SAFELY, e.g.
cua-driver keyboard whose only target is the user's frontmost window.
…eview-hardened Adds the Tier-2 coordinate-background backend (trycua/cua-driver v0.7.1, MIT, embedded MCP over stdio) behind CuDispatchBackend, selected by select-backend (fails closed off macOS / missing binary → zero tools, capability unadvertised), resolved via cua-driver-path (<Resources>/bin, dev-repo fallback), and disposed on before-quit. Hardened against an adversarial review (12 confirmed findings): - KEYBOARD FAILS CLOSED. Removed the frontmostPid() masking shim: cua-driver keyboard needs a pid and the only one resolvable is the OS-frontmost app = the user's active window. type/key now return unsupported_action instead of injecting keystrokes into whatever the user is using (non-negotiable safety). - No startup deadlock: the initialize/set_config handshake is per-request timeout-bounded and abort-aware; any handshake failure SIGKILLs the child so the next action retries fresh (was: raw request() with no timeout → a silent child wedged every future action forever). - set_config failure fails CLOSED (rejects start) instead of warn-and-continue against an unconfigured desktop scope. - Drain stderr (bounded tail) so a chatty child can't fill the OS pipe and hang. - stdin 'error' listener routes EPIPE into orderly teardown instead of crashing the Electron main process. - Bounded stdout transport buffer (32MB) — tear down a runaway/garbage stream. - select-backend logs a genuine construction failure (distinct from the legitimate binary-absent path) while still failing closed. Tests: 8 backend (incl. keyboard fail-closed, hung-handshake timeout+kill, set_config fail-closed) + 2 path resolver — all green against a mock driver.
…e e2e)
Ran a read-only protocol probe against the REAL cua-driver v0.7.1 binary
(handshake + describe + check_permissions, no capture/click/type). Findings:
- Embedded TCC inheritance CONFIRMED: check_permissions.source =
{attribution:'host', embedded:true, note:'reflect the HOST app's TCC grant,
child in host's responsibility chain'} — validates the whole embedded design.
- check_permissions structuredContent keys {accessibility, screen_recording,
screen_recording_capturable, source} match the backend's preflight mapping.
- Keyboard: type_text/press_key are background-safe (delivery_mode:'background',
no focus steal) and target an explicit pid — the mechanism is NOT frontmost-
only as the prior comment claimed. The real limiter is the flat computer
grammar carries no target pid; guessing = frontmost = the user's window. So
fail-closed stays (safe), but the rationale + upgrade path are now accurate.
- No overlay in embedded+no-daemon mode (get_agent_cursor_state → 0 instances).
Behavior unchanged (backend test still 8/8); comment-only correction.
…ing + palette) Faithful TypeScript port of trycua/cua's cursor-overlay Rust crate — the pure, backend-agnostic motion + visual core of the Codex-style agent cursor: - palette.ts — 10 palettes × 5 colours + for_instance(id) stable hash + gradient - dubins.ts — minimum-turning-radius arc–straight–arc path planner (6 solvers) - cursor-engine.ts — tick_swift_constants port: smootherstep speed profile (300→900→200 pts/s along the Dubins path) → spring settle (K=400, C=17, overshoot 0.8), + MoveTo (16px click-offset, off-screen sentinel), + Canvas paint (bloom r22, procedural arrow with tip→tail gradient 0/0.53/1, click pulse). Pure visual layer — never touches the real cursor (empirically 0px move). Backend- agnostic: driven purely by (x,y) per action, so it works for cua-driver Tier-2 AND the AX-helper Tier-1. Verified: 6/6 engine tests (Dubins endpoints/continuity, speed-profile peak, glide+spring convergence to target in ~0.55s, click-pulse timing, palette determinism) + standalone canvas demo.
…gine
A transparent, always-on-top, click-through BrowserWindow that renders the
agent cursor over the real desktop, driven by MAIN with per-action coordinates.
Proven over the real desktop via scripts/cursor-overlay-demo.mjs (12 scripted
moves/clicks; focus + real cursor untouched).
- src/overlay/{cursor-overlay.ts,-preload.ts,.html}: Canvas host running the
CursorEngine on a rAF loop that blocks on idle; receive-only preload
(ipcRenderer.on only → cannot send/inject back, S15) exposing onMove/onReset.
- main/computer-use/cursor-overlay-window.ts: createCursorOverlayController with
the S14 window options (focusable:false, setIgnoreMouseEvents(true,{forward})
armed BEFORE showInactive, screen-saver level, all-workspaces). Persistent per
session — move() sends 'overlay:move' window-local coords over IPC instead of
recreating the window; supersede-no-orphan; synchronous destroy() teardown (S18).
- engine: click/drag actions glide to the target then pulse ON ARRIVAL.
- scripts/build-cursor-overlay.mjs: esbuild bundle (renderer IIFE + CJS preload).
Tests: 5 window-contract (S14 flags, arm-before-show, persistence/no-recreate,
window-local coords, teardown/supersede/fail-closed) — all green.
…actions
Wires the Maka-owned cursor overlay to the runtime so it follows every computer
action, backend-agnostically:
- runtime: buildComputerUseTools gains an optional `overlay` hook (CuOverlayHook);
fires onActionBegin(action, {sessionId,toolCallId}) at the coordinate-authority
point (after adaptToCuAction, before backend.run) and onActionEnd in a finally.
Best-effort — a throwing overlay can never break dispatch. Sits above
backend.run, so it fires identically for cua-driver Tier-2 and ax-helper Tier-1.
- desktop: computer-use-overlay-hook.ts maps CuAction → controller.move, doing the
S15 declared-px → logical-screen transform in MAIN (÷scaleFactor + display origin,
getPrimaryDisplay). Non-coordinate actions (type/key/screenshot/wait) keep the
cursor present without moving it. select-backend threads the hook through.
- main.ts: creates the controller + hook, passes to selectComputerUseBackend, tears
the cursor down per-session on turn-end (streamEvents complete/abort/error) and
unconditionally at before-quit.
Tests: 4 hook (transform 1×/2×/origin, click/scroll/drag/move kind mapping,
non-coord ensure-without-move); runtime CU 14/14 + desktop CU 23/23 unchanged.
- build-cursor-overlay.mjs exports buildCursorOverlay() (still runnable directly). - package.json: build:overlay script + folded into the build chain. - dev.mjs: builds the overlay bundle in parallel with preload so `npm run dev` produces dist/overlay (the controller loads dist/overlay/cursor-overlay.html).
…PRESETS The computer_use ToolCategory/reason (added with the CU foundation) left the exhaustive Record<ReasonKind, ReasonPreset> map incomplete — latent since main-only builds skip @maka/ui; the full `tsc --build tsconfig.lib.json` (npm run dev) failed TS2741. Adds a MousePointer2 'caution' preset. Surfaced by the real-runtime e2e.
…p + e2e harness
Real agent-driven e2e (real connection + runtime + tools) surfaced why the
computer tool never reached the model, and it works end-to-end after these fixes:
- cua-driver-path: in an unpackaged dev run process.resourcesPath points to
Electron's OWN Resources dir, so a resourcesPath-only check looked in the wrong
place and the binary was 'not found' → selectComputerUseBackend returned NONE →
the computer tool was silently absent. resolveCuaDriverBinaryPath now tries BOTH
the packaged path AND the dev repo path. devBinaryPath() also made layout-robust
(walks to the dist root) so the esbuild dev bundle (dist/main/main.js) resolves
the same as the tsc layout.
- cursor-overlay-window: same import.meta.url-in-bundle hazard for the overlay dist
dir — walk to the dist root instead of a fixed ../../ depth.
- main.ts: dev-only MAKA_CU_E2E_PROMPT harness (auto-runs one real NL turn, auto-
approves permissions, logs tool activity) + a [cu-startup] backend/tools diag line.
Verified live: [cu-startup] backend=cua-driver tools=1; opus-4-6 called
computer{screenshot} (real 3024x1964 capture) + left_click (dispatched) + key
(fail-closed 'unsupported_action', keyboard safety holds in the real app).
…WARPS the real cursor
RED LINE: a live e2e run confirmed cua-driver's scope:'desktop' (no-pid) click
synthesizes a GLOBAL CGEvent that moves the user's REAL cursor. The earlier '0px
cursor move' finding was for the click{pid,window_id,x,y} (CGEventPostToPid) path,
NOT the desktop-scope path the backend was using. Stealing the cursor violates the
non-negotiable invariant this whole feature exists to uphold.
- click (all variants) + scroll now fail closed with unsupported_action rather than
warp the cursor. The no-warp path (window-at-point → pid+window_id → CGEventPostToPid,
or the AX element path) needs window/pid resolution — a careful follow-up, not a
speculative live test that could steal the cursor again.
- mouse_move now succeeds as a pure VISUAL agent-cursor glide (no real input, no cua
call) — matches Codex's move_cursor; the overlay hook already animates it.
- computer tool description reframed: no REAL cursor movement (a visual agent-cursor
shows attention); prefer over shelling to cliclick; keyboard unavailable.
Tests updated: click/scroll never reach cua-driver; mouse_move injects nothing.
…ay coord logging - MAKA_CU_E2E_PROMPT accepts a ';;'-delimited scenario list, each run as its own session sequentially (broad suite on one app boot) with a per-scenario summary. - overlay hook logs the declared→screen coordinate transform + kind under the e2e env, which is how the desktop-scope cursor-warp was localized to left_click.
…res safe clicking) Replaces the fail-closed stub with the real no-warp click. Resolves the window under the click point (list_windows, screen-point space, frontmost layer-0 — which also excludes Maka's always-on-top overlay) and clicks via pid+window_id. That forces cua-driver's click_at_xy_with_window_local → SLEventPostToPid/ post_to_pid, which — confirmed at the cua source (mouse.rs) AND empirically on the real binary (0px real-cursor movement across 3 clicks on a scratch window) — does NOT warp the cursor. Fails closed ONLY on empty desktop (no window under the point), where cua-driver's sole path (click_at_xy_desktop) CGWarps the real cursor. - getScale() caches get_screen_size scale_factor to convert the model's device-px coordinate ↔ logical window bounds; window-local device px = coord − origin*scale. - Safety is guaranteed by construction: click ONLY ever sends pid+window_id (never scope:desktop) or fails closed — no warp is possible. Tests: click on a window → pid+window_id, no scope:desktop; empty desktop → fail closed, no click sent; scroll still fail-closed (its desktop-scope also warps).
…ina screenshots Two issues surfaced by the live e2e (user saw TWO agent cursors + blocked screenshots): - Duplicate cursor: the backend spawned cua-driver with --embedded but WITHOUT --no-daemon-relaunch, so cua relaunched its daemon which drew its OWN agent-cursor overlay ON TOP of Maka's. Maka owns the overlay; cua's must not render. Added --no-daemon-relaunch + CUA_DRIVER_RS_MCP_NO_RELAUNCH=1 (verified: 0 cua cursor instances; capture still works in-process). - Screenshot cap: a native Retina full-display PNG (3024x1964) runs 4-6 MB, so the 2 MB S15b cap blocked real screenshots as sensitivity_blocked. Raised to 8 MB — keeping NATIVE resolution so the model's coordinate space stays device px (which the no-warp click resolver depends on). FOLLOW-UP: JPEG-compress at native res instead of a large cap (small payload, same coordinates). Tests updated (spawn argv, cap value); 34/34 CU sweep green.
…and cursor color
- Two cursors persisted because --no-daemon-relaunch alone did NOT suppress cua's
agent-cursor overlay. Add --no-overlay ('Disable the cursor overlay entirely')
to the spawn — the definitive kill switch. Maka owns the overlay.
- Cursor colour: replace the per-session palette hash with a fixed Maka brand
palette derived from the app's primary token --action = oklch(0.62 0.19 264),
so the agent cursor reads as 'Maka' (per 昊卿: match the app theme). setSession
keeps its id param for future multi-agent hue differentiation.
Scroll requires a pid and posts via scroll_wheel_at_xy → post_to_pid (confirmed at cua source: no CGWarp — the cursor warp only exists in the empty-desktop click path). So scroll gets the same treatment as click: resolve the window under the point, scroll it window-locally via pid+window_id (no cursor warp); fail closed on empty desktop. Test: scroll on a window → pid+window_id, no scope:desktop; empty → fail closed.
The 8 MB cap was a stopgap; a Retina full-display capture can still approach the provider's ~5 MB image limit and bloats every turn. Compress frames >1.5 MB to JPEG (quality 82) at NATIVE resolution via Electron nativeImage — coordinates are unchanged (the no-warp click resolver depends on native px), payload drops ~5-10×. Small crisp PNGs (simple screens) pass through untouched. Injected as an optional compressFrame(base64,mime) so the backend stays testable under node --test. Test: large frame → compressFrame applied, mimeType image/jpeg; small frame → compressor not called, stays PNG. 12/12 backend tests green.
…an instant pop) On a short turn (one click) the cursor used to snap to the target + a quick pulse and vanish — near-invisible. Now the first appearance enters from up-and-left of the target so it visibly GLIDES in via the Dubins path. Verified in the ~3s minimal overlay demo (scripts/cursor-overlay-demo.mjs) rather than the full app.
… scale_factor) + fast CLI harness - getScale trusted get_screen_size.scale_factor, which was observed reporting 1 on a Retina display in a CLI context (the app happened to get 2) → clicks mapped off-screen → 'no app window'. Now compute the TRUE ratio: device screenshot width ÷ logical screen width (cached from the last capture), falling back to scale_factor only before the first frame. Robust across contexts. - scripts/cu-cli.mjs: fast (~5s) backend smoke — drives the real createCuaDriverBackend against the binary (no Electron, no LLM) through screenshot/mouse_move/click/scroll/ key and prints outcomes + frame size. This is how backend logic should be iterated (the overlay RENDER uses the ~3s cursor-overlay-demo.mjs; a full agent turn uses the app). The CLI found this very scale bug on its first run. Backend unit tests 12/12 green.
…ision) Replicates the backend's window-resolution + device-px→window-local transform against the real cua-driver on a scratch TextEdit, using debug_image_out to draw a crosshair where the click LANDED, then reads it back (PIL) via the crosshair's line-intersection (peak red column × row — robust vs a centroid). Result: clicks land within 1px of intent at center / upper-left-quarter / lower-right of the window. Also surfaced that list_windows returns chrome windows (the menu bar, owned by the frontmost app) — the probe now filters to a real document window; the backend's containment + z-index sort handles this for content-area clicks. ~6s, no Electron/LLM — the kind of iteration the CLI is for.
…ol path Closes the fidelity gap: cu-tool-cli.mjs imports the REAL buildComputerUseTools (runtime), the REAL cua-driver backend, and the REAL overlay hook, then calls the tool impl with model-shaped action args — so it exercises S12 TCC recheck + adaptToCuAction (flat grammar → CuAction) + the overlay hook's declared-px→screen transform + the backend's window resolution/no-warp click, exactly as the app does. Only the Electron overlay BrowserWindow (visual — covered by the ~3s overlay demo) and the LLM (covered by an occasional full-app run) are stubbed. Verified live: the hook maps left_click [1500,1000] → overlay screen (750,500) at scale 2; key fails closed. ~6s.
…ire headless CLI
Move the OS-independent computer-use backend out of apps/desktop and into a new
zero-Electron package so both the GUI and the CLI can drive it:
packages/computer-use/ (new @maka/computer-use)
- cua-driver-backend / cua-driver-path / helper-backend / select-backend
- computer-use-overlay-hook (decoupled: OverlayCursorSink + CursorMoveInput
types now live here, not in the Electron overlay window)
Electron-bound pieces stay in apps/desktop: cursor-overlay-window.ts (the
BrowserWindow that renders the agent-cursor). main.ts now imports the backend +
hook from @maka/computer-use and feeds it the Electron overlay + JPEG compressor.
CLI wiring (the point of the move): runtime-bootstrap builds the headless
backend via selectComputerUseBackend() with NO overlay — the visual agent-cursor
is Electron-only, so the CLI runs computer-use blind. Because that on-screen
visibility is what makes GUI computer-use safe to watch, the CLI keeps the
capability OPT-IN behind MAKA_CLI_COMPUTER_USE=1; every action still routes
through the permission engine ('ask' mode). The cua-driver child process is
disposed via a new context.dispose() called in cli.ts's finally.
Path resolver hardened for the shared location: devBinaryPath() walks up to the
dir containing apps/desktop instead of a fixed relative depth. Binary itself is
gitignored (fetched by scripts/prepare-cua-driver.mjs, packaged via
extraResources).
Build graph: root workspaces + tsconfig.lib.json + apps/desktop & packages/cli
deps updated. Tests: 18/18 (moved package) + desktop overlay/engine green + 74/74
CLI. tsc --build tsconfig.lib.json clean.
…t entry
Activate the two packaging scripts that were written but never wired:
bundled-tools.json +cuaDriver { repo trycua/cua, version v0.7.1,
tag cua-driver-rs-v0.7.1, asset cua-driver-rs-0.7.1-darwin-universal-binary.tar.gz,
binaryName cua-driver, sha256 43a78c17… } — single source of the version pin.
package.json +prepare:cua-driver (download→verify→extract→marker),
+check:cua-driver-bundle (release gate), and check:release now runs the gate.
Mirrors the officecli prepare/check pattern. Fail-closed: prepare throws if the
sha256 isn't pinned or mismatches; check throws if the binary is absent / not
executable / marker-mismatched. The binary stays gitignored (fetched, not source).
Verified end-to-end: prepare downloaded the 10,270,955-byte tarball, sha256
matched the manifest pin (== official checksums.txt), extracted + wrote the
marker; check passed; a second prepare reported up-to-date (idempotent).
…, no-warp)
The core CuAction (startCoordinate + coordinate) and the runtime adapter already
carried left_click_drag; only the backend dispatch was missing (it fell through
to unsupported_action). Wire it to cua-driver's `drag` tool.
Safety — source-verified against cua-driver-rs v0.7.1 (commit 7caf72b): `drag`
sends its whole down→(interpolated moves)→up sequence through the SAME window-local
post_mouse_event → SLEventPostToPid/CGEventPostToPid path as click. The only
CGWarpMouseCursorPosition in the entire crate is click's pid-less scope:'desktop'
branch, and drag has no such branch (its pid is required). So a pid+window_id drag
never moves the user's REAL cursor — the non-negotiable red line holds.
Design:
- Resolve BOTH endpoints (reuse resolveWindowAt) and require the SAME window.
Cross-window drag can't be one window-local gesture, and cross-app
drag-and-drop needs a real NSDraggingSession this synthetic post_to_pid path
can't establish (cua-driver marks the result unverifiable). Fail closed on
empty desktop (no target window ⇒ no required pid) or cross-window.
- delivery_mode left DEFAULT (Background). Never 'foreground', which would
briefly reorder window z-order/frontmost (a focus disturbance).
- Tool description now advertises drag + its single-window constraint.
Tests: +3 (same-window no-warp coords, endpoint-on-empty-desktop fail-closed,
cross-window fail-closed); second mock window added for the cross-window case
without perturbing existing click/scroll probe points. @maka/computer-use 21/21,
runtime adapter green, desktop bundle + lib graph typecheck clean.
Bring 134 commits of main into the CUA branch so the draft PR is based on current
main (not the 7-05 fork point). Resolved 5 conflicts, keeping both sides' intent:
- runtime/index.ts — main's shell/workspace exports + CUA computer-use exports
- apps/desktop/package.json — build keeps build:overlay (CUA) AND build:test (main)
- cli.ts — took main's startup error-handling + context.close() shape;
the CU child-process teardown is folded into close()
- runtime-bootstrap — MakaCliRuntimeContext keeps main's close(); headless CU
dispose runs inside it; CU tools push onto allTools (the set
handed to the backend), not the already-spread `tools`
- main.ts — kept main's runBeforeQuitCleanup() teardown; the CU
backend.dispose() + overlay.destroyAll() are folded in
Verified post-merge: lib-graph typecheck clean, desktop main bundle clean,
@maka/computer-use 21/21, maka-agent CLI 185/185, runtime 1136/1139 (the one fail
is the pre-existing shell-exec process-tree-kill flaky, unrelated to this merge).
Keyboard (type/key) now works on the cua-driver backend, delivered ONLY to the window the agent last clicked/scrolled/dragged (lastTarget -> type_text / press_key, delivery_mode default Background). No established target -> fail closed; never guesses the user's frontmost window. Anthropic key chords (cmd+a, ctrl+shift+t) parse to cua-driver key + mac modifiers. hold_key stays unsupported (no cua-driver hold-duration primitive). Drop the half-baked, self-built ax-helper (maka-cu-helper Swift) backend entirely -- cua-driver is now the sole backend. This removes the confirmed red-line hole where ax-helper type/key fell back to NSWorkspace.frontmost (= the user's active window). Deleted helper-backend.ts + native/maka-cu-helper/, simplified select-backend to cua-driver only. Audit fixes (from the PR apache#699 multi-dimension review): - packaging: check:release now RE-HASHES the actual cua-driver binary bytes against the pinned sha256 (was: trusted the sidecar marker only); prepare self-heals a swapped binary; unpinned/placeholder sha256 fails closed; temp extraction dir cleaned on the error path; contract test pins the gate into check:release. - overlay: removed dead idle-fade machinery (idleAlpha was provably always 1; the fade branch is unreachable since the rAF loop halts when not moving). - tests: lock the primary device/logical scale path (screenshot_width/ logical_width, not scale_factor), the z-order tiebreak + layer!=0/off-screen exclusion, and delivery_mode != foreground on click/scroll. Tests: computer-use cua-driver-backend 22/22, runtime cu-tools 14/14, desktop cursor-engine + overlay-window + build-hygiene 13/13; lib + desktop typecheck clean.
The @maka/computer-use workspace was added but never inserted into the root package.json sequential build — so on a fresh CI checkout its dist/*.d.ts was never emitted before its consumers (packages/cli, apps/desktop) typechecked, failing every job with `Cannot find module '@maka/computer-use'` (and the downstream implicit-any errors in main.ts). The root build is a MANUAL ordered list (not tsc project-reference auto-build), so a new package must be added by hand. Insert @maka/computer-use right after @maka/runtime (its only lib deps are core + runtime) and before its consumers in all four scripts: build, build:test, test, test:dist. test/test:dist previously also skipped the package's own suite, so its keyboard/safety tests were never gated in CI — now they run. Verified: from a cleared computer-use dist, `npm run build:test` completes with zero `Cannot find module` errors.
hqhq1025
force-pushed
the
codex/cu-model-loop-ux
branch
from
July 12, 2026 14:51
4377827 to
1daec08
Compare
hqhq1025
marked this pull request as ready for review
July 12, 2026 14:51
hqhq1025
force-pushed
the
codex/cu-model-loop-ux
branch
from
July 12, 2026 15:35
8f1eac7 to
9c4a799
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #699. This PR aligns Maka's execution and cursor presentation lifecycle with the reproducible Codex App Computer Use architecture while keeping cua-driver as the sole native executor.
mouse_move.readyForInteractionandfinishedfences.{actionId, phase}for fixedreadyForInteraction|finishedphases.Codex reproduction evidence
The implementation is grounded in
/Users/haoqing/Documents/Learning/codex-computer-use-lab:prepareToInteract, optional UI settle, Skyshot refresh, and separate cursor interaction/completion timing;Maka copies these lifecycle contracts, not Codex's private input implementation.
Cursor fence behavior
presentationReadySource=timeoutrather than being mistaken for a renderer acknowledgement.Verification
readyForInteractionacknowledgement;dispatch + 120000ms;finishedacknowledgement followed later;Example final timing from
cu-presentation-only-final-20260712:Current E2E boundary
The full action-matrix operation-fence rerun currently stops before input because the user's full-screen ChatGPT window wins every fixture point. This is the expected wrong-target fail-closed behavior. The presentation-only smoke independently verifies the new renderer/runtime fence without synthesizing input or requiring another target window. The prior 62/62 full action matrix remains the latest full execution baseline.
Notes
representedObject is not a WeakPtrToElectronMenuModelAsNSObjectwarnings observed during native scroll delivery did not change behavior or trip the safety monitor.