Skip to content

feat(cu): add desktop presentation lifecycle - #896

Merged
Astro-Han merged 1 commit into
apache:mainfrom
hqhq1025:codex/cu-presentation-lifecycle
Jul 13, 2026
Merged

feat(cu): add desktop presentation lifecycle#896
Astro-Han merged 1 commit into
apache:mainfrom
hqhq1025:codex/cu-presentation-lifecycle

Conversation

@hqhq1025

@hqhq1025 hqhq1025 commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Upstream stack notice

This is stack PR F. It depends on #895 and must not merge before it.

The Files tab is cumulative until preceding fork branches are rebased after merge.

Review the exact 24-file presentation net diff now in fork-local PR hqhq1025#5.

Current rebase verification: overlay and full repository build; @maka/computer-use 109/109; Desktop typecheck; Knip overlay entry audit; upstream typecheck, test, and e2e all passed.


Goal

Add the Desktop Computer Use presentation lifecycle as a reviewable layer above verified targeting and dispatch.

What this PR does

  • adds a focusless, click-through, transparent agent-cursor BrowserWindow
  • builds the overlay renderer and receive-mostly preload in normal build/dev flows
  • maps only Runtime-bound observation coordinates into screen presentation coordinates
  • waits up to 1 second for readyForInteraction, then fails open into native dispatch
  • never waits for visual finished before returning the tool result
  • commits visual completion only after Runtime accepts the post-dispatch lease and fresh observation
  • cancels presentation for failed, stale, intervened, aborted, or otherwise discarded results
  • validates mouse_move against the bound window and returns an executor-resolved completion point without injecting native pointer input
  • serializes operations through the single visual cursor while keeping no-overlay session execution independent
  • immediately releases active and queued presentation waits on session clear
  • binds renderer acknowledgements to session, action, BrowserWindow generation, and current window identity
  • tears down on load failure, renderer loss, display-layout change, session terminal boundaries, main-window close, and app shutdown
  • supports the union bounds of multiple displays, including negative origins

Reverse-engineering constraints applied

The implementation follows the local Codex Computer Use lab findings:

  • presentation identity is late-bound and is not execution evidence
  • the visible completion coordinate comes only from executor resolvedScreenPoint
  • presentation readiness is bounded and cannot redefine the execution target
  • intervention/stale/stop states discard the result and must not show a success pulse
  • stale renderer acknowledgements cannot settle a new operation
  • the overlay cannot take focus or receive mouse input

Non-goals

  • provider-specific Computer Use adapters or model loops
  • physical-input, lock/unlock, blocked-URL, or service-loss event producers
  • deterministic cumulative Electron E2E
  • release packaging/signing/notarization
  • Headless presentation support

Verification

  • npm --workspace @maka/runtime test — full suite passed
  • Runtime focused Computer Use tests — 47 passed
  • npm --workspace @maka/computer-use test — 103 passed
  • npm --workspace @maka/desktop run typecheck
  • npm --workspace @maka/desktop test — 2419 passed
  • npm --workspace @maka/desktop run build
  • overlay controller/preload focused tests — 16 passed
  • git diff --check

Real Electron smoke boundary

An earlier protocol revision produced a visible nonblank cursor screenshot. After adding session+generation acknowledgements, this desktop environment could not reliably launch a second isolated Electron app while Codex/ChatGPT Electron was active, so the current protocol's real-window smoke is intentionally deferred to the cumulative PR G harness rather than claimed as complete here.

Stacked on #4.

@hqhq1025
hqhq1025 marked this pull request as ready for review July 13, 2026 16:30
@hqhq1025

Copy link
Copy Markdown
Contributor Author

@astrohan This Computer Use stack slice is ready for review. Please follow the dependency and merge order stated at the top of the PR; the exact net diff is linked there.

@hqhq1025

Copy link
Copy Markdown
Contributor Author

@astrohan P0 blocker found in live concurrent use: the current compatibility driver PID-bound CGEvent path does not move the pointer or steal focus, but it can still interfere with the user physical mouse button state. Do not merge the Computer Use stack as fully background-safe yet. I am preparing a fail-closed follow-up that disables every CGEvent mouse/keyboard path and retains only AX/CDP/screenshot operations until the native event backend can prove isolation.

@hqhq1025
hqhq1025 force-pushed the codex/cu-presentation-lifecycle branch from 93b2bf9 to e645f29 Compare July 13, 2026 18:04
@hqhq1025

Copy link
Copy Markdown
Contributor Author

@astrohan P0 follow-up #905 now contains the fail-closed fix for the live physical-click interference. Coordinate CGEvent click/scroll/drag/key paths are disabled by default; please include #905 before treating the stack as concurrent-user safe.

@hqhq1025
hqhq1025 force-pushed the codex/cu-presentation-lifecycle branch from e645f29 to 3e29b92 Compare July 13, 2026 18:14

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved on content. I found no P0 or P1. The branch still needs to follow its parents before merge; none of these findings requires another fix round.

P2

  1. hasOpenWindows() counts every Electron BrowserWindow, including the cursor overlay (apps/desktop/src/main/main-window.ts:430-432). If the main window closes while an overlay remains alive, Dock activation or a second launch takes the focus branch instead of recreating the main window (apps/desktop/src/main/main.ts:492-500). The controller then tries to focus a missing main window, leaving the app without usable UI until the overlay is torn down or the process is relaunched.
  2. clearSession() resolves presentation-ready waiters, but runWithPresentation() does not recheck the presentation generation after that wait (packages/runtime/src/computer-use-tools.ts:978-988, 1512-1522). An action without an action lease, such as wait, can still enter backend dispatch after the clear and hold the process-wide presentation queue until it finishes. Desktop's immediate runtime abort narrows this path, so it remains P2.

P3

  1. Overlay cancel reports finished and clears activeActionId, but it does not stop the cursor engine's active path or click pulse (apps/desktop/src/overlay/cursor-overlay.ts:128-133). The canceled animation can continue visibly.
  2. Runtime releases presentation ownership as soon as backend dispatch returns and does not wait for fence.finished (packages/runtime/src/computer-use-tools.ts:965-977). A following session can start while the renderer is still completing the prior visual tail, which may be dropped.
  3. The bloom gradient is centered on the action hotspot, but its circle is drawn around the arrow body position (apps/desktop/src/renderer/computer-use-overlay/engine/cursor-engine.ts:152-165). The bloom is visibly offset and clipped by roughly the arrow-tip length.

The focused presentation tests passed on the reviewed patch. A restack is still required before squash merge.

@Astro-Han
Astro-Han force-pushed the codex/cu-presentation-lifecycle branch from 3e29b92 to fc2fd19 Compare July 13, 2026 18:55

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Restack-only rereview on fc2fd195. Its patch-id exactly matches the reviewed presentation change. Typecheck, test, and e2e are green. No P0 or P1.

P2

  1. hasOpenWindows() counts the cursor overlay, so Dock activation can fail to recreate a closed main window while the overlay remains.
  2. clearSession() wakes presentation readiness without a post-wait generation check; an unleased action such as wait can still dispatch and hold the global presentation queue.

P3

  1. Cancel reports the presentation finished without stopping the cursor animation.
  2. Presentation ownership is released before the renderer's finished fence, so the next session can drop the prior visual tail.
  3. The bloom gradient and its painted circle use different centers, leaving the bloom offset by the arrow-tip length.

All remain non-blocking. Approved for squash merge with no additional fix round.

@Astro-Han
Astro-Han merged commit 6a0a4fe into apache:main Jul 13, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants