Skip to content

feat(cu): add runtime observation and action leases - #892

Merged
Astro-Han merged 1 commit into
apache:mainfrom
hqhq1025:codex/cu-runtime-foundation
Jul 13, 2026
Merged

feat(cu): add runtime observation and action leases#892
Astro-Han merged 1 commit into
apache:mainfrom
hqhq1025:codex/cu-runtime-foundation

Conversation

@hqhq1025

Copy link
Copy Markdown
Contributor

Upstream stack notice

This is stack PR B. It depends on #857 and must not merge before #857.

Because GitHub cannot use a branch from the contributor fork as this PR base, the Files tab currently includes PR A. After #857 merges, this branch will be rebased onto main and the diff will shrink to the Runtime-owned slice.

Review the exact 9-file net diff now in fork-local PR hqhq1025#1.

Current rebase verification: full repository build; Runtime 1452 passed with 7 platform skips; full repository typecheck.


Stack position

PR B, stacked on codex/cu-foundation-core (upstream draft PR #857).

This fork-local PR preserves the real stacked diff because the upstream repository cannot use a fork branch as a PR base. After #857 merges, this branch will be retargeted to maka-agent/main and opened upstream.

Summary

Adds the provider-neutral Runtime layer without adding a native executor, Desktop wiring, provider adapter, or cursor presentation.

  • one-shot frame/epoch observations and canonical action fingerprints;
  • capture-local window binding for coordinate actions;
  • semantic element identity binding;
  • per-session invocation queues without global serialization;
  • session generation leases for intervention, lock, re-observe, blocked URL, and user stop;
  • stale/replay/duplicate action rejection;
  • fresh full observation requirement after every mutation;
  • unknown dispatch outcome requires re-observation;
  • same-turn terminal tombstones with explicit new-turn recovery;
  • privacy split between persisted summaries and provider-visible AX/screenshot state;
  • model-visible screenshot output through the generic tool result mapping.

Codex lab alignment

  • maxInFlight: 1 is implemented per Computer Use session, not globally;
  • approval does not create an active session: actions require a successful observation;
  • unrelated dynamic AX/DOM changes do not synthesize user intervention;
  • physical intervention, lock, blocked URL, and user stop invalidate old leases;
  • ambiguous/stale semantic identity remains an executor responsibility and must fail closed;
  • raw AX labels, values, window titles, and app inventory do not enter persisted tool results.

Non-goals

  • cua-driver transport and service lifecycle;
  • page/element refetch implementation;
  • Desktop event producers;
  • provider-specific scaling or native tools;
  • cursor/PiP presentation;
  • real-model E2E.

Verification

  • focused Computer Use Runtime tests: 55/55;
  • Runtime full test: 1420 passed, 2 skipped, 0 failed;
  • Core: 833/833;
  • UI: 125/125;
  • Desktop: 2384/2385 in the aggregate run; the only Rive child-process timeout passed 10/10 on immediate isolated rerun;
  • full repository build passed;
  • full repository typecheck passed;
  • adversarial review findings for lock recovery, queued keyboard actions, clearSession tombstones, and app-title persistence were fixed and covered by regression tests.

@hqhq1025
hqhq1025 marked this pull request as ready for review July 13, 2026 16:29
@hqhq1025

Copy link
Copy Markdown
Contributor Author

@astrohan This Computer Use stack slice is ready for review. Please follow the dependency and merge order stated at the top of the PR; the exact net diff is linked there.

@hqhq1025
hqhq1025 force-pushed the codex/cu-runtime-foundation branch from 388c01d to 7c99c35 Compare July 13, 2026 18:03

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. I found no P0 or P1. These P2s are worth tracking, but none should hold up this PR.

P2

  1. clearSession() only marks an existing session-state record as stopped. A first invocation queued before sessionState() runs can survive a direct clear and later reactivate the session (computer-use-tools.ts:857, 1245). Desktop Stop aborts the runtime signal in the same task, so this is not a P1 in the shipped path, but the public cleanup contract is incomplete.
  2. screenshot, list_apps, cursor_position, and wait take neither an observation lease nor an action lease (computer-use-tools.ts:859-890, 898-1010). A caller that uses clearSession() without aborting the signal can still read host state after user_stopped.
  3. Terminal states are not absorbing. screenLocked(), reobserveRequired(), and similar events can overwrite user_stopped or blocked_url, and a later unlock can reopen observation (cua-session-state.ts:84-113).
  4. Controlled Computer Use failures return { text, error }, then get normalized to ordinary text and recorded as successful tool invocations (tool-runtime.ts:568-600). This also resets the identical-failure loop gate.
  5. Backend exceptions can persist raw UI-derived text. A targeted screenshot calls observeApp() without a Computer Use-specific exception projection (computer-use-tools.ts:981-985), while the generic runtime stores the thrown message after secret-pattern redaction only (tool-runtime.ts:643-710, 828-832). AX labels and window titles that are not secret-shaped can enter session history.
  6. When a backend returns observation and a separate screenshot, freshFullObservation() registers the observation without merging the screenshot dimensions (computer-use-tools.ts:748-773). The next bound action can fail with target_missing because the new frame has no usable coordinate bounds.
  7. cursor_position drops resolvedScreenPoint. The action can succeed without returning the coordinates the model asked for.
  8. permissionArgs() parses the action-specific schema before the tool-call record and runtime error boundary exist (tool-runtime.ts:280-311). A wire-valid but action-invalid call, such as a click without coordinates, can escape without a persisted tool_call or tool_result, telemetry, or loop-gate accounting.

Validation: the isolated Runtime suite passed with 1,462 tests and 7 platform skips. Typecheck and git diff --check also passed.

@Astro-Han
Astro-Han merged commit 766d078 into apache:main Jul 13, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants