Skip to content

fix(cu): preserve delivered action state - #922

Merged
Astro-Han merged 1 commit into
apache:mainfrom
hqhq1025:codex/cu-delivery-state
Jul 13, 2026
Merged

fix(cu): preserve delivered action state#922
Astro-Han merged 1 commit into
apache:mainfrom
hqhq1025:codex/cu-delivery-state

Conversation

@hqhq1025

Copy link
Copy Markdown
Contributor

Summary

Follow-up to the non-blocking executor findings on merged #910.

  • return outcome_unknown when AX or CDP text delivery cannot be confirmed by readback
  • preserve outcome_unknown when a delivered semantic action cannot produce its required fresh screenshot
  • store observations only after screenshot normalization succeeds, avoiding unreachable FIFO entries
  • update model-facing copy for the uniquely targeted Electron CDP text path

Safety boundary

Pre-dispatch target identity, freshness, occlusion, physical-input, and compatibility-input checks are unchanged. This PR only prevents delivered actions from being mislabeled as safely retryable capture failures.

Verification

  • Computer Use full suite: 127/127
  • Computer Use typecheck
  • Runtime build
  • git diff --check

@hqhq1025

Copy link
Copy Markdown
Contributor Author

@Astro-Han CI typecheck/test/e2e is fully green. This addresses the delivered-state and unreachable-observation findings from merged #910: AX/CDP text readback mismatch and post-dispatch fresh-capture failure now return outcome_unknown; failed screenshot normalization no longer consumes FIFO capacity; model copy reflects the unique Electron CDP text path. Computer Use full suite passed 127/127 locally. Ready for review.

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. I did not find a P0 or P1. This is the follow-up round for #910, so the remaining P2/P3 items are non-blocking.

  1. [P2] Preserve delivered state for every fresh-observation failure after a semantic action. The new catches at cua-driver-backend.ts:1916-1932 and :1984-2005 handle only CuaDriverCaptureError, which covers screenshot normalization. A later get_window_state failure, page-identity lookup error, compressor exception, or request-lifecycle error still rejects or becomes an ordinary capture/service failure after the action was delivered. I reproduced this through the page-identity seam: the AX click reached the driver once, the fresh identity read threw, and runSemantic rejected instead of returning outcome_unknown. Convert any post-delivery verification failure to the delivered-but-unverified result.

  2. [P2] Text writes still lose delivery uncertainty when the verifier itself fails. The AX and CDP changes cover a parseable readback mismatch, but not a failing readback request. After native set_value succeeds, a failed snapshotTarget is still reported as capture_failed; after CDP insert_text succeeds, an inspection error or exception follows the same path. Those cases must also return outcome_unknown so the model does not safely retry text that may already be present.

  3. [P3] Update the module-level keyboard invariant. The comment at cua-driver-backend.ts:18-24 still says Electron targets fail before dispatch, while the implementation and corrected tool description now support uniquely targeted Electron CDP text.

@Astro-Han
Astro-Han merged commit 633bd51 into apache:main Jul 13, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants