fix(cu): preserve delivered action state - #922
Conversation
|
@Astro-Han CI typecheck/test/e2e is fully green. This addresses the delivered-state and unreachable-observation findings from merged #910: AX/CDP text readback mismatch and post-dispatch fresh-capture failure now return outcome_unknown; failed screenshot normalization no longer consumes FIFO capacity; model copy reflects the unique Electron CDP text path. Computer Use full suite passed 127/127 locally. Ready for review. |
Astro-Han
left a comment
There was a problem hiding this comment.
Approved. I did not find a P0 or P1. This is the follow-up round for #910, so the remaining P2/P3 items are non-blocking.
-
[P2] Preserve delivered state for every fresh-observation failure after a semantic action. The new catches at
cua-driver-backend.ts:1916-1932and:1984-2005handle onlyCuaDriverCaptureError, which covers screenshot normalization. A laterget_window_statefailure, page-identity lookup error, compressor exception, or request-lifecycle error still rejects or becomes an ordinary capture/service failure after the action was delivered. I reproduced this through the page-identity seam: the AX click reached the driver once, the fresh identity read threw, andrunSemanticrejected instead of returningoutcome_unknown. Convert any post-delivery verification failure to the delivered-but-unverified result. -
[P2] Text writes still lose delivery uncertainty when the verifier itself fails. The AX and CDP changes cover a parseable readback mismatch, but not a failing readback request. After native
set_valuesucceeds, a failedsnapshotTargetis still reported ascapture_failed; after CDPinsert_textsucceeds, an inspection error or exception follows the same path. Those cases must also returnoutcome_unknownso the model does not safely retry text that may already be present. -
[P3] Update the module-level keyboard invariant. The comment at
cua-driver-backend.ts:18-24still says Electron targets fail before dispatch, while the implementation and corrected tool description now support uniquely targeted Electron CDP text.
Summary
Follow-up to the non-blocking executor findings on merged #910.
outcome_unknownwhen AX or CDP text delivery cannot be confirmed by readbackoutcome_unknownwhen a delivered semantic action cannot produce its required fresh screenshotSafety boundary
Pre-dispatch target identity, freshness, occlusion, physical-input, and compatibility-input checks are unchanged. This PR only prevents delivered actions from being mislabeled as safely retryable capture failures.
Verification
git diff --check