fix(cu): classify and gate semantic failures - #918
Conversation
|
@astrohan CI is green. Additional live evidence: the real gpt-5.6-sol AppKit ambiguity scenario returned |
Astro-Han
left a comment
There was a problem hiding this comment.
Approved on 4307ad77. I found no P0 or P1. GitHub CI is green. On current main, the full rebuild and stale check passed, the focused loop-gate suite passed 15/15, and the Computer Use tool tests passed 58/58.
P2
-
Preserve and correctly scope the ambiguity gate across re-observation. An ambiguous semantic refetch consumes the active frame and returns no fresh observation, so the real recovery path must call
observebefore retrying. That call has nocomputerSemanticSignature, which clearslastAmbiguousComputerSignatureatpackages/runtime/src/tool-runtime.ts:407-411; its successful result clears it again at lines 709-711. In an integrated reproduction,ambiguous click(obs-1) -> observe -> same click(obs-2)reached the implementation all three times. The added test skips the required observe call.The cached key at lines 924-929 also contains only the action, frame-local
element_id, value, and text. It has no app, window, role, or label scope. If the gate is preserved across observe, the same index in another window can be blocked as the old target, while the same target can bypass the gate when its index changes. Carry the bound element identity and window scope in the failure metadata, preserve the gate through observation calls, and clear it only when the target or action actually changes.
Summary
Fixes two Runtime issues found by a real-provider, real-AppKit AX model run:
{ text, error: 'stale_frame' }were projected to text and incorrectly classified as successful tool invocations;observation_id.Changes
errorvalues as failed tool results for persistence, events, telemetry, and the generic loop gate;ambiguous_target;observation_id;user_intervened,target_missing, and ordinary stale/re-observe recovery uncached.Real evidence
A live
gpt-5.6-solrun throughAiSdkBackend -> ToolRuntime -> maka_computer -> cua-driver -> AppKit AXattempted a duplicated stale element. Before this fix, thestale_frameresult was recorded as telemetry success and the model could retry after another observation. The real run remained safe (zero model dispatch and zero target mutation), but the Runtime accounting and retry control were wrong.Verification