fix(headless): harden real-provider smoke reliability - #972
Merged
Conversation
Astro-Han
added a commit
that referenced
this pull request
Jul 14, 2026
… send has no total usage #972 made the terminal LLM-call record fail-closed on usage evidence, but an aborted send (mid-turn exhaust, user stop, stream error) never resolves the SDK totalUsage promise, so the record carrying the capacity verdict diagnostics was skipped entirely. Every COMPLETED step reports real usage at its finish-step boundary; accumulate those samples per send and fall back to the sum at terminal record time. No completed step means no evidence and the record is still skipped, preserving the #972 no-fabrication invariant.
Astro-Han
added a commit
that referenced
this pull request
Jul 15, 2026
…able step sample An unusable completed-step sample (normalizeAiSdkUsage returns undefined, #972) made the accumulated sum a PARTIAL cost, and LlmCallRecord has no partial marker — downstream would read it as the whole call. Track sample completeness per send and use the sum only when every completed step reported usable usage; otherwise keep the fail-closed no-record behavior. The terminal outcome never depended on this record: stopReason and the exhausted detail are durable on the CompleteEvent, now asserted explicitly. Also rewrite the stale pre-#972 'missing tokens normalize to 0' comment at the capacity hook's usage read.
Astro-Han
added a commit
that referenced
this pull request
Jul 15, 2026
…ad verdict owner (#996) * feat(runtime): extend history compact checkpoint protocol to mid_turn phase Add a phase (pre_turn|mid_turn) and head-anchor reference to the V2 HistoryCompactCheckpoint so a checkpoint can fold a contiguous prefix that reaches into the current turn's completed steps while re-rendering the covered head anchor (the current turn's user message) verbatim on replay. Coverage stays a contiguous event prefix so the digest math is unchanged; pre_turn checkpoint ids stay byte-stable. projectHistoryCompactCheckpointReplay centralises the deterministic [block, head anchor, tail] projection. * feat(runtime): add pure mid-turn capacity measurement and safe-boundary engine Turn-agnostic, side-effect-free helpers for the active-turn context invariant: estimateNextRequestTokens anchors on the last step's real provider usage plus a char/4 tail delta (whole-projection char/4 on cold start); exceedsHighWater and exceedsContextWindow gate the two failure tiers; selectMidTurnSafeBoundary picks the largest covered prefix that ends on an immutable non-partial event and never straddles a tool call/result pair, reporting no_safe_completed_span otherwise. * feat(runtime): add mid-turn history compact policy surface (default off) HistoryCompactPolicy.midTurn carries enabled + reserveTokens + reserveTailEvents. MAKA_CONTEXT_HISTORY_COMPACT_MID_TURN opts in (default off, PR 3 sinks it on), reusing the shared MAKA_CONTEXT_HISTORY_COMPACT_RESERVE_TOKENS (16384) high-water reserve. A standalone revert leaves every surface's behavior unchanged. * feat(core): add context_budget_exhausted complete outcome A first-class CompleteEvent.stopReason for when the runtime cannot produce a provider-safe request even after mid-turn compaction, with a detail field (no_safe_completed_span | summarizer_failed | head_anchor_exceeds_capacity). failureClassFromCompleteStopReason maps it to a distinct failure class so the turn is recorded as an explicit budget outcome rather than a provider error. * feat(runtime): add mid-turn capacity compaction orchestration planMidTurnCapacityCompaction ties the measurement engine, safe-boundary selection, and the V2 checkpoint protocol into one deterministic decision: skip below the high-water; fold a safe completed prefix into a mid_turn checkpoint (re-rendering the head anchor verbatim and continuing with the preserved tail) via the injected summarizeHistoryCompact seam; roll forward from a matching previous checkpoint. Two failure tiers per the design: below the window a failure fails open, above the window it returns an explicit context_budget_exhausted outcome (no_safe_completed_span / summarizer_failed / head_anchor_exceeds_capacity). Recovery re-projection replay-validates against the same ledger prefix. * feat(core): add phase dimension to compaction decision diagnostics CompactionDecisionDiagnostic and the runtime CompactionDecision carry an optional phase ('pre_turn' | 'mid_turn'); absent on legacy data means pre_turn. Mid-turn capacity compaction records its trigger, replacement, fail-open, and exhausted decisions on the existing compactionDecisions channel with this dimension. * feat(runtime): replay mid_turn checkpoints against the full content projection A mid_turn checkpoint's coverage reaches into the compacted turn's own completed steps, so replay matches it against the full compactable-event projection before the turn-granular guards (tail selection would otherwise retain the covered span and miss the prefix, and a single giant turn must not be rejected as insufficient_turns). Replay stays the deterministic [block, verbatim head anchor, uncovered tail] and the pre_turn path is unchanged. Exports isHistoryCompactContentEvent as the shared predicate for the backend's mid-turn projection. * feat(runtime): wire mid-turn capacity compaction into the streaming backend Completes the issue #882 PR 1 invariant end to end. AiSdkFlow forwards ctx.branch and the persisted head anchor through BackendSendInput; the backend taps its send() queue to accumulate the current turn's content RuntimeEvents with exact ledger identity (same mapper, ids, and branch as the flow), tracks each finished step's real provider usage, and composes a mid-turn prepareStep hook (gated on historyCompact.midTurn.enabled, default off). Between steps it measures the next request as last-step usage plus a char/4 tool-result delta against contextWindow - reserve; over the high water it plans a safe-boundary fold, durably records the mid_turn checkpoint BEFORE replacing the projection, and continues the same turn on the materialized [compact block, verbatim head anchor, preserved tail] without re-executing completed tool calls. Failures under the window fail open with a mid_turn diagnostic; over the window the turn ends with the explicit complete stopReason context_budget_exhausted (no_safe_completed_span / summarizer_failed / head_anchor_exceeds_capacity), aborting the stream and handling AI SDK's graceful abort wind-down. Streaming integration tests cover trigger, persist-before-replace ordering, prompt replacement, branch recovery re-projection, all three exhausted details, fail-open, and the flow plumbing. * fix(runtime): close mid-turn compaction correctness gaps from external review Four verified findings, fixed at their owners: - Full-request re-estimate (F1): after folding, the plan re-estimates the complete next request (usage-anchored estimate minus the covered span's share plus the [block, anchor, tail] projection) instead of comparing only the replacement events to the window, so a huge fixed overhead with a tiny foldable span is exhausted (head_anchor_exceeds_capacity), and a replacement that would GROW past a window the raw request fits fails open (replacement_exceeds_window) rather than replacing. - Partial-free coverage (F5): the safe boundary retreats strictly before the first partial anywhere in the prefix (not just at the cut), and buildHistoryCompactCheckpoint rejects any coverage containing a partial snapshot — a digest over a replaced/deleted snapshot can never replay. - Anchor integrity fail-closed (F6): the builder requires the head anchor to be the covered turn's user event, and matchHistoryCompactCheckpointPrefix fails a mid_turn match as coverage_miss when the anchor reference is corrupted (uncovered id, wrong turn, or non-user role) instead of silently replaying without the user message. - Replay before the high-water skip (F3): an accepted mid_turn checkpoint is a correctness invariant, not a capacity optimization, so its replay match now precedes the below-high-water early return; recovery tests run on normal thresholds instead of a degenerate highWaterRatio. * fix(runtime): keep context_budget_exhausted detail in the durable terminal state The complete-event mapping dropped contextBudgetExhaustedDetail, so the persisted RuntimeEvent could not distinguish no_safe_completed_span / summarizer_failed / head_anchor_exceeds_capacity. completeRuntimeEvent now maps the full CompleteEvent and records the typed detail alongside stopReason/failureClass in the terminal stateDelta, locked by a flow-mapping round-trip test. * refactor: source mid-turn coverage from the durable run ledger, composed first Root fix for the two review P1s about the backend integration (F4, F2): one composed provider-visible projection, coverage only from events already confirmed on disk, and no mirrored state carrying the hard capacity invariant. Durable-read seam (F4): AgentRun exposes loadTurnRuntimeEvents() — it waits for every ledger write enqueued so far, then reads the store — and the kernel injects it into backends next to the checkpoint loader/recorder (BackendFactoryContext, cli and desktop factories pass it through). The mid-turn trigger reads the current turn's persisted RuntimeEvents as its coverage pool, so a checkpoint can never be recorded before its covered source events are durable (the crash window is gone) and byte-identity with recovery replay holds by construction, including under ctx.branch. Last-step real usage now comes synchronously from the SDK's own step results (the same numbers as the finish-step chunk), so the wall-clock waitForSteps synchronization is deleted along with TappedAsyncEventQueue, the SessionEvent mirror mapping, and the reconstructed InvocationContext; the BackendSendInput.branch plumb that existed only for that mirror is removed. A lagging ledger read only shrinks the tail delta of the usage-anchored estimate, and every failure-driven skip (ledger_read_failed, head_anchor_not_durable) records an explicit failedOpen decision — no silent skips. Composition order (F2): composePrepareStep now runs the capacity hook before activeToolResultPrune and semantic/active-full compaction, so prune re-archives large tool results in the rebuilt tail instead of having its placeholders undone, and on the exact step the capacity hook replaced, semantic/active-full compaction yields with a recorded mid_turn_capacity_precedence decision — one step never runs two summarizers. Integration tests now drive the durable-read fixture (consumer persists mapped events exactly like AgentRun before the seam serves them) and add the review-named combinations: ledger-read fail-open diagnostics, midTurn x activeToolResultPrune tail re-convergence, and midTurn x semanticCompact precedence. * fix(runtime): keep open tool calls out of coverage and stop double-counting the tail Two engine findings from the second external review: - Open tool span (N5): straddlesToolPair skipped spans missing one side, so with a zero tail reserve an unmatched function_call could be folded and its later response would arrive as an orphan. A call without a response is now an open span — any cut past the call is unsafe; a response without a call stays inert (its call precedes the pool). - Tail double-count (N2): the post-fold re-estimate added back the whole [block, anchor, tail] replacement although the usage-anchored estimate already contains the retained tail, misreporting rescuable turns as head_anchor_exceeds_capacity (repro: covered 505, tail ~400, estimate 700, window 500). The formula now adds back only the covered span's substitute [block, anchor]; the repro is a regression test. * fix(runtime): pin the mid-turn head anchor to the compacted turn A self-consistent anchor (role user, matching self-reported turnId) could resolve to ANOTHER covered turn's user event — e.g. a prior turn's prompt — and both build and match accepted it, so the replay silently dropped the real current prompt. The compacted turn is the coverage's through turn: builder and matcher now require anchor.turnId to equal it and the event to be an author='user' user event, failing closed (build error / coverage_miss) otherwise; locked by prior-turn-anchor tests on both paths. * fix(runtime): gate the mid-turn trigger on a durable tool-result watermark Root fix for the second recurrence of the sync seam (N1), plus truthful write diagnostics (N6): Durable watermark: the SDK's step results are the source of truth for which tool calls completed, so the trigger derives an explicit watermark from options.steps and, before measuring or selecting coverage, loops until the durable turn ledger contains the FINAL function_call/function_response for every one of them. Each iteration re-reads through the seam — which re-awaits the run's serialized write queue and re-checks store availability after the wait — and the only exits are the watermark itself, an abort (failedOpen ledger_wait_aborted), or a read failure (failedOpen ledger_read_failed): condition-driven, no wall clock. This closes both halves of the repro: a lagging ledger can no longer under-count the tail delta (letting an over-window request out) nor re-count the same results as a fresh delta at the next boundary. The review's consumer-scheduling perturbation is a real fixture mode now: the full integration suite runs twice (immediate + slow consumer, 11 tests each) and a negative control without the watermark fails 10 of the slow-mode tests. Truthful write diagnostics: historyCompactWritesAttempted/WriteFailures are recorded only on the tiers where the recorder actually ran — pre-recorder fail-opens no longer claim a write, a write failure under the window records failedOpen write_failed with the counters, and over the window the exhaust path now carries a separate diagnostic reason so write_failed lands in the durable diagnostics (via the terminal LLM-call record) even though the terminal enum keeps summarizer_failed. A post-write materialization skip records the successful write it performed. The head-anchor gate also requires author='user', matching the checkpoint protocol. * fix(runtime): withhold the turn-ledger seam from child sessions A child run has no top-level prior context, so a mid-turn checkpoint built from its child-only ledger would claim to cover a session-scoped projection prefix and — through the session-global checkpoint cache/CAS, which compares coverage only by size — replace the parent's checkpoint and coverage_miss the parent projection. ensureChildActive no longer injects loadTurnRuntimeEvents (the backend requires the seam, so child mid-turn capacity compaction cannot arm), with the lineage-partitioning follow-up documented at the seam. A kernel test locks both sides: the parent backend reads its durable turn ledger through the seam; the child factory context has no seam and performs no read. * fix(runtime): move the mid-turn capacity verdict to a final-payload estimate owner Review round 3 (findings A, C, D): capacity estimation had no single owner — the trigger counted durable response chars, the engine issued a post-fold window verdict against the raw ledger span, and the verdict ran before the active tool-result prune could rescue the step. Now every prepareStep hook only shapes; one owner at the end of the pipeline measures the final (messages + active tool schema) payload and issues the pass/terminate verdict: - estimate = last step's real usage + SIGNED char/4 delta against the previous request's measured payload, so a rolling second compaction is judged by the real replacement projection (A), and same-turn load_tools schema growth counts like any other payload growth (D); - the verdict runs after pruning, and a trigger miss forces one bounded capacity re-entry before context_budget_exhausted (C); - the engine loses its post-fold window claim entirely; the hook refuses a materialized replacement that does not shrink the real payload (runaway summary) as a shaping decision, keeping the raw projection. * fix(runtime): replace the mid-turn durable watermark with a seq-ack durability boundary Review round 3 (finding B): the watermark waited only for the FINAL tool call/response pair, but a step's thinking/text completion events are enqueued later, at the pump's finish-step flush — under a slow consumer the ledger could satisfy the watermark while the step's already-emitted assistant text was still missing, and because the replacement projection replaces the whole message list, that text was silently dropped from the next request. The old 'a lagging read only shrinks the delta' claim was wrong and is corrected. No event-kind predicate can close this class of gap, so the wait now counts the event stream itself: the producer stamps a monotonic sequence at enqueue (AsyncEventQueue.pushedCount), the consumer acks after fully PROCESSING each event (the generator pull in drain() is the ack, so deliberately-unpersisted events can never deadlock it), and the capacity hook reads the ledger exactly once, after the pump has flushed every completed step boundary and consumedCount has caught pushedCount. Exits: boundary, abort, detached consumer, or read failure — the polling watermark predicate is deleted. * fix(runtime): make the capacity estimate baseline and checkpoint lifecycle truthful Review round 4 — four findings inside the verdict owner's implementation, architecture unchanged: - Estimate baseline is now the last request's INPUT tokens only: the signed payload delta already carries the step's freshly generated output and tool results, so an input+output baseline double-counted them (~500-token requests estimated as ~900, falsely exhausting rescuable turns). A usage sample without a positive input count is unusable, not zero — the estimate falls back to the whole-payload cold start instead of '0 + delta', so a huge request with a tiny delta can no longer slip past the window. - The head anchor in a replacement projection now renders through the same decoration owner (appendTurnTailPrompt) as the raw projection's user message, so the volatile turn tail (cwd, shell context, task state) is never silently dropped by compaction — or counted as shrinkage. - Lifecycle is validate → persist → apply: the replacement is materialized and shrink-checked BEFORE the checkpoint is recorded, so a rejected checkpoint never becomes the session's latest (replay applies checkpoints ahead of any high-water check and would have kept re-selecting it). Persistence still precedes application; validation failures attach no write counters because the recorder was never reached. - A non-shrinking fold terminates as summarizer_failed (the summarizer's output is unusable), not head_anchor_exceeds_capacity, keeping the replacement_not_smaller diagnostic reason. * fix(runtime): gate mid-turn checkpoints on replay admissibility and count the system prompt in the payload measure Review round 5 (2 P1 + 1 P3): - validate = materializable AND smaller AND replay-admissible: before persisting, reuse evaluateHistoryCompactCheckpointReplay (the same single gate the recovery path runs) so an accepted checkpoint can never be rejected at the next replay and re-inject the covered span - midTurnRequestPayloadChars now includes the system prompt chars sent through the separate system field; constant between adjacent requests so signed deltas are unchanged, but the cold-start whole-payload estimate no longer under-counts by the system prompt - fix stale priorUsageTokens doc: input-only, never input+output * docs(runtime): align stale mid-turn comments with the validate-before-persist lifecycle and full payload measure * fix(runtime): record accumulated completed-step usage when an aborted send has no total usage #972 made the terminal LLM-call record fail-closed on usage evidence, but an aborted send (mid-turn exhaust, user stop, stream error) never resolves the SDK totalUsage promise, so the record carrying the capacity verdict diagnostics was skipped entirely. Every COMPLETED step reports real usage at its finish-step boundary; accumulate those samples per send and fall back to the sum at terminal record time. No completed step means no evidence and the record is still skipped, preserving the #972 no-fabrication invariant. * fix(runtime): fail the aborted-send usage fallback closed on any unusable step sample An unusable completed-step sample (normalizeAiSdkUsage returns undefined, #972) made the accumulated sum a PARTIAL cost, and LlmCallRecord has no partial marker — downstream would read it as the whole call. Track sample completeness per send and use the sum only when every completed step reported usable usage; otherwise keep the fail-closed no-record behavior. The terminal outcome never depended on this record: stopReason and the exhausted detail are durable on the CompleteEvent, now asserted explicitly. Also rewrite the stale pre-#972 'missing tokens normalize to 0' comment at the capacity hook's usage read.
Astro-Han
added a commit
that referenced
this pull request
Jul 15, 2026
* fix(headless): harden real-provider smoke reliability (#972) * fix(headless): fail closed on missing usage * fix(headless): count model steps accurately * fix(headless): retry OpenCode apt setup * fix(headless): persist failures with missing usage * fix(runtime): preserve missing usage semantics * fix(headless): preserve unavailable cell metrics * fix(runtime): normalize AI SDK detail usage * fix(headless): count runtime steps per turn * fix(headless): preserve unknown TSV usage * test(headless): align continuation step counts * fix: preserve unmetered request telemetry * fix(storage): avoid atomic temp file collisions * test(desktop): clean up failed E2E launches * fix(storage): serialize settings initialization * fix(headless): stop when provider cost is unknown * fix(runtime): enforce per-turn step budgets * fix(headless): version persisted usage semantics * test(runtime): align model step budget contract * fix: preserve incomplete provider usage semantics * fix: fail closed on incomplete usage evidence * fix(headless): propagate unknown cost through optimization * fix: close usage evidence replay gaps * fix: close final cost observation gaps * fix: invalidate incomplete usage checkpoints * fix(storage): preserve legacy usage history * fix(headless): require usage evidence for A/B gates * fix: preserve usage across processes and views * fix: preserve authoritative usage aggregation * Revert "fix: preserve authoritative usage aggregation" This reverts commit 7320705. * Revert "fix: preserve usage across processes and views" This reverts commit 0dc3e76. * Revert "fix(storage): preserve legacy usage history" This reverts commit 4a2ab0c. * refactor: narrow usage reliability scope * refactor: restore headless smoke scope * fix(runtime): reject incomplete provider usage * fix(headless): exclude unmetered attested runs (cherry picked from commit 4b736dc) (reland after #1005 squash revert) * feat(runtime): mid-turn capacity compaction with a single final-payload verdict owner (#996) * feat(runtime): extend history compact checkpoint protocol to mid_turn phase Add a phase (pre_turn|mid_turn) and head-anchor reference to the V2 HistoryCompactCheckpoint so a checkpoint can fold a contiguous prefix that reaches into the current turn's completed steps while re-rendering the covered head anchor (the current turn's user message) verbatim on replay. Coverage stays a contiguous event prefix so the digest math is unchanged; pre_turn checkpoint ids stay byte-stable. projectHistoryCompactCheckpointReplay centralises the deterministic [block, head anchor, tail] projection. * feat(runtime): add pure mid-turn capacity measurement and safe-boundary engine Turn-agnostic, side-effect-free helpers for the active-turn context invariant: estimateNextRequestTokens anchors on the last step's real provider usage plus a char/4 tail delta (whole-projection char/4 on cold start); exceedsHighWater and exceedsContextWindow gate the two failure tiers; selectMidTurnSafeBoundary picks the largest covered prefix that ends on an immutable non-partial event and never straddles a tool call/result pair, reporting no_safe_completed_span otherwise. * feat(runtime): add mid-turn history compact policy surface (default off) HistoryCompactPolicy.midTurn carries enabled + reserveTokens + reserveTailEvents. MAKA_CONTEXT_HISTORY_COMPACT_MID_TURN opts in (default off, PR 3 sinks it on), reusing the shared MAKA_CONTEXT_HISTORY_COMPACT_RESERVE_TOKENS (16384) high-water reserve. A standalone revert leaves every surface's behavior unchanged. * feat(core): add context_budget_exhausted complete outcome A first-class CompleteEvent.stopReason for when the runtime cannot produce a provider-safe request even after mid-turn compaction, with a detail field (no_safe_completed_span | summarizer_failed | head_anchor_exceeds_capacity). failureClassFromCompleteStopReason maps it to a distinct failure class so the turn is recorded as an explicit budget outcome rather than a provider error. * feat(runtime): add mid-turn capacity compaction orchestration planMidTurnCapacityCompaction ties the measurement engine, safe-boundary selection, and the V2 checkpoint protocol into one deterministic decision: skip below the high-water; fold a safe completed prefix into a mid_turn checkpoint (re-rendering the head anchor verbatim and continuing with the preserved tail) via the injected summarizeHistoryCompact seam; roll forward from a matching previous checkpoint. Two failure tiers per the design: below the window a failure fails open, above the window it returns an explicit context_budget_exhausted outcome (no_safe_completed_span / summarizer_failed / head_anchor_exceeds_capacity). Recovery re-projection replay-validates against the same ledger prefix. * feat(core): add phase dimension to compaction decision diagnostics CompactionDecisionDiagnostic and the runtime CompactionDecision carry an optional phase ('pre_turn' | 'mid_turn'); absent on legacy data means pre_turn. Mid-turn capacity compaction records its trigger, replacement, fail-open, and exhausted decisions on the existing compactionDecisions channel with this dimension. * feat(runtime): replay mid_turn checkpoints against the full content projection A mid_turn checkpoint's coverage reaches into the compacted turn's own completed steps, so replay matches it against the full compactable-event projection before the turn-granular guards (tail selection would otherwise retain the covered span and miss the prefix, and a single giant turn must not be rejected as insufficient_turns). Replay stays the deterministic [block, verbatim head anchor, uncovered tail] and the pre_turn path is unchanged. Exports isHistoryCompactContentEvent as the shared predicate for the backend's mid-turn projection. * feat(runtime): wire mid-turn capacity compaction into the streaming backend Completes the issue #882 PR 1 invariant end to end. AiSdkFlow forwards ctx.branch and the persisted head anchor through BackendSendInput; the backend taps its send() queue to accumulate the current turn's content RuntimeEvents with exact ledger identity (same mapper, ids, and branch as the flow), tracks each finished step's real provider usage, and composes a mid-turn prepareStep hook (gated on historyCompact.midTurn.enabled, default off). Between steps it measures the next request as last-step usage plus a char/4 tool-result delta against contextWindow - reserve; over the high water it plans a safe-boundary fold, durably records the mid_turn checkpoint BEFORE replacing the projection, and continues the same turn on the materialized [compact block, verbatim head anchor, preserved tail] without re-executing completed tool calls. Failures under the window fail open with a mid_turn diagnostic; over the window the turn ends with the explicit complete stopReason context_budget_exhausted (no_safe_completed_span / summarizer_failed / head_anchor_exceeds_capacity), aborting the stream and handling AI SDK's graceful abort wind-down. Streaming integration tests cover trigger, persist-before-replace ordering, prompt replacement, branch recovery re-projection, all three exhausted details, fail-open, and the flow plumbing. * fix(runtime): close mid-turn compaction correctness gaps from external review Four verified findings, fixed at their owners: - Full-request re-estimate (F1): after folding, the plan re-estimates the complete next request (usage-anchored estimate minus the covered span's share plus the [block, anchor, tail] projection) instead of comparing only the replacement events to the window, so a huge fixed overhead with a tiny foldable span is exhausted (head_anchor_exceeds_capacity), and a replacement that would GROW past a window the raw request fits fails open (replacement_exceeds_window) rather than replacing. - Partial-free coverage (F5): the safe boundary retreats strictly before the first partial anywhere in the prefix (not just at the cut), and buildHistoryCompactCheckpoint rejects any coverage containing a partial snapshot — a digest over a replaced/deleted snapshot can never replay. - Anchor integrity fail-closed (F6): the builder requires the head anchor to be the covered turn's user event, and matchHistoryCompactCheckpointPrefix fails a mid_turn match as coverage_miss when the anchor reference is corrupted (uncovered id, wrong turn, or non-user role) instead of silently replaying without the user message. - Replay before the high-water skip (F3): an accepted mid_turn checkpoint is a correctness invariant, not a capacity optimization, so its replay match now precedes the below-high-water early return; recovery tests run on normal thresholds instead of a degenerate highWaterRatio. * fix(runtime): keep context_budget_exhausted detail in the durable terminal state The complete-event mapping dropped contextBudgetExhaustedDetail, so the persisted RuntimeEvent could not distinguish no_safe_completed_span / summarizer_failed / head_anchor_exceeds_capacity. completeRuntimeEvent now maps the full CompleteEvent and records the typed detail alongside stopReason/failureClass in the terminal stateDelta, locked by a flow-mapping round-trip test. * refactor: source mid-turn coverage from the durable run ledger, composed first Root fix for the two review P1s about the backend integration (F4, F2): one composed provider-visible projection, coverage only from events already confirmed on disk, and no mirrored state carrying the hard capacity invariant. Durable-read seam (F4): AgentRun exposes loadTurnRuntimeEvents() — it waits for every ledger write enqueued so far, then reads the store — and the kernel injects it into backends next to the checkpoint loader/recorder (BackendFactoryContext, cli and desktop factories pass it through). The mid-turn trigger reads the current turn's persisted RuntimeEvents as its coverage pool, so a checkpoint can never be recorded before its covered source events are durable (the crash window is gone) and byte-identity with recovery replay holds by construction, including under ctx.branch. Last-step real usage now comes synchronously from the SDK's own step results (the same numbers as the finish-step chunk), so the wall-clock waitForSteps synchronization is deleted along with TappedAsyncEventQueue, the SessionEvent mirror mapping, and the reconstructed InvocationContext; the BackendSendInput.branch plumb that existed only for that mirror is removed. A lagging ledger read only shrinks the tail delta of the usage-anchored estimate, and every failure-driven skip (ledger_read_failed, head_anchor_not_durable) records an explicit failedOpen decision — no silent skips. Composition order (F2): composePrepareStep now runs the capacity hook before activeToolResultPrune and semantic/active-full compaction, so prune re-archives large tool results in the rebuilt tail instead of having its placeholders undone, and on the exact step the capacity hook replaced, semantic/active-full compaction yields with a recorded mid_turn_capacity_precedence decision — one step never runs two summarizers. Integration tests now drive the durable-read fixture (consumer persists mapped events exactly like AgentRun before the seam serves them) and add the review-named combinations: ledger-read fail-open diagnostics, midTurn x activeToolResultPrune tail re-convergence, and midTurn x semanticCompact precedence. * fix(runtime): keep open tool calls out of coverage and stop double-counting the tail Two engine findings from the second external review: - Open tool span (N5): straddlesToolPair skipped spans missing one side, so with a zero tail reserve an unmatched function_call could be folded and its later response would arrive as an orphan. A call without a response is now an open span — any cut past the call is unsafe; a response without a call stays inert (its call precedes the pool). - Tail double-count (N2): the post-fold re-estimate added back the whole [block, anchor, tail] replacement although the usage-anchored estimate already contains the retained tail, misreporting rescuable turns as head_anchor_exceeds_capacity (repro: covered 505, tail ~400, estimate 700, window 500). The formula now adds back only the covered span's substitute [block, anchor]; the repro is a regression test. * fix(runtime): pin the mid-turn head anchor to the compacted turn A self-consistent anchor (role user, matching self-reported turnId) could resolve to ANOTHER covered turn's user event — e.g. a prior turn's prompt — and both build and match accepted it, so the replay silently dropped the real current prompt. The compacted turn is the coverage's through turn: builder and matcher now require anchor.turnId to equal it and the event to be an author='user' user event, failing closed (build error / coverage_miss) otherwise; locked by prior-turn-anchor tests on both paths. * fix(runtime): gate the mid-turn trigger on a durable tool-result watermark Root fix for the second recurrence of the sync seam (N1), plus truthful write diagnostics (N6): Durable watermark: the SDK's step results are the source of truth for which tool calls completed, so the trigger derives an explicit watermark from options.steps and, before measuring or selecting coverage, loops until the durable turn ledger contains the FINAL function_call/function_response for every one of them. Each iteration re-reads through the seam — which re-awaits the run's serialized write queue and re-checks store availability after the wait — and the only exits are the watermark itself, an abort (failedOpen ledger_wait_aborted), or a read failure (failedOpen ledger_read_failed): condition-driven, no wall clock. This closes both halves of the repro: a lagging ledger can no longer under-count the tail delta (letting an over-window request out) nor re-count the same results as a fresh delta at the next boundary. The review's consumer-scheduling perturbation is a real fixture mode now: the full integration suite runs twice (immediate + slow consumer, 11 tests each) and a negative control without the watermark fails 10 of the slow-mode tests. Truthful write diagnostics: historyCompactWritesAttempted/WriteFailures are recorded only on the tiers where the recorder actually ran — pre-recorder fail-opens no longer claim a write, a write failure under the window records failedOpen write_failed with the counters, and over the window the exhaust path now carries a separate diagnostic reason so write_failed lands in the durable diagnostics (via the terminal LLM-call record) even though the terminal enum keeps summarizer_failed. A post-write materialization skip records the successful write it performed. The head-anchor gate also requires author='user', matching the checkpoint protocol. * fix(runtime): withhold the turn-ledger seam from child sessions A child run has no top-level prior context, so a mid-turn checkpoint built from its child-only ledger would claim to cover a session-scoped projection prefix and — through the session-global checkpoint cache/CAS, which compares coverage only by size — replace the parent's checkpoint and coverage_miss the parent projection. ensureChildActive no longer injects loadTurnRuntimeEvents (the backend requires the seam, so child mid-turn capacity compaction cannot arm), with the lineage-partitioning follow-up documented at the seam. A kernel test locks both sides: the parent backend reads its durable turn ledger through the seam; the child factory context has no seam and performs no read. * fix(runtime): move the mid-turn capacity verdict to a final-payload estimate owner Review round 3 (findings A, C, D): capacity estimation had no single owner — the trigger counted durable response chars, the engine issued a post-fold window verdict against the raw ledger span, and the verdict ran before the active tool-result prune could rescue the step. Now every prepareStep hook only shapes; one owner at the end of the pipeline measures the final (messages + active tool schema) payload and issues the pass/terminate verdict: - estimate = last step's real usage + SIGNED char/4 delta against the previous request's measured payload, so a rolling second compaction is judged by the real replacement projection (A), and same-turn load_tools schema growth counts like any other payload growth (D); - the verdict runs after pruning, and a trigger miss forces one bounded capacity re-entry before context_budget_exhausted (C); - the engine loses its post-fold window claim entirely; the hook refuses a materialized replacement that does not shrink the real payload (runaway summary) as a shaping decision, keeping the raw projection. * fix(runtime): replace the mid-turn durable watermark with a seq-ack durability boundary Review round 3 (finding B): the watermark waited only for the FINAL tool call/response pair, but a step's thinking/text completion events are enqueued later, at the pump's finish-step flush — under a slow consumer the ledger could satisfy the watermark while the step's already-emitted assistant text was still missing, and because the replacement projection replaces the whole message list, that text was silently dropped from the next request. The old 'a lagging read only shrinks the delta' claim was wrong and is corrected. No event-kind predicate can close this class of gap, so the wait now counts the event stream itself: the producer stamps a monotonic sequence at enqueue (AsyncEventQueue.pushedCount), the consumer acks after fully PROCESSING each event (the generator pull in drain() is the ack, so deliberately-unpersisted events can never deadlock it), and the capacity hook reads the ledger exactly once, after the pump has flushed every completed step boundary and consumedCount has caught pushedCount. Exits: boundary, abort, detached consumer, or read failure — the polling watermark predicate is deleted. * fix(runtime): make the capacity estimate baseline and checkpoint lifecycle truthful Review round 4 — four findings inside the verdict owner's implementation, architecture unchanged: - Estimate baseline is now the last request's INPUT tokens only: the signed payload delta already carries the step's freshly generated output and tool results, so an input+output baseline double-counted them (~500-token requests estimated as ~900, falsely exhausting rescuable turns). A usage sample without a positive input count is unusable, not zero — the estimate falls back to the whole-payload cold start instead of '0 + delta', so a huge request with a tiny delta can no longer slip past the window. - The head anchor in a replacement projection now renders through the same decoration owner (appendTurnTailPrompt) as the raw projection's user message, so the volatile turn tail (cwd, shell context, task state) is never silently dropped by compaction — or counted as shrinkage. - Lifecycle is validate → persist → apply: the replacement is materialized and shrink-checked BEFORE the checkpoint is recorded, so a rejected checkpoint never becomes the session's latest (replay applies checkpoints ahead of any high-water check and would have kept re-selecting it). Persistence still precedes application; validation failures attach no write counters because the recorder was never reached. - A non-shrinking fold terminates as summarizer_failed (the summarizer's output is unusable), not head_anchor_exceeds_capacity, keeping the replacement_not_smaller diagnostic reason. * fix(runtime): gate mid-turn checkpoints on replay admissibility and count the system prompt in the payload measure Review round 5 (2 P1 + 1 P3): - validate = materializable AND smaller AND replay-admissible: before persisting, reuse evaluateHistoryCompactCheckpointReplay (the same single gate the recovery path runs) so an accepted checkpoint can never be rejected at the next replay and re-inject the covered span - midTurnRequestPayloadChars now includes the system prompt chars sent through the separate system field; constant between adjacent requests so signed deltas are unchanged, but the cold-start whole-payload estimate no longer under-counts by the system prompt - fix stale priorUsageTokens doc: input-only, never input+output * docs(runtime): align stale mid-turn comments with the validate-before-persist lifecycle and full payload measure * fix(runtime): record accumulated completed-step usage when an aborted send has no total usage #972 made the terminal LLM-call record fail-closed on usage evidence, but an aborted send (mid-turn exhaust, user stop, stream error) never resolves the SDK totalUsage promise, so the record carrying the capacity verdict diagnostics was skipped entirely. Every COMPLETED step reports real usage at its finish-step boundary; accumulate those samples per send and fall back to the sum at terminal record time. No completed step means no evidence and the record is still skipped, preserving the #972 no-fabrication invariant. * fix(runtime): fail the aborted-send usage fallback closed on any unusable step sample An unusable completed-step sample (normalizeAiSdkUsage returns undefined, #972) made the accumulated sum a PARTIAL cost, and LlmCallRecord has no partial marker — downstream would read it as the whole call. Track sample completeness per send and use the sum only when every completed step reported usable usage; otherwise keep the fail-closed no-record behavior. The terminal outcome never depended on this record: stopReason and the exhausted detail are durable on the CompleteEvent, now asserted explicitly. Also rewrite the stale pre-#972 'missing tokens normalize to 0' comment at the capacity hook's usage read. (cherry picked from commit 8ef9373) (reland after #1005 squash revert) * fix(ui): restore quiet composer picker triggers (#999) (cherry picked from commit ecf515d) (reland after #1005 squash revert) * fix(ui): keep in-flight live turn armed when persisted history covers all steps (#1000) Symptom: the desktop composer's "in progress" indicator flickers off during a running turn. At every step-to-step lull, when all tool/thinking evidence is already covered by the persisted transcript, the busy state drops to idle until the next event recreates the projection. Cause: reconcileTerminalLiveTurn deleted the whole live-turn projection (returning undefined) whenever the filtered steps array became empty, even for a NON-terminal projection. app-shell calls it on every messages/activeLiveTurn change mid-turn, so the projection vanished and turnInFlight (projection exists && !terminal) went false. Fix: an empty result only deletes the projection when current.terminal, mirroring the existing precedent in settleLiveTurnStep. A non-terminal projection survives as { ...current, steps: [] } with its arm preserved. (cherry picked from commit 8153519) (reland after #1005 squash revert)
Astro-Han
added a commit
that referenced
this pull request
Jul 15, 2026
…1017) * feat(runtime): classify provider context-length overflow errors Add an exclusion-first ContextLength bucket to classifyError, matching the raw provider error message against a ported overflow-pattern table (the providers Maka ships) while excluding throttling/quota wording that merely mentions tokens. This is the reactive-recovery trigger for issue #882 PR 2; the status-based classes still win, so an explicit 429/5xx never lands here. * feat(runtime): reactive context-overflow compact-and-retry recovery Second line of defense for issue #882 (PR 2). A request-level provider failure surfaces as a fullStream error chunk — both when the transport throws (finishReason then rejects with NoOutputGeneratedError) and when it streams an error part. The old pump caught the rejected finishReason as `stop` and emitted a fabricated end_turn completion with success telemetry. The pump now captures that error chunk and, at most once per send, folds the durable turn ledger and resends on a context-length overflow — a pi-style single compact-and-retry latch reusing the PR 1 mid-turn compaction machinery (planMidTurnCapacityCompaction, the checkpoint protocol, the anchor-tail decoration, and the replay-admissibility gate), tagged with reason 'overflow'. When recovery is impossible or spent — a non-context-length error, no mid-turn seam, no safe completed span, or a second overflow — the real provider error becomes the terminal outcome, never a fabricated success and never a synthesized context_budget_exhausted (the provider, not the runtime, rejected). The compaction core is extracted into computeMidTurnCompactionReplacement, shared by the proactive prepareStep hook and the reactive path so there is one fold implementation, one acceptance standard, and one persist order. errorReasonFromClass maps ContextLength to a context_overflow error reason. * fix(runtime): classify overflow on original-error fields and tighten fallback patterns Review findings P2-1/P2-2: match the overflow signatures against the composite original-error text (name + code + status + message) so a structured code like OpenAI's context_length_exceeded classifies even with a generic HTTP message, and constrain the two over-broad fallbacks — the Copilot form now requires a token-count subject and the generic 'too many tokens' an input/prompt/context subject — so a file-size limit or a max_tokens parameter error never triggers a persisted compaction retry. * fix(runtime): give the overflow retry single authoritative owners for baseline, usage, and step budget Review round-1 P1 findings, all owner-boundary errors in the retry loop: - P1-1: the shrink baseline for a reactive fold is now the verdict owner's per-request payload measure (state.lastRequestPayloadChars) — the request the provider actually rejected — instead of the attempt-initial messages, which undercount by every same-turn tool step and refused folds that genuinely shrank the real request (replacement_not_smaller, 0 retry). - P1-2: send-level usage is owned by the cross-attempt per-step accumulator. After a retry the terminal record carries BOTH attempts' completed steps; the last attempt's totalUsage is authoritative only for a single-attempt send, and an unusable sample in any attempt fails the whole record closed (#972) — a later attempt's valid totalUsage cannot wash it back. - P1-3: the step limit is a send-level cap counted by runtimeSteps across attempts. startStream takes a per-call maxSteps override and a retry gets only the remaining budget; with the budget spent the overflow is terminal. The extraction contract locks the adapter-owned default + per-call override. * fix(runtime): require an input subject for the token-limit overflow fallback Review round-2 P2-C: the bare /token limit exceeded/ fallback also matched OUTPUT caps ('Output token limit exceeded', 'Maximum output token limit exceeded'), which history compaction cannot fix, so they triggered a pointless persisted compaction retry. The fallback now carries the same input/prompt/context subject constraint as the round-1 'too many tokens' fix; the input-side form keeps classifying. * fix(runtime): keep send-level per-step state authoritative across overflow retries Review round-2 P1-A + P1-B: a reactive retry re-invokes streamText, which resets two attempt-scoped views that send-level state was derived from. - P1-A: the SDK numbers prepareStep steps per streamText call, while flushedSteps / replacedStepNumber / lastShapeFailure / the semantic-compact yield all keep send-level state. An attempt-local durability bound already satisfied by a PREVIOUS attempt's flushed boundary let a post-retry capacity compaction read the ledger before the retry step's streamed assistant text was durable — and the replacement projection then dropped it from both the covered span and the tail. One translation point in send() now rebases each attempt's local step numbers onto the send-global clock (completed steps at attempt start) before any hook sees them. - P1-B: active tools were re-derived per streamText call from the ledger seed plus that call's own steps, so a retry (fresh call, empty steps) silently revoked a group loaded before the overflow — the gated tool vanished from the provider request and the execute boundary rejected it. The availability owner now holds a send-scoped monotonic activation set: groups accumulate from every attempt's steps and never unload within a send. Cross-turn behavior is unchanged (rebuilt per send from the ledger seed). Both repros ride the new fixture levers: a slow appendMessage that parks the pump inside flushStep while the consumer has drained the queue (P1-A), and a gated tool group loaded before the overflow (P1-B). * fix(runtime): give every prepareStep hook a send-global steps view across overflow retries A reactive overflow retry starts a fresh streamText call, so the SDK's per-call `steps` restarted empty mid-send. The active tool-result prune derives its eligible tool-call IDs from `steps`; an empty view revoked the prune on the retry request and the ledger-rebuilt recovery projection resurrected archived raw tool results (review round-3 P1) — the third instance of attempt-local SDK state consumed as send-level state, after the step clock (round-2 P1-A) and tool activations (round-2 P1-B). Converge all of them into the existing single translation point: sendScopedPrepareStep archives each dead attempt's observed steps and hands every hook `[...completedAttemptSteps, ...options.steps]` alongside the already-rebased send-global stepNumber. Consumers stay untouched and any future steps consumer is send-correct by construction; steps folded into a checkpoint remain in the view because ID-based consumers only act on messages present in the projection. This also lets tool-availability drop its round-2 bespoke monotonic activation set and return to deriving activations statelessly from the (now send-global) steps; its round-2 regression test stays green. Two pairing consequences of the union view, each at its single owner: the capacity hook only anchors its next-request estimate on the last step's usage when the verdict owner has a payload baseline for the same request, and a successful overflow recovery resets that baseline so the retry starts from the whole-payload cold-start estimate instead of a stale pairing against the rejected request. * fix(runtime): require an input subject for the Copilot token-count overflow pattern Review round-4 P1: /token count of N exceeds the limit of M/ also matches output and completion caps ('output token count of 8192 exceeds the limit of 4096'), which history compaction cannot fix — the misclassification triggered a persisted compaction and a doomed retry. The pattern now requires the same input subject as the sibling generic fallbacks; the real Copilot form ('prompt token count of X exceeds the limit of Y') still classifies, with output/completion negatives locked. * fix(runtime): exclude explicit output caps at the overflow classifier's exclusion owner Review round-5 P1: the input-subject constraints can be bypassed by a generic prefix — 'Invalid request: output token count of 8192 exceeds the limit of 4096' classified as ContextLength because 'request' satisfied the subject alternation without modifying the token count. The invariant is categorical, not positional: history compaction can only fix INPUT overflow, so explicit output/completion/max_tokens cap wording is now excluded at the exclusion-first owner regardless of surrounding words. Exclusions stay adjacency-tight; OpenAI's classic input-overflow message (mentioning 'the completion' and 'max_tokens' amounts) keeps classifying, locked by new positives alongside the prefixed negatives. * fix(runtime): tier the overflow classifier — definitive signals, then vetoes, then fuzzy subjects Review round-6 P1s: blocklisting output-cap word orders can never converge ('completion has too many tokens', 'max_tokens token limit exceeded' bypassed the previous exclusions), and the unconditional noun-phrase exclusion vetoed a genuine input overflow whose message breaks usage down into prompt AND completion token counts alongside a context_length_exceeded code. The classifier is now tiered by evidence strength instead of patched by wording: (1) definitive provider signals — including structured codes — win unconditionally; (2) throttling/quota wording and complete output-cap RELATIONS (subject and predicate, not noun phrases) veto; (3) ambiguous token-limit wording counts only with an input-like subject, and 'request' is dropped from that subject list because a generic 'Invalid request:' prefix carries no input semantics. Both round-6 bypasses and the over-exclusion are locked as tests. * fix(runtime): classify overflow errors by evidence strength, not pattern patches Three rounds of word-table patches to the overflow classifier kept being pierced (review round 7, four P1s) because the design let weak evidence outrank strong evidence: - A real AI SDK APICallError carries the provider's structured error JSON in `data` (or raw in `responseBody`); there is no top-level `.code`, so `data.error.code = 'context_length_exceeded'` with a generic 'Bad Request' message never classified. - `text.includes('rate')` ran before overflow detection at equal strength with an explicit 429, so 'Failed to generate response: context_length_exceeded' became RateLimit ('generate' contains 'rate'). - The output-cap veto only knew the subject-before-predicate voice, so the passive 'too many tokens were requested for the completion' slipped through as ContextLength. - The free-text 'definitive' tier was unconditional, so a bare capacity statement ('maximum context length is N tokens') quoted inside a ThrottlingException overrode the throttle/quota veto. Redesign classifyError at its owner by descending evidence strength: abort, then explicit numeric statuses/codes from fields, then the structured provider code (new extraction walking data.error.code/.type and the same paths in responseBody JSON — the exact shapes createJsonErrorResponseHandler produces), then free-text overflow relations, and only last the rate/auth/timeout/network substring heuristics. Free text collapses from three tiers to two: vetoes first (throttle/quota + output-cap relations in both voices plus the count-of form), then all positive overflow relations, none unconditional — only a structured provider code is. Structured-code positives in tests now use the real APICallError shape instead of the invented top-level code. * fix(runtime): normalize the classifier's real input domain before ranking evidence The classifier's callers hand it whatever the AI SDK surfaced, and that input domain is not just Error instances (review round 8, 4 P1s): - In-stream error parts carry the provider's PARSED error value: OpenAI Chat emits the inner {message, type?, code?} object, OpenAI Responses the whole {type:'error', error:{type, code, message}} chunk, Anthropic the inner {type, message} object, and openai-compatible a bare message string. The instanceof Error gate classified all of them Other, so a genuine in-stream overflow could never reach reactive recovery. - Generic 5xx ranked above specific overflow evidence, so LiteLLM-style 503 wrappers around a provider overflow became ProviderUnavailable. - A bare 413 with no body (Cerebras) carried no text signal at all, yet HTTP 413 is itself input-side evidence. - The output-cap veto missed the embedded-role permutation ('Too many completion tokens were requested…'), letting a trailing capacity statement classify as overflow. Introduce a single evidence normalizer at the classifier owner: it maps Error | string | plain object into {composite text, statusCode, code, structuredCodes} using the exact shapes the providers produce (data / responseBody for APICallError, code/type on the value or its error wrapper for stream parts). classifyError then ranks normalized evidence strictly by strength: abort > 402 > 429 > 401/403 > structured overflow code > bare 413 > vetoable free-text overflow > generic 5xx > weak word heuristics. The weak rate heuristic becomes word-shaped so 'generate' / 'separate' are no longer rate limits (P2), the output-cap veto gains the embedded-role and role-tokens-exceed permutations (P1-4), and the responseBody test fixture now uses a body the OpenAI errorSchema genuinely rejects, proving the fallback (P3). An end-to-end reactive test drives recovery from a plain-object in-stream error part with the production finish-after-error stream shape. * fix(runtime): feed the raw responseBody into text evidence and correct the in-stream fixture shapes Two review round-9 findings: - P2: when the provider error JSON fails the schema, the real createJsonErrorResponseHandler degrades message to the statusText and keeps the provider wording ONLY in responseBody. The normalizer read the body just for structured codes, so an OpenAI-compatible {error: string} overflow ('Your input exceeds the context window…') classified as AI_APICallError. The raw body now joins the composite text so positives AND vetoes run over the full evidence; the test constructs the error through the real handler and also locks an output-cap body against misclassification. - P3: the round-8 e2e fixture mixed provider families (Responses-shaped error value with a Chat-shaped 'error' finish trailer), a stream that no locked provider produces. The Chat shape (inner error object + finishReason 'error' trailer) is now the main test and a Responses variant (whole error chunk + finishReason 'other' trailer, which the isErrorChunk branch never reassigns) locks recovery against per-family trailer drift.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Harden real-provider head-to-head benchmark runs against transient infrastructure failures and misleading accounting:
apt-get updateand curl installation with bounded backoffmissing_token_usageThis keeps transient setup failures retryable, preserves the original failure evidence, and prevents missing usage or stream chunks from distorting benchmark economics.
Verification
env HOME=/tmp/maka-headless-test-home npm --workspace @maka/headless testnpm run typechecknpm run check:stalepython3 -m py_compile packages/headless/harbor/opencode_agent.pyapt-get updateand curl installation successfully through the retry-wrapped command