fix(ui): keep in-flight live turn armed when persisted history covers all steps - #1000
Merged
Conversation
… all steps
Symptom: the desktop composer's "in progress" indicator flickers off during
a running turn. At every step-to-step lull, when all tool/thinking evidence is
already covered by the persisted transcript, the busy state drops to idle until
the next event recreates the projection.
Cause: reconcileTerminalLiveTurn deleted the whole live-turn projection
(returning undefined) whenever the filtered steps array became empty, even for
a NON-terminal projection. app-shell calls it on every messages/activeLiveTurn
change mid-turn, so the projection vanished and turnInFlight (projection exists
&& !terminal) went false.
Fix: an empty result only deletes the projection when current.terminal, mirroring
the existing precedent in settleLiveTurnStep. A non-terminal projection survives
as { ...current, steps: [] } with its arm preserved.
Astro-Han
added a commit
that referenced
this pull request
Jul 15, 2026
* fix(headless): harden real-provider smoke reliability (#972) * fix(headless): fail closed on missing usage * fix(headless): count model steps accurately * fix(headless): retry OpenCode apt setup * fix(headless): persist failures with missing usage * fix(runtime): preserve missing usage semantics * fix(headless): preserve unavailable cell metrics * fix(runtime): normalize AI SDK detail usage * fix(headless): count runtime steps per turn * fix(headless): preserve unknown TSV usage * test(headless): align continuation step counts * fix: preserve unmetered request telemetry * fix(storage): avoid atomic temp file collisions * test(desktop): clean up failed E2E launches * fix(storage): serialize settings initialization * fix(headless): stop when provider cost is unknown * fix(runtime): enforce per-turn step budgets * fix(headless): version persisted usage semantics * test(runtime): align model step budget contract * fix: preserve incomplete provider usage semantics * fix: fail closed on incomplete usage evidence * fix(headless): propagate unknown cost through optimization * fix: close usage evidence replay gaps * fix: close final cost observation gaps * fix: invalidate incomplete usage checkpoints * fix(storage): preserve legacy usage history * fix(headless): require usage evidence for A/B gates * fix: preserve usage across processes and views * fix: preserve authoritative usage aggregation * Revert "fix: preserve authoritative usage aggregation" This reverts commit 7320705. * Revert "fix: preserve usage across processes and views" This reverts commit 0dc3e76. * Revert "fix(storage): preserve legacy usage history" This reverts commit 4a2ab0c. * refactor: narrow usage reliability scope * refactor: restore headless smoke scope * fix(runtime): reject incomplete provider usage * fix(headless): exclude unmetered attested runs (cherry picked from commit 4b736dc) (reland after #1005 squash revert) * feat(runtime): mid-turn capacity compaction with a single final-payload verdict owner (#996) * feat(runtime): extend history compact checkpoint protocol to mid_turn phase Add a phase (pre_turn|mid_turn) and head-anchor reference to the V2 HistoryCompactCheckpoint so a checkpoint can fold a contiguous prefix that reaches into the current turn's completed steps while re-rendering the covered head anchor (the current turn's user message) verbatim on replay. Coverage stays a contiguous event prefix so the digest math is unchanged; pre_turn checkpoint ids stay byte-stable. projectHistoryCompactCheckpointReplay centralises the deterministic [block, head anchor, tail] projection. * feat(runtime): add pure mid-turn capacity measurement and safe-boundary engine Turn-agnostic, side-effect-free helpers for the active-turn context invariant: estimateNextRequestTokens anchors on the last step's real provider usage plus a char/4 tail delta (whole-projection char/4 on cold start); exceedsHighWater and exceedsContextWindow gate the two failure tiers; selectMidTurnSafeBoundary picks the largest covered prefix that ends on an immutable non-partial event and never straddles a tool call/result pair, reporting no_safe_completed_span otherwise. * feat(runtime): add mid-turn history compact policy surface (default off) HistoryCompactPolicy.midTurn carries enabled + reserveTokens + reserveTailEvents. MAKA_CONTEXT_HISTORY_COMPACT_MID_TURN opts in (default off, PR 3 sinks it on), reusing the shared MAKA_CONTEXT_HISTORY_COMPACT_RESERVE_TOKENS (16384) high-water reserve. A standalone revert leaves every surface's behavior unchanged. * feat(core): add context_budget_exhausted complete outcome A first-class CompleteEvent.stopReason for when the runtime cannot produce a provider-safe request even after mid-turn compaction, with a detail field (no_safe_completed_span | summarizer_failed | head_anchor_exceeds_capacity). failureClassFromCompleteStopReason maps it to a distinct failure class so the turn is recorded as an explicit budget outcome rather than a provider error. * feat(runtime): add mid-turn capacity compaction orchestration planMidTurnCapacityCompaction ties the measurement engine, safe-boundary selection, and the V2 checkpoint protocol into one deterministic decision: skip below the high-water; fold a safe completed prefix into a mid_turn checkpoint (re-rendering the head anchor verbatim and continuing with the preserved tail) via the injected summarizeHistoryCompact seam; roll forward from a matching previous checkpoint. Two failure tiers per the design: below the window a failure fails open, above the window it returns an explicit context_budget_exhausted outcome (no_safe_completed_span / summarizer_failed / head_anchor_exceeds_capacity). Recovery re-projection replay-validates against the same ledger prefix. * feat(core): add phase dimension to compaction decision diagnostics CompactionDecisionDiagnostic and the runtime CompactionDecision carry an optional phase ('pre_turn' | 'mid_turn'); absent on legacy data means pre_turn. Mid-turn capacity compaction records its trigger, replacement, fail-open, and exhausted decisions on the existing compactionDecisions channel with this dimension. * feat(runtime): replay mid_turn checkpoints against the full content projection A mid_turn checkpoint's coverage reaches into the compacted turn's own completed steps, so replay matches it against the full compactable-event projection before the turn-granular guards (tail selection would otherwise retain the covered span and miss the prefix, and a single giant turn must not be rejected as insufficient_turns). Replay stays the deterministic [block, verbatim head anchor, uncovered tail] and the pre_turn path is unchanged. Exports isHistoryCompactContentEvent as the shared predicate for the backend's mid-turn projection. * feat(runtime): wire mid-turn capacity compaction into the streaming backend Completes the issue #882 PR 1 invariant end to end. AiSdkFlow forwards ctx.branch and the persisted head anchor through BackendSendInput; the backend taps its send() queue to accumulate the current turn's content RuntimeEvents with exact ledger identity (same mapper, ids, and branch as the flow), tracks each finished step's real provider usage, and composes a mid-turn prepareStep hook (gated on historyCompact.midTurn.enabled, default off). Between steps it measures the next request as last-step usage plus a char/4 tool-result delta against contextWindow - reserve; over the high water it plans a safe-boundary fold, durably records the mid_turn checkpoint BEFORE replacing the projection, and continues the same turn on the materialized [compact block, verbatim head anchor, preserved tail] without re-executing completed tool calls. Failures under the window fail open with a mid_turn diagnostic; over the window the turn ends with the explicit complete stopReason context_budget_exhausted (no_safe_completed_span / summarizer_failed / head_anchor_exceeds_capacity), aborting the stream and handling AI SDK's graceful abort wind-down. Streaming integration tests cover trigger, persist-before-replace ordering, prompt replacement, branch recovery re-projection, all three exhausted details, fail-open, and the flow plumbing. * fix(runtime): close mid-turn compaction correctness gaps from external review Four verified findings, fixed at their owners: - Full-request re-estimate (F1): after folding, the plan re-estimates the complete next request (usage-anchored estimate minus the covered span's share plus the [block, anchor, tail] projection) instead of comparing only the replacement events to the window, so a huge fixed overhead with a tiny foldable span is exhausted (head_anchor_exceeds_capacity), and a replacement that would GROW past a window the raw request fits fails open (replacement_exceeds_window) rather than replacing. - Partial-free coverage (F5): the safe boundary retreats strictly before the first partial anywhere in the prefix (not just at the cut), and buildHistoryCompactCheckpoint rejects any coverage containing a partial snapshot — a digest over a replaced/deleted snapshot can never replay. - Anchor integrity fail-closed (F6): the builder requires the head anchor to be the covered turn's user event, and matchHistoryCompactCheckpointPrefix fails a mid_turn match as coverage_miss when the anchor reference is corrupted (uncovered id, wrong turn, or non-user role) instead of silently replaying without the user message. - Replay before the high-water skip (F3): an accepted mid_turn checkpoint is a correctness invariant, not a capacity optimization, so its replay match now precedes the below-high-water early return; recovery tests run on normal thresholds instead of a degenerate highWaterRatio. * fix(runtime): keep context_budget_exhausted detail in the durable terminal state The complete-event mapping dropped contextBudgetExhaustedDetail, so the persisted RuntimeEvent could not distinguish no_safe_completed_span / summarizer_failed / head_anchor_exceeds_capacity. completeRuntimeEvent now maps the full CompleteEvent and records the typed detail alongside stopReason/failureClass in the terminal stateDelta, locked by a flow-mapping round-trip test. * refactor: source mid-turn coverage from the durable run ledger, composed first Root fix for the two review P1s about the backend integration (F4, F2): one composed provider-visible projection, coverage only from events already confirmed on disk, and no mirrored state carrying the hard capacity invariant. Durable-read seam (F4): AgentRun exposes loadTurnRuntimeEvents() — it waits for every ledger write enqueued so far, then reads the store — and the kernel injects it into backends next to the checkpoint loader/recorder (BackendFactoryContext, cli and desktop factories pass it through). The mid-turn trigger reads the current turn's persisted RuntimeEvents as its coverage pool, so a checkpoint can never be recorded before its covered source events are durable (the crash window is gone) and byte-identity with recovery replay holds by construction, including under ctx.branch. Last-step real usage now comes synchronously from the SDK's own step results (the same numbers as the finish-step chunk), so the wall-clock waitForSteps synchronization is deleted along with TappedAsyncEventQueue, the SessionEvent mirror mapping, and the reconstructed InvocationContext; the BackendSendInput.branch plumb that existed only for that mirror is removed. A lagging ledger read only shrinks the tail delta of the usage-anchored estimate, and every failure-driven skip (ledger_read_failed, head_anchor_not_durable) records an explicit failedOpen decision — no silent skips. Composition order (F2): composePrepareStep now runs the capacity hook before activeToolResultPrune and semantic/active-full compaction, so prune re-archives large tool results in the rebuilt tail instead of having its placeholders undone, and on the exact step the capacity hook replaced, semantic/active-full compaction yields with a recorded mid_turn_capacity_precedence decision — one step never runs two summarizers. Integration tests now drive the durable-read fixture (consumer persists mapped events exactly like AgentRun before the seam serves them) and add the review-named combinations: ledger-read fail-open diagnostics, midTurn x activeToolResultPrune tail re-convergence, and midTurn x semanticCompact precedence. * fix(runtime): keep open tool calls out of coverage and stop double-counting the tail Two engine findings from the second external review: - Open tool span (N5): straddlesToolPair skipped spans missing one side, so with a zero tail reserve an unmatched function_call could be folded and its later response would arrive as an orphan. A call without a response is now an open span — any cut past the call is unsafe; a response without a call stays inert (its call precedes the pool). - Tail double-count (N2): the post-fold re-estimate added back the whole [block, anchor, tail] replacement although the usage-anchored estimate already contains the retained tail, misreporting rescuable turns as head_anchor_exceeds_capacity (repro: covered 505, tail ~400, estimate 700, window 500). The formula now adds back only the covered span's substitute [block, anchor]; the repro is a regression test. * fix(runtime): pin the mid-turn head anchor to the compacted turn A self-consistent anchor (role user, matching self-reported turnId) could resolve to ANOTHER covered turn's user event — e.g. a prior turn's prompt — and both build and match accepted it, so the replay silently dropped the real current prompt. The compacted turn is the coverage's through turn: builder and matcher now require anchor.turnId to equal it and the event to be an author='user' user event, failing closed (build error / coverage_miss) otherwise; locked by prior-turn-anchor tests on both paths. * fix(runtime): gate the mid-turn trigger on a durable tool-result watermark Root fix for the second recurrence of the sync seam (N1), plus truthful write diagnostics (N6): Durable watermark: the SDK's step results are the source of truth for which tool calls completed, so the trigger derives an explicit watermark from options.steps and, before measuring or selecting coverage, loops until the durable turn ledger contains the FINAL function_call/function_response for every one of them. Each iteration re-reads through the seam — which re-awaits the run's serialized write queue and re-checks store availability after the wait — and the only exits are the watermark itself, an abort (failedOpen ledger_wait_aborted), or a read failure (failedOpen ledger_read_failed): condition-driven, no wall clock. This closes both halves of the repro: a lagging ledger can no longer under-count the tail delta (letting an over-window request out) nor re-count the same results as a fresh delta at the next boundary. The review's consumer-scheduling perturbation is a real fixture mode now: the full integration suite runs twice (immediate + slow consumer, 11 tests each) and a negative control without the watermark fails 10 of the slow-mode tests. Truthful write diagnostics: historyCompactWritesAttempted/WriteFailures are recorded only on the tiers where the recorder actually ran — pre-recorder fail-opens no longer claim a write, a write failure under the window records failedOpen write_failed with the counters, and over the window the exhaust path now carries a separate diagnostic reason so write_failed lands in the durable diagnostics (via the terminal LLM-call record) even though the terminal enum keeps summarizer_failed. A post-write materialization skip records the successful write it performed. The head-anchor gate also requires author='user', matching the checkpoint protocol. * fix(runtime): withhold the turn-ledger seam from child sessions A child run has no top-level prior context, so a mid-turn checkpoint built from its child-only ledger would claim to cover a session-scoped projection prefix and — through the session-global checkpoint cache/CAS, which compares coverage only by size — replace the parent's checkpoint and coverage_miss the parent projection. ensureChildActive no longer injects loadTurnRuntimeEvents (the backend requires the seam, so child mid-turn capacity compaction cannot arm), with the lineage-partitioning follow-up documented at the seam. A kernel test locks both sides: the parent backend reads its durable turn ledger through the seam; the child factory context has no seam and performs no read. * fix(runtime): move the mid-turn capacity verdict to a final-payload estimate owner Review round 3 (findings A, C, D): capacity estimation had no single owner — the trigger counted durable response chars, the engine issued a post-fold window verdict against the raw ledger span, and the verdict ran before the active tool-result prune could rescue the step. Now every prepareStep hook only shapes; one owner at the end of the pipeline measures the final (messages + active tool schema) payload and issues the pass/terminate verdict: - estimate = last step's real usage + SIGNED char/4 delta against the previous request's measured payload, so a rolling second compaction is judged by the real replacement projection (A), and same-turn load_tools schema growth counts like any other payload growth (D); - the verdict runs after pruning, and a trigger miss forces one bounded capacity re-entry before context_budget_exhausted (C); - the engine loses its post-fold window claim entirely; the hook refuses a materialized replacement that does not shrink the real payload (runaway summary) as a shaping decision, keeping the raw projection. * fix(runtime): replace the mid-turn durable watermark with a seq-ack durability boundary Review round 3 (finding B): the watermark waited only for the FINAL tool call/response pair, but a step's thinking/text completion events are enqueued later, at the pump's finish-step flush — under a slow consumer the ledger could satisfy the watermark while the step's already-emitted assistant text was still missing, and because the replacement projection replaces the whole message list, that text was silently dropped from the next request. The old 'a lagging read only shrinks the delta' claim was wrong and is corrected. No event-kind predicate can close this class of gap, so the wait now counts the event stream itself: the producer stamps a monotonic sequence at enqueue (AsyncEventQueue.pushedCount), the consumer acks after fully PROCESSING each event (the generator pull in drain() is the ack, so deliberately-unpersisted events can never deadlock it), and the capacity hook reads the ledger exactly once, after the pump has flushed every completed step boundary and consumedCount has caught pushedCount. Exits: boundary, abort, detached consumer, or read failure — the polling watermark predicate is deleted. * fix(runtime): make the capacity estimate baseline and checkpoint lifecycle truthful Review round 4 — four findings inside the verdict owner's implementation, architecture unchanged: - Estimate baseline is now the last request's INPUT tokens only: the signed payload delta already carries the step's freshly generated output and tool results, so an input+output baseline double-counted them (~500-token requests estimated as ~900, falsely exhausting rescuable turns). A usage sample without a positive input count is unusable, not zero — the estimate falls back to the whole-payload cold start instead of '0 + delta', so a huge request with a tiny delta can no longer slip past the window. - The head anchor in a replacement projection now renders through the same decoration owner (appendTurnTailPrompt) as the raw projection's user message, so the volatile turn tail (cwd, shell context, task state) is never silently dropped by compaction — or counted as shrinkage. - Lifecycle is validate → persist → apply: the replacement is materialized and shrink-checked BEFORE the checkpoint is recorded, so a rejected checkpoint never becomes the session's latest (replay applies checkpoints ahead of any high-water check and would have kept re-selecting it). Persistence still precedes application; validation failures attach no write counters because the recorder was never reached. - A non-shrinking fold terminates as summarizer_failed (the summarizer's output is unusable), not head_anchor_exceeds_capacity, keeping the replacement_not_smaller diagnostic reason. * fix(runtime): gate mid-turn checkpoints on replay admissibility and count the system prompt in the payload measure Review round 5 (2 P1 + 1 P3): - validate = materializable AND smaller AND replay-admissible: before persisting, reuse evaluateHistoryCompactCheckpointReplay (the same single gate the recovery path runs) so an accepted checkpoint can never be rejected at the next replay and re-inject the covered span - midTurnRequestPayloadChars now includes the system prompt chars sent through the separate system field; constant between adjacent requests so signed deltas are unchanged, but the cold-start whole-payload estimate no longer under-counts by the system prompt - fix stale priorUsageTokens doc: input-only, never input+output * docs(runtime): align stale mid-turn comments with the validate-before-persist lifecycle and full payload measure * fix(runtime): record accumulated completed-step usage when an aborted send has no total usage #972 made the terminal LLM-call record fail-closed on usage evidence, but an aborted send (mid-turn exhaust, user stop, stream error) never resolves the SDK totalUsage promise, so the record carrying the capacity verdict diagnostics was skipped entirely. Every COMPLETED step reports real usage at its finish-step boundary; accumulate those samples per send and fall back to the sum at terminal record time. No completed step means no evidence and the record is still skipped, preserving the #972 no-fabrication invariant. * fix(runtime): fail the aborted-send usage fallback closed on any unusable step sample An unusable completed-step sample (normalizeAiSdkUsage returns undefined, #972) made the accumulated sum a PARTIAL cost, and LlmCallRecord has no partial marker — downstream would read it as the whole call. Track sample completeness per send and use the sum only when every completed step reported usable usage; otherwise keep the fail-closed no-record behavior. The terminal outcome never depended on this record: stopReason and the exhausted detail are durable on the CompleteEvent, now asserted explicitly. Also rewrite the stale pre-#972 'missing tokens normalize to 0' comment at the capacity hook's usage read. (cherry picked from commit 8ef9373) (reland after #1005 squash revert) * fix(ui): restore quiet composer picker triggers (#999) (cherry picked from commit ecf515d) (reland after #1005 squash revert) * fix(ui): keep in-flight live turn armed when persisted history covers all steps (#1000) Symptom: the desktop composer's "in progress" indicator flickers off during a running turn. At every step-to-step lull, when all tool/thinking evidence is already covered by the persisted transcript, the busy state drops to idle until the next event recreates the projection. Cause: reconcileTerminalLiveTurn deleted the whole live-turn projection (returning undefined) whenever the filtered steps array became empty, even for a NON-terminal projection. app-shell calls it on every messages/activeLiveTurn change mid-turn, so the projection vanished and turnInFlight (projection exists && !terminal) went false. Fix: an empty result only deletes the projection when current.terminal, mirroring the existing precedent in settleLiveTurnStep. A non-terminal projection survives as { ...current, steps: [] } with its arm preserved. (cherry picked from commit 8153519) (reland after #1005 squash revert)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
During a running turn, the desktop composer's busy indicator (streaming hint + Stop button) flickered off in every step-to-step lull — the moment a step's tool results and thinking were persisted and nothing was actively streaming — then reappeared on the next event. Users saw an idle composer while the sidebar and header still showed the session as 进行中. Refs #646.
Root cause:
reconcileTerminalLiveTurndeleted the whole live-turn projection whenever the persisted transcript covered every live step, even for a non-terminal projection. The renderer reconciles on everymessages/activeLiveTurnchange (app-shell.tsx), so each mid-turn lull droppedturnInFlight— the arm that #646 introduced precisely to keep Stop available across those lulls.Fix: delete an empty projection only when it is
terminal, mirroring the existing guard insettleLiveTurnStep. A non-terminal projection now survives with empty steps, keeping the turn armed (composer shows the calm continuing hint instead of idle). Terminal cleanup viacomplete/abort/erroris unchanged, so a backgrounded session still cannot get a stuck Stop button.Verification
{ turnId, phase, steps: [] }); it returnedundefinedbefore the fix and passes after.packages/uisuite: 153/153 pass.apps/desktopmain suite (includesstreaming-handoff.test.ts, which drivesreconcilePersistedMessagesover this seam): 2526/2526 pass.running).