fix(runtime): decide the tool call event's ledger lane at push time - #2240
Conversation
|
Thanks for the fix — I traced the causal chain against the source and it all checks out: the one-shot lane guess, the P1 — the branch cannot merge: it conflicts with #2233, which already merged the same fix in a different shape. On rebase, my recommendation is to keep your model in the operationId region and drop #2233's predicate (~40 lines: P2 — P2 — the ledger tests pin 1 of the 8 refusal paths. Only schema-rejection is covered end to end (plus the exclusive-step case already on main). The two loop gates, deferred-not-loaded, and boundary-read-failure have no ledger-level test on any branch — and those four are exactly the ones that were broken pre-fix (tagged T1 → orphan). A table-driven variant of P3 (optional): the refusal-text tests only assert the "neither set" wording — "both set" is untested; and the PR body's "reproduces the production issue byte for byte" is only true against your merge-base — on current main #2233 already fixed the symptom, so the headline test guards the refactor (still valuable, worth a sentence in the body). None of this questions the fix itself — design and implementation both check out. Happy to re-review once it's rebased. |
03aaa52 to
ab7a8f7
Compare
|
Thanks — all four land. Rebased onto P1. Taken as recommended: push-time model in the operationId region, P2, the error class. You are right and I had it wrong in the comment itself: Worth noting my own adversarial review pass raised this same site and talked P2, coverage. Now table-driven over seven paths — schema rejection, P3. "Both set" has its own case now, asserting it does not read as the Ready for another look. |
A pre-dispatch tool refusal used to kill the turn. The call event's lane was guessed when the event was built: an `operationId` claims the T1 dispatch protocol, so AgentRun skips the generic projection of that `function_call` and waits for `commitToolPrepared` — which sits after every refusal return. The call fact was never persisted, the synthetic result landed on the generic lane with nothing to attach to, and the ledger refused the `orphan_response` (apache#2234). apache#2233 closed that by predicting the refusals instead: every guard hoisted into a `preflightRejected` boolean read before the operationId. It is correct only while the prediction and the guards agree, and nothing holds them together — a new refusal path, or a guard that grows a condition its hoisted twin does not, silently restores the orphan. This decides the lane where it is known. `pushCallEvent('preflight'|'dispatch')` pushes at most once, every refusal routes through one `refuseBeforeDispatch` helper that keeps call and result together on the generic lane, and only `prepareDurableToolAttempt` asks for T1. The decision is the code path taken, so it cannot drift. apache#2233's predicate, its hoisted boundary read and its early slot reservation go with it; the boundary read and the reservation return to the guards they belong to, and everything else apache#2233 restored is untouched. Two things apache#2233 did not cover, both from the same failure: - A rejected append latched the RuntimeEvent store unavailable, and `commitTerminalRun` returns early on an unavailable store, so one refused event cost the run its own terminal write — a run stuck at `running` with no terminal event. `ToolLedgerRejectionError` now marks a refusal of one bad candidate against a healthy store, and only that skips the latch. Damage found by the workspace health scan throws `ToolLedgerCorruptionError` and keeps failing closed. Be precise about what that second class buys, because the obvious rationale is wrong: the health scan runs only for tool-bearing events, so a damaged ledger refuses tool facts and would have taken the terminal event. The latch is what keeps it out, which reproduces apache#2234's shape for an already-damaged workspace. Left standing deliberately — it is a behaviour change on a path this commit does not otherwise touch — and tracked in apache#2313, with a test that pins the current price rather than hiding it. - `agent_swarm`'s selector refusal said only "Provide exactly one of subagent_id or legacy profile", naming neither which of the two mistakes it was nor one value that would work — and the field list an args violation appends is the top-level one, not `items[n]` where the violation was. Tests: `pre-dispatch-refusal-ledger.test.ts` drives all seven refusal paths through the real `scanToolLedger`; on the tagged lane every one of them reproduces the production `orphan_response`. The terminal-write and ledger-corruption cases run against a `canonical` store, the only durability production ships. Both error classes are pinned where they are produced, in `sqlite-runtime-store.test.ts`: their messages are byte-identical to the plain `Error` strings they replaced, so nothing else in that suite would notice a regression to `throw new Error(...)` — and the latch exemption would silently stop working. Refs apache#2234
ab7a8f7 to
31f67c2
Compare
|
Force-pushed The corrupt-ledger test I cited to you was vacuousIn my P2 reply I said the split came "with a test that pins it". That test pinned nothing. I proved it this round by deleting the latch outright ( The mechanism: the The fix is not more assertions on the same double; the double itself over-modelled the store. Production gates the workspace health scan behind Which exposed a false rationaleMy comment justified corruption staying fail-closed with "nothing this run emits next can be trusted to land." That is false, per the same gate: a run's terminal RuntimeEvent bears no tool fact, so a damaged ledger would have taken it. The latch, not the corruption, is what costs the run its terminal fact — #2234's shape, standing for any workspace with pre-existing ledger damage (the health scan has no I have not changed the behaviour. It predates this PR — before the split, every append failure latched — and whether a run that cannot write tool facts should still get to say it ended is a real decision, not a drive-by. The comment now states the true trade-off, the test pins the price explicitly (it asserts the terminal event does not land, so deciding #2313 either way must change it), and #2313 has the write-up. The split had no test at its producerBoth new classes were only ever thrown by test doubles in the runtime suite. Nothing asserted SmallThe Verification on Also filed from the packaged-build verification, all adjacent to but outside this PR: #2310 (a transition refusal that names no exit left a genuinely-completed task |
|
Thanks for the rewrite — this is exactly the shape we hoped for, and I verified it end to end. The push-time lane model is correct by construction: at-most-once push via the closure flag, all 8 refusal paths route through Four optional notes, none blocking:
Merging now — this closes #2234 properly. |
Summary
A pre-dispatch tool refusal used to kill the turn. The call event's lane was
guessed when the event was built: an
operationIdclaims the T1 dispatchprotocol, so
AgentRun.acceptMappedEventskips the generic projection of thatfunction_calland waits forcommitToolPrepared— which sits after everyrefusal return. The call fact was never persisted,
writeSyntheticToolResultput the refusal on the generic lane with nothing to attach to, and the ledger
refused the
orphan_response(#2234).#2233 closed that by predicting the refusals instead: every guard hoisted into a
preflightRejectedboolean read before theoperationIdis assigned.What this is now
The lane is decided where it is known.
pushCallEvent('preflight' | 'dispatch')pushes at most once, every refusal routes through one
refuseBeforeDispatchhelper that keeps call and result together on the generic lane, and only
prepareDurableToolAttemptasks for T1. The decision is the code path taken,so prediction and guard cannot drift apart — which is the failure mode the
enumeration leaves open: a new refusal path, or a guard that grows a condition
its hoisted twin does not, silently restores the orphan.
Per review, #2233's
preflightRejected,rejectedBeforeClientBoundaryand earlyslot reservation are removed; the boundary read and the reservation go back to
the guards they belong to. Everything else #2233 restored is untouched.
Two things #2233 does not cover, both falling out of the same failure:
RuntimeEvent store unavailable, and
commitTerminalRunreturns early on anunavailable store — a run left at
runningwith no terminal event.ToolLedgerRejectionErrornow marks a refusal of one bad candidate against ahealthy store, and only that skips the latch.
agent_swarm's selector refusal. It said onlyProvide exactly one of subagent_id or legacy profile.— naming neither which of the two mistakes itwas nor a single value that would be accepted, while the field list an args
violation appends is the top-level one, not
items[n]where the violationactually was.
Review responses
P1 — rebase. Done, and taken as recommended: your model in the operationId
region, #2233's predicate dropped.
git merge-treeis clean.P2 — the conflated error class. You are right, and the failure you describe
is reachable:
assertWorkspaceToolLedgerHealthyrefuses well-formed writesbecause of damage elsewhere, so "the store is healthy" — the whole premise of
skipping the latch — is false there. Split into
ToolLedgerCorruptionError,which stays on the fail-closed path.
A second adversarial round on the split found three test defects and one false
rationale, all fixed in the current head (see the comment below for the full
account):
it passed with latching deleted outright, because its double refused every
append. Rewritten so the double refuses exactly what production refuses
(tool-bearing events only); it now fails if the latch is removed.
byte-identical to the plain
Errorstrings they replaced — a regression tothrow new Error(...)would have kept every suite green while the latchexemption silently died. Two
sqlite-runtime-store.test.tscases now assertthe class (and code/eventId) where the throws happen, including corruption
detected across sessions.
run emits next can be trusted to land" — is false: the health scan only runs
for tool-bearing events, so a damaged ledger refuses tool facts and would have
taken the terminal event. The latch is what keeps it out, which is bug(runtime): every pre-dispatch tool refusal kills the turn — the synthetic result lands as an orphan_response the ledger rejects #2234's
shape for an already-damaged workspace. Behaviour deliberately unchanged here
(it predates this PR, and changing it is a call about what fail-closed should
mean); the comment now says what is actually true, the test pins the price
explicitly, and the decision is tracked in bug(runtime): a corrupt ledger costs a run its terminal fact via the latch, not via the corruption #2313.
P2 — coverage.
pre-dispatch-refusal-ledger.test.tsis now table-driven overseven paths: schema rejection, exclusive-step admission, both loop gates
(repeated identical call, repeated ambiguous Computer Use target),
deferred-not-loaded, boundary-read failure, and client-capability non-bypass.
Each asserts the scanner is clean and that the refused call carries no dispatch
fact. Forced onto the tagged lane, all seven produce
orphan_response. Stilluncovered and called out in the file: the subagent cap, which needs five
settlements held open at once and exercises the slot-release path rather than
the lane.
P3. "Both set" now has its own case, asserting it does not read as the
"neither set" sentence. And you are right about the headline claim — on current
main#2233 already fixed the symptom, so these tests guard the refactor ratherthan reproduce a live bug. The PR body no longer says otherwise.
Verification
@maka/core788 pass,@maka/runtime3171 pass / 0 fail,@maka/storage694pass with two unrelated failures pre-existing on this machine
(
package-import.test.jsasserts an empty stderr and Node 22.17 prints thenode:sqliteExperimentalWarning;root-authority-dependency.test.jsfailsidentically with this branch's changes stashed).
tscclean across core / storage / runtime.go orphan, the terminal-write case loses its terminal event, deleting the
latch fails the corrupt-ledger test, and regressing either error class to a
plain
Errorwith the identical message fails its storage test.Review focus
The call event's
queue.pushnow happens later than either previous shape — atthe refusal, or at
prepareDurableToolAttempt. The onlyawaitbetweenappendMessage(callMsg)and the latest possible push isreadExecutionBoundary()on
client_capabilitytools; nothing in that window emits an event referencingthe
toolUseIdor reads the ledger back. The tool context'soperationIdnowreads
pushedCallEvent?.operationId— the id the emitted fact actually carries.Adjacent, deliberately not in scope:
subagent-tools.tscarries the identicalchild-selector
superRefinewith the old thin message, soagent_spawngives amodel the same dead end. One call, no
items[n]path — a follow-up.