[core] Call-site-addressed correlation ids - #3179
Draft
VaguelySerious wants to merge 21 commits into
Draft
VaguelySerious wants to merge 21 commits into
VaguelySerious wants to merge 21 commits into
Conversation
…by event-log position Two production runs on `@workflow/core@5.0.0-beta.36` burned all three divergence-recovery replays at the same event and terminated with CORRUPTED_EVENT_LOG: wrun_41KYJENABV0GSF5YTE9EETV5DD (step vs wait) wrun_41KYJEE01S0GPC9RWT5MEKVCX8 (step vs hook) Replay divergence: step event step_created for step_X belongs to "A", but the current step consumer is "B" `useStep` proxies draw deterministic ULIDs in invocation order, so the ULID -> stepName allocation is a function of the order in which promise resolutions are delivered to workflow code. The delivery-barrier registry pinned that order to event-log position for hook payloads and wait completions, but step results were delivered straight off the serial `promiseQueue` — and their latency varies between replays of the SAME invocation, because the first replay pays full hydration while later replays memo-hit primitive results in the shared `ReplayPayloadCache`. A step completion adjacent in the log to a `wait_completed` was therefore delivered wait-first on a cold replay and step-first on a warm one; whichever order the invocation that wrote the follow-up `step_created` events happened to see became law, and every replay computing the other order diverged permanently. Step results and step failures now register a 'step' delivery barrier at their event-log index and resolve from a detached continuation after every relevant earlier-in-log delivery, mirroring the hook payload path: hydration stays inside the serial queue slot (which also releases `pendingDeliveries`), while the barrier wait and the resolve run off the queue so a queue slot never blocks on a resolution the queue itself drives. Waits and hook payloads likewise defer behind earlier step results. Two details are what actually make the ordering hold, and both were found by testing rather than by reading the code: The deferral set is captured while CONSUMING the event, not at the start of the hydration slot. Captured at slot start it is not merely less deterministic, it is usually empty: an earlier delivery whose own slot runs first on the serial queue has typically already resolved and deregistered its barrier before the later slot begins, so the later delivery does not defer at all. Every event in one drain window is consumed before any slot runs, so consumption time sees all of them. A delivery that had to wait then yields a macrotask before resolving. An earlier delivery being "delivered" only means its `resolve()` ran; the branch it woke may need arbitrarily many further microtask hops before it reaches its next `useStep` call (a `for await` over a hook resumes the generator, settles the promise from `next()`, and only then runs the loop body). Ordering the `resolve()` calls alone therefore buys a fixed hop or two of margin and leaves a hop-count race that holds only for the shortest consumers; yielding a macrotask lets the earlier branch drain completely, whatever its shape. One asymmetry is load-bearing: a step result skips any earlier delivery that will not resolve on its own, i.e. one blocked directly or transitively on a buffered hook payload no consumer has claimed. Such a payload is delivered only when the workflow next reads the hook, and reaching that read commonly requires the step result itself, so gating the step on it stalls the run until the barrier's idle safety net fires — which then releases every delivery queued behind that payload at once and loses the very race the ordering exists to protect. Waits and hooks keep gating on unclaimed payloads, where waiting for the claim IS the guarantee. Tests come in two files. `step-delivery-ordering.test.ts` is byte-identical to the file in the repro-only companion PR #3137 apart from two `it.fails` markers there (which let a repro-only branch have green CI); `sed 's/it\.fails(/it(/g' | cmp` verifies it. Each of its five cases replays one committed log twice through a shared `ReplayPayloadCache`, and the two warm-replay cases fail on main with the production error text. `step-delivery-hop-count.test.ts` exists because those five cases cannot tell "delivered in log order" apart from "resolves a hop or two later than before". It replays logs a live run legitimately produced — the live invocation received the two events in separate deliveries, so the first branch finished long before the second event existed — while the replay receives both in one drain window, and pads the consumer with a varying number of extra awaits so hop count is the only variable. It covers step results against both wait completions and hook payloads, plus step FAILURES against wait completions, since a rejection decides whether a `catch` continuation runs and so which ULID the `useStep` there draws. All 18 cases fail on main; of the 12 that predate the macrotask, 9 still fail with the resolve-ordering-only version of this fix; all 18 pass here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
Follow-up on the step-delivery barrier work, addressing three cases the registry did not yet cover. Each has a regression test in the new `delivery-barrier-coverage.test.ts` that reproduces the production `ReplayDivergenceError` when its fix is reverted. - Step results now defer behind earlier STEP results. The old exclusion assumed the serial `promiseQueue` fixes step-vs-step order, which stopped holding once a step began resolving from a detached continuation instead of its queue slot: two steps consumed in different drain windows can disagree on their deferral set, and the earlier one — parked on the macrotask yield — gets overtaken. - `sleep.ts` and `hook.ts` (waiting-consumer path) now capture their deferral at event-consumption time, as `step.ts` already does. Reading the registry after their queue work misses an earlier step or hook that delivered and retired its barrier in the meantime, skipping both the gate and the macrotask yield. The buffered hook payload path deliberately keeps evaluating at claim time; a consumption-time snapshot there stalls the e2e `hookWithSleepWorkflow`. - Abort deliveries participate in the registry. `_setAborted` fires the signal's listeners, which may invoke a step and draw a ULID, so an abort is as branch-deciding as any other delivery. Also memoizes `resolvesOnItsOwn`. The walk is exponential in the number of live hook/wait barriers, and the registry is not bounded — a fan-out of `Promise.race([hook, sleep])` branches accumulates one barrier per branch per kind (49 measured for 24 branches). At 40 barriers a single scan took 92s before, and is instant after.
…process A replay-context event creation previously described its snapshot with a single watermark, which only proves no event landed above it. It cannot detect a *missing* event below it, so a replay working from a log with a hole still committed events derived from that hole — and because correlation IDs are positional ordinals of one seeded sequence, a one-event difference renames every downstream entity and corrupts the log. Creations now also send the snapshot's event count and its cursor, and a rejection restarts the replay inside the same invocation instead of re-posting the rejected payload (whose IDs the corrected log invalidates) or paying a queue round trip. A world may attach the missing events to its 412, in which case the first restart needs no event-log request. Also guards the suspension `attr_set` write, and re-sorts a merged event log by event ID when an append arrives out of order. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Also makes the v4 event tests derive their mock origin from the override like the rest of the file already does, so a non-empty override does not fail unit tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The matching world-vercel guard shipped and is live in production, so the e2e lanes exercise both halves against the default endpoint. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Resolves the overlap with #3110, which introduced the same event-log merge consolidation this branch had added as `mergeEvents`: `appendUniqueEvents` now carries the optional id set from main plus the out-of-order re-sort and warning, and `mergeEvents` is gone. Main's `withPreconditionRetry` edit drops out with the function itself. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Correlation ids are the Nth draw of one per-run ULID sequence, which makes every id an ordinal: two concurrent replays that disagree about a single event assign different ids to every entity after it, so one replay appends events the other can neither match nor consume and the run fails with CORRUPTED_EVENT_LOG out of a one-event difference. With WORKFLOW_CALLSITE_CORRELATION_IDS=1 an id is derived from the call site that creates the entity — a step's name plus an argument fingerprint, a hook's pinned token — plus a per-scope invocation counter, so the same entity gets the same id in both replays and a late write collides idempotently instead of renaming everything downstream. Ids stay syntactically valid ULIDs; hook tokens for unpinned hooks are derived from the correlation id rather than drawn from the run's PRNG stream. The flag defaults off, and with it off the generator is the positional sequence itself, so nothing about the default path changes. Also sets the flag on the nextjs-turbopack workbench for the race repro.
🦋 Changeset detectedLatest commit: 7cc2c79 The changes in this PR will be included in the next version bump. This PR includes changesets to release 16 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
…ation-ids # Conflicts: # packages/core/src/hook-sleep-interaction.test.ts # packages/core/src/private.ts # packages/core/src/step.test.ts # packages/core/src/workflow.ts
- Evaluate a step's call-site scope lazily: fingerprinting stringifies the arguments, which observably invokes argument getters, so it must not run under the positional scheme where the scope is ignored. - Keep STABLE_ULID on the positional sequence under both schemes. - Cover the blocked-branch fan-out shape (five production corruptions on 5.0.0-beta.43): a dense prefix that ends before a sibling branch's launch completion suppresses all of that branch's draws, rebinding an ordinal from a step to a wait. Positional scheme asserted as the control; the call-site scheme keeps the ids stable under extension. - Revert the measurement-only vercel.json flag from the workbench app. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Sim WorldSimulated world deterministic testing for races. Traces 🟠 world-sim scenario book — 1 fail of 41 total
Full trace: |
The existing storms race deliveries that are all present in a replay's loaded prefix, so they exercise wake-ORDER divergence. The blocked-branch scenario parks each branch on a launch step BEFORE its hook/watchdog race and aims the resume burst at the tail of the round's launch-completion spread: a hook-woken replay can then hold a log that ends just before a sibling's launch step_completed, so the sibling mints zero draws (not even its watchdog wait) and the woken branch's finalize takes the ordinal a fresher writer gives the sibling's wait. One correlation id, two entity kinds, CORRUPTED_EVENT_LOG on an unconsumable step_created — the shape observed in production on 5.0.0-beta.43, which already carries the wake- order fix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
compactConfig whitelists what the sticky comment's history keeps, so the new scenario's attempt count and burst knobs have to be listed there or the config line renders without them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rio' into peter/callsite-correlation-ids
…ngerprint - The flag now opts OUT (=0) instead of in, so deployments built from this branch run call-site ids without env configuration — the A/B against the blocked-branch repro scenario this branch is stacked on. - fingerprintValue no longer JSON.stringifies: stringification invokes accessor properties, so an argument getter that mutates workflow state ran once more per step call than the body wrote it to (caught by the retained-VM parity suite). Data properties are walked by descriptor; accessors contribute a marker without being invoked. - Files whose fixtures pin or reconstruct the positional sequence are pinned to the opt-out scheme with a note; rewriting them is the prerequisite for retiring the opt-out. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Backends order hooks.list by hook id (world-vercel's index key, world-postgres orderBy(hookId), world-local), so the positional scheme's monotonic ids gave hook listings creation order by accident — and the e2e suite, the dashboard, and plausibly users rely on it. Hook ids now carry their creation ordinal in the top 20 bits of the ULID's random section, restoring the ordering while the scope and per-scope ordinal keep feeding the identity hash. The cost, noted in code: a pinned hook's id becomes sensitive to how many hooks preceded it, the same per-kind residual waits already carry. This was every non-Windows e2e lane failure on the previous push: the webhookWorkflow test maps tokens by hooks.list position. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
(AI) Derives correlation ids from call sites instead of a per-run draw sequence, targeting the residual
CORRUPTED_EVENT_LOGclass that survives #3554. Default on in this PR (WORKFLOW_CALLSITE_CORRELATION_IDS=0opts back into the positional sequence), and stacked on #3696 so the twoevent-log-race-reprolanes form an A/B: the base PR runs the blocked-branch scenario on positional ids, this PR runs the identical scenario with call-site ids on.Problem
Correlation ids today are the Nth draw of one seeded ULID sequence per run, so every id is an ordinal over the whole body. Two concurrent replays of the same run that hold dense prefixes of different lengths can legitimately disagree about how many draws precede a given call: a branch whose resolution is absent from the shorter prefix stays parked at its
awaitand mints nothing, so every later draw shifts by its missing draws. Both replays are individually deterministic; the binding of ordinal to entity is what is unstable under log extension. The second writer then commits events the first can neither match (ReplayDivergenceError) nor consume (Unconsumed event), amplifying a one-event disagreement intoCORRUPTED_EVENT_LOG.#3554 pinned the wake order of deliveries present in the prefix. It cannot cover a delivery absent from the prefix: those draws are missing, not misordered. Five production runs on
5.0.0-beta.43(which contains #3554) failed exactly this way on 2026-08-20, each binding one ordinal to both astep_and await_entity in a fan-out that races a hook against a watchdog sleep — the shape #3696'sblocked-branchscenario reproduces.What this changes
An id is derived from the call site:
Scopes: steps use the step name plus an argument fingerprint; hooks use a pinned token when there is one; waits, attribute writes and abort controllers use a per-kind scope. Ids stay syntactically valid 26-char ULIDs, so backend id validation is unaffected and no wire or World change is needed. Unpinned hook tokens are derived from the hook's correlation id instead of the run's PRNG stream, which was positional for the same reason.
The argument fingerprint is a hand-rolled walk rather than
JSON.stringify: stringification invokes accessor properties, and an argument getter that mutates workflow state would observably run once more per step call than the body wrote it to (the retained-VM parity suite pins that count). Data properties are walked by descriptor; accessors contribute a stable marker without being invoked.A stale replay that reaches the same call site as the canonical one mints the same id, so its write is an idempotent duplicate (skipped since #3381) instead of a rename of everything downstream.
Tests
correlation-id.test.ts: unit tests (id shape, scheme parity, per-scope ordinals, hash diffusion, fingerprint and token derivation).callsite-correlation-ids.test.ts: end-to-end throughrunWorkflow, nothing hardcoded. Includes the production blocked-branch fan-out shape: two dense prefixes differing by one siblingstep_completed. Under the positional scheme (asserted as the control) a finalize step's ordinal rebinds to a sibling's watchdog wait, reproducing the production step-vs-wait ulid collision; under the call-site scheme the ids are identical across both prefixes.Known gaps
sleep()has no distinguishing input), so wait ids remain ordinals within the wait kind and a disagreement about sleep count still renames later waits. The end state wants compiler-injected static call-site ids; this PR is the runtime-derivable subset.runtime-tuning.mdx.🤖 Generated with Claude Code