fix(core): repay a force-claim victim's wake independent of the claimer's log tail by pranaygp · Pull Request #4398 · vercel/workflow

pranaygp · 2026-09-24T23:29:43Z

Follow-up to #4392 (merged); rebased onto main. Closes #4393.
Server-side model: vercel/workflow-server#1059.

Problem

A run that force-claims a token writes a forced hook_created and then publishes a wake for the run it took the token from. If the invocation dies between the two, the only recovery was forcedCreationOwingWake: on the next replay, republish if the forced creation is still the last row the claimer wrote itself. Any own row landing after the creation hid the debt:

a step or wait terminal from another invocation, which can always happen;
since fix(core): write hook events alongside a suspension's steps, not ahead of them #4392, a step, wait, attribute or other hook row the same suspension writes concurrently.

A victim parked only on await hook then never woke.

Fix (option 1 from the issue)

forcedCreationsOwingWake(events, nowMs) in hook-wake.ts now returns every forced hook_created (with eventData.forceClaimedFrom) in the loaded log whose createdAt is within FORCE_CLAIM_WAKE_REPUBLISH_WINDOW_MS (24 h). republishOwedForceClaimVictimWakes republishes each one under the existing hook-force-claim-<hookId> key. It never looks at what came after the creation, so no row can hide the debt. Both engines share this rule.

What is unchanged:

Chain case: a run that took the token from A and was then taken from by C still repays A's wake.
Self-claims and victims with no recorded workflowName are still skipped. On the replay path they are now skipped silently, because the creating invocation already logged the warning.
A delivery's hook_received and a sealed-log noop never mattered to the window rule.

What changed in the engines:

node:vm (handleSuspension): the republish no longer blocks the suspension. The rule does not depend on write order, so it runs alongside the suspension's writes and is joined before return (try/finally around settlePhase). A per-invocation forceClaimVictimWakes set, created in runtime.ts and also filled by createHookEvent for the invocation's own forced creations, keeps it to one send per hook per invocation across the replay loop's suspensions.
QuickJS: republishes once per invocation on the loaded log. The entrypoint only runs once per invocation, so no set is needed.

The window bound: 24 h from the row's `createdAt`

Why it's long enough. The replay that has to repay a wake is a redelivery of the invocation that wrote the creation. A VQS message is retained for 24 h from its send (retentionSeconds defaults to 86400; world-vercel does not override it), and the creation is written after the send. So every such redelivery arrives within 24 h of the creation. That covers VQS's own redelivery backoff too (≤900 s per hop, MAX_QUEUE_DELIVERIES = 48, about 9-10 h).
Why it's cheap. VQS's idempotency window is min(retention, 24h) (the @vercel/queue SendOptions.idempotencyKey docs), so every republish inside the window collapses into the one delivered wake. world-postgres's handler skips a key it has already completed (an in-process LRU of 10k entries). world-local dedupes a key only while its message is in flight, so there a republish can deliver the victim one extra replay, which reads nothing new.
Slot identity. The time comes from event.createdAt, never the event id: a slot id decodes to epoch 0. On slot-identity runs createdAt is the writer's client occurredAt, which workflow-server clamps to within 1 h ahead / 7 d behind server time (slotEventCreatedAt). Normal skew is milliseconds to seconds; worst case the window's edge moves by up to 1 h. A future-dated creation counts as recent, and an unparseable createdAt counts as recent too (errs toward a wake). A republish that slips past VQS's window is one extra wake, which is harmless.
Steady-state cost. No extra reads: the scan is over the log already loaded. Runs that never force-claimed send nothing. A run that did force-claim sends one deduped message per invocation for 24 h after the claim. On node:vm that send runs concurrently with the suspension's writes, so it adds no latency there.

Why not options 2 or 3

A durable "wake published" marker (option 2) is exact, but it needs a new event or field in the protocol and an extra write on the force path in every World.
Having the World own the wake (option 3) needs a queue in world-local and world-postgres.

The window closes the gap with neither, and its only imprecision is a harmless duplicate wake.

Docs, tests, changeset

create-hook.mdx: the force-claim paragraph now describes a durable wake, republished within 24 h under one idempotency key. It no longer says "repaired on a best-effort basis". Doc comments in hook-wake.ts, suspension-handler.ts and quickjs-entrypoint.ts no longer point at Force-claim victim wake recovery shouldn't depend on the claimer's log tail #4393.
suspension-handler.test.ts ('force-claim victim wake'):
- repaid past own step_created/step_completed/wait_completed
- repaid past a delivery's hook_received
- chain (foreign hook_disposed{forceClaimedBy}, then an own row)
- repaid after the run's own hook_disposed
- no republish outside the window
- several recent creations, each under its own key
- self-claim and legacy victim skipped
- one send per invocation across suspensions, including the invocation's own forced creation
- the republish does not hold the suspension's writes
- forced hooks on different tokens are created concurrently (the first create is held until the second is issued, so the old one-token-at-a-time order deadlocks), each victim woken once under its own key
quickjs-force-claim-wake.test.ts: recent creation, hook_received, own step/wait rows, chain, outside the window, and the same concurrent-forced-tokens test.
Changeset: @workflow/core patch.

Assumptions / open points

world-local mutual force-claims. If two runs force-claim from each other and both stay parked (not terminal), each replay's republish wakes the other. World-local does not dedupe completed keys, so they can ping-pong for up to 24 h. The old tail rule had the same loop with no time bound, as long as neither run wrote another row. VQS and world-postgres dedupe the keys, so the loop stops there. I did not change world-local's dedupe semantics in this PR.
node:vm replay that completes without suspending. The node:vm engine republishes from handleSuspension, so a replay that finishes the run without suspending does not republish. The same was true of the tail rule. QuickJS republishes on load and is not affected.

Docs Preview

Page	v5
`createHook()`: taking over a token another run holds	/docs/api-reference/workflow/create-hook#take-over-a-token-another-run-holds

Checks

cd packages/core && pnpm vitest run src: 2673 passed
tsc --noEmit for core: ok
biome check on the changed files: no new diagnostics

🤖 Generated with Claude Code

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

vercel · 2026-09-24T23:29:51Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
example-nextjs-workflow-turbopack	Ready	Preview, v0	Sep 25, 2026 12:24am UTC
example-nextjs-workflow-webpack	Ready	Preview, v0	Sep 25, 2026 12:24am UTC
example-workflow	Ready	Preview, v0	Sep 25, 2026 12:24am UTC
workbench-astro-workflow	Ready	Preview, v0	Sep 25, 2026 12:24am UTC
workbench-express-workflow	Ready	Preview, v0	Sep 25, 2026 12:24am UTC
workbench-fastify-workflow	Building	Preview, v0	Sep 25, 2026 12:24am UTC
workbench-hono-workflow	Ready	Preview, v0	Sep 25, 2026 12:24am UTC
workbench-nestjs-workflow	Ready	Preview, v0	Sep 25, 2026 12:24am UTC
workbench-nitro-workflow	Ready	Preview, v0	Sep 25, 2026 12:24am UTC
workbench-nuxt-workflow	Ready	Preview, v0	Sep 25, 2026 12:24am UTC
workbench-python-workflow	Ready	Preview, v0	Sep 25, 2026 12:24am UTC
workbench-sveltekit-workflow	Ready	Preview, v0	Sep 25, 2026 12:24am UTC
workbench-tanstack-start-workflow	Ready	Preview, v0	Sep 25, 2026 12:24am UTC
workbench-vite-workflow	Ready	Preview, v0	Sep 25, 2026 12:24am UTC
workflow-docs	Ready	Preview, v0	Sep 25, 2026 12:24am UTC
workflow-swc-playground	Ready	Preview, v0	Sep 25, 2026 12:24am UTC
workflow-tarballs	Ready	Preview, v0	Sep 25, 2026 12:24am UTC
workflow-web	Ready	Preview, v0	Sep 25, 2026 12:24am UTC

changeset-bot · 2026-09-24T23:29:53Z

🦋 Changeset detected

Latest commit: a866bbe

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages

Name	Type
@workflow/core	Patch
@workflow/builders	Patch
@workflow/cli	Patch
@workflow/next	Patch
@workflow/nitro	Patch
@workflow/vitest	Patch
@workflow/web-shared	Patch
@workflow/web	Patch
workflow	Patch
@workflow/world-testing	Patch
@workflow/astro	Patch
@workflow/nest	Patch
@workflow/rollup	Patch
@workflow/sveltekit	Patch
@workflow/vite	Patch
@workflow/nuxt	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

github-actions · 2026-09-24T23:30:01Z

🧪 E2E Test Results

❌ Some tests failed

❌ Failed E2E Tests

▲ Vercel Production (62 failed)

python-node (62 failed):

promiseAllWorkflow | wrun_41M3AZ5JNA0GKJEKZVX1BZBFVB | 🔍 observability
promiseRaceWorkflow | wrun_41M3AZ5JNA0GKJEKZVX1BZBFVC | 🔍 observability
promiseAnyWorkflow | wrun_41M3AZ5JNA0GKJEKZVX1BZBFVD | 🔍 observability
retainedInterleavingWorkflow | wrun_41M3AZ5V3X0GJTE4YM4TB00K6A | 🔍 observability
hookWorkflow | wrun_41M3AZ5ZEN0GNRYAX49EXRRRMS | 🔍 observability
hookWorkflow is not resumable via public webhook endpoint | wrun_41M3AZ600K0GN82JKDQ93KAG6Q | 🔍 observability
sleepingWorkflow | wrun_41M3AZ67YR0GQJCCD05TJSMVEN | 🔍 observability
parallelSleepWorkflow | wrun_41M3AZ698W0GZW69RYNG3Z46RX | 🔍 observability
sleepWinsRaceWorkflow | wrun_41M3AZ69ZR0GQ1TGEWC3FRW8EW | 🔍 observability
stepWinsRaceWorkflow | wrun_41M3AZ6EFH0GQR5CK1B6Y1GRGT | 🔍 observability
nullByteWorkflow | wrun_41M3AZ6FS00GS5XF37F5R1914S | 🔍 observability
outputStreamWorkflow no startIndex (reads all chunks)
outputStreamWorkflow positive startIndex (skips first chunk)
outputStreamWorkflow negative startIndex (reads from end)
outputStreamWorkflow - getTailIndex and getChunks getTailIndex returns correct index after stream completes
outputStreamWorkflow - getTailIndex and getChunks getChunks returns same content as reading the stream
outputStreamInsideStepWorkflow - getWritable() called inside step functions | wrun_41M3AZ6Y9Q0GPR1GWQ0F0RCMEN | 🔍 observability
utf8StreamWorkflow | wrun_41M3AZ6ZCT0GTB0SWT9H65Y14W | 🔍 observability
writableForwardedFromWorkflowWorkflow | wrun_41M3AZ723Q0GRDYPBZVM0DB016 | 🔍 observability
writableForwardedFromStepWorkflow | wrun_41M3AZ74EH0GPHNG3FFMR15BGZ | 🔍 observability
promiseRaceStressTestWorkflow | wrun_41M3AZ772X0GK3FEJAYF3N7Q8S | 🔍 observability
error handling retry behavior regular Error retries until success
error handling retry behavior FatalError fails immediately without retries
error handling retry behavior RetryableError respects custom retryAfter delay
error handling retry behavior maxRetries=0 disables retries
error handling catchability FatalError can be caught and detected with FatalError.is()
error handling catchability step throw round-trips FatalError with cause chain to workflow catch
error handling catchability workflow throw round-trips FatalError + cause through run_failed event
error handling not registered StepNotRegisteredError fails the step but workflow can catch it
error handling not registered StepNotRegisteredError fails the run when not caught in workflow
hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M3AZ8PQ70GJPE6P397EG105Z | 🔍 observability
concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M3AZ8J1A0GWEF11KDJQBN0TF | 🔍 observability
hookGetConflictWorkflow - awaiting hook.getConflict() registers hook without payload | wrun_41M3AZ8HH20GKEQ3TZYVN64TDK | 🔍 observability
'hookGetConflictWithPriorStepWorkflow' - hook.getConflict() does not block step execution | wrun_41M3AZ8K9H0GWGBFHJB8268JTZ | 🔍 observability
'hookGetConflictWithParallelStepWorkfl…' - hook.getConflict() does not block step execution | wrun_41M3AZ8RQ10GYJ9PM1S9DQN9XD | 🔍 observability
hookGetConflictThenStepParallelWorkflow - hook.getConflict() continuation step runs alongside other steps | wrun_41M3AZ8SW80GJYCZV7FC27N2JW | 🔍 observability
hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M3AZ8W800GPDG7E5XCNWD5E6 | 🔍 observability
hookClaimOnlyMutexWorkflow - hook works as a pure run mutex without payload data | wrun_41M3AZ9HTH0GV3AAFNAW3ZWC32 | 🔍 observability
hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue | wrun_41M3AZ9A730GQWTV6H3K5E1HBG | 🔍 observability
hookSignalOwnerWorkflow - duplicate forwards its payload to the owner via resumeHook | wrun_41M3AZ9B050GT82T8H8X5PDQH3 | 🔍 observability
resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M3AZA4Q90GR3K8X3BSBKP666 | 🔍 observability
hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_41M3AZA9240GJB0B7E295K3BG7 | 🔍 observability
hookTokenReuseLoopWorkflow - same run recreates a hook with the same token after dispose() | wrun_41M3AZA6VV0GK1CZ27AZ3HW0G1 | 🔍 observability
spawnWorkflowFromStepWorkflow - spawning a child workflow using start() inside a step | wrun_41M3AZASP40GVS11YFQFKBEZH6 | 🔍 observability
cancelRun - cancelling a running workflow | wrun_41M3AZBS6A0GZH60SDPWE0ZA5J | 🔍 observability
cancelRun via CLI - cancelling a running workflow | wrun_41M3AZBSHB0GX4C51DDQ2SRW2A | 🔍 observability
hookWithSleepWorkflow - hook payloads delivered correctly with concurrent sleep | wrun_41M3AZC2370GQJR3Y20WQ7016P | 🔍 observability
hookWithSleepFinalStepWorkflow - step only on final payload | wrun_41M3AZC40K0GXAD7S6C1WMM2JK | 🔍 observability
sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M3AZC6K90GGVCTYXD0GBGD4W | 🔍 observability
sleepWithSequentialStepsWorkflow - sequential steps work with concurrent sleep (control) | wrun_41M3AZC6S10GPBVEJN6ECPSRM3 | 🔍 observability
AbortController abortTimeoutWorkflow: timeout cancels long-running step
AbortController abortParallelWorkflow: abort cancels all parallel steps
AbortController abortReasonWorkflow: abort reason preserved across boundaries
AbortController abortViaHookWorkflow: external hook triggers abort on in-flight step
metadataFromHelperWorkflow - getWorkflowMetadata/getStepMetadata work from module-level helper (#1577) | wrun_41M3AZCNVR0GKSPSHE7H00R0HM | 🔍 observability
resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M3AZCQ590GRWKGGS9T8NH21N | 🔍 observability
setAttributes start: initial attributes are seeded on run creation
setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
setAttributes Promise.all of disjoint-key writes: every key lands
setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run

⚠️ Flaky E2E Tests (passed on retry)

These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.

a run can take over its own earlier hook (sveltekit · local-dev / local / node / stable)

🛠 Infra Events (absorbed by the harness)

Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.

127 infra events

cold-start-warmup · suite warmup (tanstack-start) · at 00:24:48Z · abandoned wrun_01M3AZ62GB0S14G4Z87ZWDV2MT
cold-start-warmup · suite warmup (python) · at 00:24:56Z · abandoned wrun_41M3AZ37090GHY0JJM0MV6J3Y9 · (+7 more)
run-pickup-stall · promiseAnyWorkflow (python) · at 00:25:12Z · abandoned wrun_41M3AZ6VY80GXNYKEJN07VP5BX
run-pickup-stall · promiseAllWorkflow (python) · at 00:25:12Z · abandoned wrun_41M3AZ6VY80GXNYKEJN07VP5BV
run-pickup-stall · promiseRaceWorkflow (python) · at 00:25:12Z · abandoned wrun_41M3AZ6VY80GXNYKEJN07VP5BW
run-pickup-stall · retainedInterleavingWorkflow (python) · at 00:25:12Z · abandoned wrun_41M3AZ6VY80GXNYKEJN07VP5BY
run-pickup-stall · hookWorkflow (python) · at 00:25:12Z · abandoned wrun_41M3AZ6VYE0GPNKGJRW1HZQA3G
run-pickup-stall · retainedInterleavingWorkflow (python) · at 00:26:00Z · abandoned wrun_41M3AZ8ART0GGVC649X4N65T9N
run-pickup-stall · hookWorkflow (python) · at 00:26:00Z · abandoned wrun_41M3AZ8AS10GJQWBDSYDBWKJRG
run-pickup-stall · promiseRaceWorkflow (python) · at 00:26:13Z · abandoned wrun_41M3AZ8Q8Y0GTRA1T0KP1Y4DGP
run-pickup-stall · promiseAllWorkflow (python) · at 00:26:13Z · abandoned wrun_41M3AZ8QC60GSEC5FDKZZAE33K
run-pickup-stall · promiseAnyWorkflow (python) · at 00:26:13Z · abandoned wrun_41M3AZ8QGC0GGQR0HZBH6FRJ2N
run-pickup-stall · sleepingWorkflow (python) · at 00:26:51Z · abandoned wrun_41M3AZ9WJJ0GN7NH71Q33E8X9T
run-pickup-stall · hookWorkflow is not resumable via public webhook endpoint (python) · at 00:26:51Z · abandoned wrun_41M3AZ9W4C0GGEE82NH9EBKYGB
run-pickup-stall · parallelSleepWorkflow (python) · at 00:27:16Z · abandoned wrun_41M3AZAK340GKP7HGMWNXNXFPK
run-pickup-stall · stepWinsRaceWorkflow (python) · at 00:27:16Z · abandoned wrun_41M3AZAKSD0GMZ8TDZY041JF0M
run-pickup-stall · sleepWinsRaceWorkflow (python) · at 00:27:16Z · abandoned wrun_41M3AZAKJZ0GHC2MNS51JHKGEX
run-pickup-stall · hookWorkflow is not resumable via public webhook endpoint (python) · at 00:28:43Z · abandoned wrun_41M3AZC9ZH0GVRE6P2AB55Q8K5
run-pickup-stall · sleepingWorkflow (python) · at 00:28:44Z · abandoned wrun_41M3AZCA5M0GW2N5VA3Y67H5H9
run-pickup-stall · sleepWinsRaceWorkflow (python) · at 00:28:50Z · abandoned wrun_41M3AZDBFE0GYGKBYG2BQKPMY6
run-pickup-stall · parallelSleepWorkflow (python) · at 00:28:50Z · abandoned wrun_41M3AZDAZN0GHGSJE9ERWG9PEX
run-pickup-stall · stepWinsRaceWorkflow (python) · at 00:28:51Z · abandoned wrun_41M3AZDBP00GP30ESJ467JNCYQ
run-pickup-stall · nullByteWorkflow (python) · at 00:29:11Z · abandoned wrun_41M3AZE5AA0GH1RG8D06CAGKSE
run-pickup-stall · no startIndex (reads all chunks) (python) · at 00:29:11Z · abandoned wrun_41M3AZE5GM0GTS2BJM7D68WQJB
run-pickup-stall · positive startIndex (skips first chunk) (python) · at 00:29:44Z · abandoned wrun_41M3AZF5JN0GMXA99A8SSDA4YN
run-pickup-stall · negative startIndex (reads from end) (python) · at 00:29:45Z · abandoned wrun_41M3AZF62F0GWZ8QADEZQKNR7V
run-pickup-stall · getTailIndex returns correct index after stream completes (python) · at 00:29:47Z · abandoned wrun_41M3AZF8NQ0GW4ATRYHBGRVJAP
run-pickup-stall · getTailIndex returns -1 before any chunks are written (python) · at 00:30:11Z · abandoned wrun_41M3AZFZXA0GXN55DERP8Z78HV
run-pickup-stall · getChunks returns same content as reading the stream (python) · at 00:30:11Z · abandoned wrun_41M3AZG03N0GJRGJYVJ496S52F
run-pickup-stall · outputStreamInsideStepWorkflow - getWritable() called inside step functions (python) · at 00:30:27Z · abandoned wrun_41M3AZGG0Q0GH5NQSMEBBTTDBT
run-pickup-stall · utf8StreamWorkflow (python) · at 00:30:44Z · abandoned wrun_41M3AZH05Q0GQH4HNB2GH242SV
run-pickup-stall · nullByteWorkflow (python) · at 00:30:47Z · abandoned wrun_41M3AZH38Q0GT31HP55B29Q6G4
run-pickup-stall · no startIndex (reads all chunks) (python) · at 00:30:48Z · abandoned wrun_41M3AZH3RR0GJ7SCG31V0XCCXJ
run-pickup-stall · crossContextSerdeWorkflow - classes defined in step code are deserializable in workflow context (nextjs-webpack) · at 00:31:05Z · abandoned wrun_01M3AZHNRZQ47XKTFTQ7P0Y9W6
run-pickup-stall · errorSubclassRoundTripWorkflow - first-class Error subclasses survive every serialization boundary (nextjs-webpack) · at 00:31:10Z · abandoned wrun_01M3AZHTFS98SSNNHD1HW7BZZ6
run-pickup-stall · stepFunctionAsStartArgWorkflow - step function reference passed as start() argument (nextjs-webpack) · at 00:31:10Z · abandoned wrun_01M3AZHTGT0ATMMM3YSGA9D8FN
run-pickup-stall · cancelRun - cancelling a running workflow (nextjs-webpack) · at 00:31:10Z · abandoned wrun_01M3AZHV3W5PZ2S87MJEF8MHEE
run-pickup-stall · positive startIndex (skips first chunk) (python) · at 00:31:11Z · abandoned wrun_41M3AZHTPN0GXZNGF0NCPWGX6C
run-pickup-stall · negative startIndex (reads from end) (python) · at 00:31:28Z · abandoned wrun_41M3AZJBBY0GKH43P8WMTN1ZBP
run-pickup-stall · getTailIndex returns correct index after stream completes (python) · at 00:31:48Z · abandoned wrun_41M3AZJYJQ0GK90MGG1FDWH11S
run-pickup-stall · getChunks returns same content as reading the stream (python) · at 00:31:50Z · abandoned wrun_41M3AZK0SJ0GTTXDYMNWQ5C7HK
run-pickup-stall · outputStreamInsideStepWorkflow - getWritable() called inside step functions (python) · at 00:32:11Z · abandoned wrun_41M3AZKN9N0GPADJ34MZ8YTJZ4
run-pickup-stall · writableForwardedFromWorkflowWorkflow (python) · at 00:32:28Z · abandoned wrun_41M3AZM5YY0GVJGESGWGHW33JC
run-pickup-stall · writableForwardedFromStepWorkflow (python) · at 00:32:46Z · abandoned wrun_41M3AZMQXN0GM4AZBJJVN7TV82
run-pickup-stall · utf8StreamWorkflow (python) · at 00:32:48Z · abandoned wrun_41M3AZMS5Q0GJWM7HAVCRYGSY2
run-pickup-stall · promiseRaceStressTestWorkflow (python) · at 00:32:51Z · abandoned wrun_41M3AZMWV60GSFH7DPVN4ER53F
run-pickup-stall · regular Error retries until success (python) · at 00:33:12Z · abandoned wrun_41M3AZNGPX0GXN8216NNB4SYJP
run-pickup-stall · FatalError fails immediately without retries (python) · at 00:33:52Z · abandoned wrun_41M3AZPQE60GTRRF03KH3GPFAP
run-pickup-stall · RetryableError respects custom retryAfter delay (python) · at 00:34:12Z · abandoned wrun_41M3AZQB9X0GMKERWXWJS8PPNJ
run-pickup-stall · maxRetries=0 disables retries (python) · at 00:34:28Z · abandoned wrun_41M3AZQV4Z0GVKFZ7EFQ12ZHM5
run-pickup-stall · FatalError can be caught and detected with FatalError.is() (python) · at 00:34:46Z · abandoned wrun_41M3AZRD3N0GW362MYY6M45VXD
run-pickup-stall · step throw round-trips FatalError with cause chain to workflow catch (python) · at 00:34:48Z · abandoned wrun_41M3AZREBR0GNTNNXCNCKP7XS5
run-pickup-stall · workflow throw round-trips FatalError + cause through run_failed event (python) · at 00:34:51Z · abandoned wrun_41M3AZRJ160GRTZH8YRVRANNKQ
run-pickup-stall · StepNotRegisteredError fails the step but workflow can catch it (python) · at 00:35:12Z · abandoned wrun_41M3AZS5WY0GQC7V3KDXHE43M9
run-pickup-stall · StepNotRegisteredError fails the run when not caught in workflow (python) · at 00:35:33Z · abandoned wrun_41M3AZSNR00GM7ZCG5XWXPYSJB
run-pickup-stall · promiseRaceStressTestWorkflow (python) · at 00:36:36Z · abandoned wrun_41M3AZVBJ90GJJPPPY58D8X20W
run-pickup-stall · hookCleanupTestWorkflow - hook token reuse after workflow completion (python) · at 00:37:09Z · abandoned wrun_41M3AZVM820GTAPEWXXMZVTHW9
run-pickup-stall · regular Error retries until success (python) · at 00:37:12Z · abandoned wrun_41M3AZVGB00GP7PG4QG0CAVKGC
run-pickup-stall · writableForwardedFromWorkflowWorkflow (python) · at 00:37:24Z · abandoned wrun_41M3AZVHKK0GH5JR7SE1T63WXW
run-pickup-stall · writableForwardedFromStepWorkflow (python) · at 00:37:26Z · abandoned wrun_41M3AZVHSG0GWCFK1XED774HQH
run-pickup-stall · RetryableError respects custom retryAfter delay (python) · at 00:38:06Z · abandoned wrun_41M3AZXEV30GNJE6EV7NHA4NHZ
run-pickup-stall · FatalError fails immediately without retries (python) · at 00:38:08Z · abandoned wrun_41M3AZX65B0GPTSYABMBDNB5D1
run-pickup-stall · maxRetries=0 disables retries (python) · at 00:38:18Z · abandoned wrun_41M3AZY87K0GTGCCYMTYZ2JXCW
run-pickup-stall · FatalError can be caught and detected with FatalError.is() (python) · at 00:38:30Z · abandoned wrun_41M3AZZ6H30GQB51W1G9AP704N
run-pickup-stall · workflow throw round-trips FatalError + cause through run_failed event (python) · at 00:38:30Z · abandoned wrun_41M3AZZ6ZG0GGEH6WR7ERP6H5C
run-pickup-stall · step throw round-trips FatalError with cause chain to workflow catch (python) · at 00:38:31Z · abandoned wrun_41M3AZZ6SM0GGAR3JF6HZB7W78
run-pickup-stall · StepNotRegisteredError fails the step but workflow can catch it (python) · at 00:38:35Z · abandoned wrun_41M3AZZC5B0GW2DGYZMVNXSSJ5
run-pickup-stall · StepNotRegisteredError fails the run when not caught in workflow (python) · at 00:38:59Z · abandoned wrun_41M3B003KV0GGY8BB2XQRJJDR4
run-pickup-stall · hookCleanupTestWorkflow - hook token reuse after workflow completion (python) · at 00:39:30Z · abandoned wrun_41M3B012550GSQWKRGJJQ3YQ57
run-pickup-stall · hookGetConflictWorkflow - awaiting hook.getConflict() registers hook without payload (python) · at 00:39:34Z · abandoned wrun_41M3B015N80GPWPBH2RBW8EZ2S
run-pickup-stall · 'hookGetConflictWithPriorStepWorkflow' - hook.getConflict() does not block step execution (python) · at 00:39:35Z · abandoned wrun_41M3B016RB0GQH8AB60VS8PX8P
run-pickup-stall · 'hookGetConflictWithParallelStepWorkfl…' - hook.getConflict() does not block step execution (python) · at 00:40:00Z · abandoned wrun_41M3B01ZQH0GZY1TNP5QGFSRCY
run-pickup-stall · hookGetConflictThenStepParallelWorkflow - hook.getConflict() continuation step runs alongside other steps (python) · at 00:40:18Z · abandoned wrun_41M3B02GGJ0GRA470FA7NSW3TT
run-pickup-stall · hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (python) · at 00:40:33Z · abandoned wrun_41M3B02Z8J0GVC875W1BNNSAGW
run-pickup-stall · hookGetConflictWorkflow - awaiting hook.getConflict() registers hook without payload (python) · at 00:40:34Z · abandoned wrun_41M3B030ZH0GGDHFCFV2S3277N
run-pickup-stall · 'hookGetConflictWithPriorStepWorkflow' - hook.getConflict() does not block step execution (python) · at 00:40:36Z · abandoned wrun_41M3B0324M0GZ6101QBM3Q0KZ8
run-pickup-stall · 'hookGetConflictWithParallelStepWorkfl…' - hook.getConflict() does not block step execution (python) · at 00:41:01Z · abandoned wrun_41M3B03V450GYSACHR471D0Q1Q
run-pickup-stall · hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (python) · at 00:41:20Z · abandoned wrun_41M3B04DBH0GG0ZCD9D4YTD1NW
run-pickup-stall · hookClaimOnlyMutexWorkflow - hook works as a pure run mutex without payload data (python) · at 00:41:35Z · abandoned wrun_41M3B04W9T0GW1A9PXMPCJE804
run-pickup-stall · hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (python) · at 00:41:36Z · abandoned wrun_41M3B04XFP0GNBJE4F93RDS0YK
run-pickup-stall · hookGetConflictThenStepParallelWorkflow - hook.getConflict() continuation step runs alongside other steps (python) · at 00:41:48Z · abandoned wrun_41M3B0594J0GWEGRZBGCHRMGTD
run-pickup-stall · hookSignalOwnerWorkflow - duplicate forwards its payload to the owner via resumeHook (python) · at 00:42:02Z · abandoned wrun_41M3B05PEM0GKQBESQBPJ1JKTX
run-pickup-stall · resume-or-start route pattern - resumeHook retried after start() reaches the new run (python) · at 00:42:07Z · abandoned wrun_41M3B05VH60GPG46S44S7SX88J
run-pickup-stall · hookClaimOnlyMutexWorkflow - hook works as a pure run mutex without payload data (python) · at 00:42:22Z · abandoned wrun_41M3B06AF70GHG1T951FKDZ9AK
run-pickup-stall · hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (python) · at 00:42:23Z · abandoned wrun_41M3B06BB50GTMXNKTYAS4AEKW
run-pickup-stall · hookSignalOwnerWorkflow - duplicate forwards its payload to the owner via resumeHook (python) · at 00:42:49Z · abandoned wrun_41M3B074JX0GP6F761HCY92KW8
run-pickup-stall · resume-or-start route pattern - resumeHook retried after start() reaches the new run (python) · at 00:42:54Z · abandoned wrun_41M3B079RF0GKFDKQ3TBVB6680
run-pickup-stall · hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running (python) · at 00:43:09Z · abandoned wrun_41M3B07RBR0GV6NRBT2PR3BCJF
run-pickup-stall · hookTokenReuseLoopWorkflow - same run recreates a hook with the same token after dispose() (python) · at 00:43:10Z · abandoned wrun_41M3B07SFN0GK429XM2G9B47PX
run-pickup-stall · spawnWorkflowFromStepWorkflow - spawning a child workflow using start() inside a step (python) · at 00:43:19Z · abandoned wrun_41M3B081SZ0GRRYF3NA1CR69BP
run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 00:43:41Z · abandoned wrun_41M3B08PXE0GYPCKQWSHZ3QRBP
run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 00:43:41Z · abandoned wrun_41M3B08QHA0GY45RBC5RXF7AWQ
run-pickup-stall · hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running (python) · at 00:43:57Z · abandoned wrun_41M3B096P90GXGK10W7CJ9MB56
run-pickup-stall · hookTokenReuseLoopWorkflow - same run recreates a hook with the same token after dispose() (python) · at 00:43:58Z · abandoned wrun_41M3B097F70GMMAJZ4SE371NKG
run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 00:44:13Z · abandoned wrun_41M3B09P5S0GVQS11HYSC55QE7
run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 00:44:14Z · abandoned wrun_41M3B09Q470GRSTDSNKVGTFSFF
run-pickup-stall · hookWithSleepWorkflow - hook payloads delivered correctly with concurrent sleep (python) · at 00:44:44Z · abandoned wrun_41M3B0AMPB0GMV46FA91RC6TBZ
run-pickup-stall · hookWithSleepFinalStepWorkflow - step only on final payload (python) · at 00:44:45Z · abandoned wrun_41M3B0ANGD0GV3F8KC6J0E52EE
run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 00:44:45Z · abandoned wrun_41M3B0ANKE0GM3QNT9FNRQ7QMP
run-pickup-stall · sleepWithSequentialStepsWorkflow - sequential steps work with concurrent sleep (control) (python) · at 00:44:46Z · abandoned wrun_41M3B0AP8M0GMSJQVPFYZE13B4
run-pickup-stall · spawnWorkflowFromStepWorkflow - spawning a child workflow using start() inside a step (python) · at 00:45:20Z · abandoned wrun_41M3B0BQQR0GRTKMHJETAQ9809
run-pickup-stall · hookWithSleepWorkflow - hook payloads delivered correctly with concurrent sleep (python) · at 00:45:31Z · abandoned wrun_41M3B0C31S0GYE9TRG81YSSVX2
run-pickup-stall · hookWithSleepFinalStepWorkflow - step only on final payload (python) · at 00:45:32Z · abandoned wrun_41M3B0C3SW0GQTCP498WGNBWFQ
run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 00:45:46Z · abandoned wrun_41M3B0CGZ20GXYV1YA5YJ5E593
run-pickup-stall · sleepWithSequentialStepsWorkflow - sequential steps work with concurrent sleep (control) (python) · at 00:45:47Z · abandoned wrun_41M3B0CHS50GH93YZWTHAV0PKE
run-pickup-stall · abortTimeoutWorkflow: timeout cancels long-running step (python) · at 00:46:31Z · abandoned wrun_41M3B0DR1Y0GSV0Q3B2CF3WKZ0
run-pickup-stall · abortParallelWorkflow: abort cancels all parallel steps (python) · at 00:46:33Z · abandoned wrun_41M3B0DSBK0GR9V5MR2KBCZYAZ
run-pickup-stall · metadataFromHelperWorkflow - getWorkflowMetadata/getStepMetadata work from module-level helper (Vercel World Regression: getWorkflowMetadata() fully breaking runs #1577) (python) · at 00:47:19Z · abandoned wrun_41M3B0ECC50GTESMGHZ509D9M1
run-pickup-stall · abortReasonWorkflow: abort reason preserved across boundaries (python) · at 00:47:23Z · abandoned wrun_41M3B0EBJ20GMN28MTQNCPGVEK
run-pickup-stall · abortViaHookWorkflow: external hook triggers abort on in-flight step (python) · at 00:48:26Z · abandoned wrun_41M3B0FJMY0GYYF247FR8EFACF
run-pickup-stall · abortTimeoutWorkflow: timeout cancels long-running step (python) · at 00:48:59Z · abandoned wrun_41M3B0H48S0GZFPV4K6H84H1CT
run-pickup-stall · abortParallelWorkflow: abort cancels all parallel steps (python) · at 00:49:00Z · abandoned wrun_41M3B0HD7Y0GZH94AZENRFHF8X
run-pickup-stall · start: reserved-prefix initial attributes are seeded with allowReservedAttributes (python) · at 00:49:06Z · abandoned wrun_41M3B0J0R40GVDDGN602MZ1JM0
run-pickup-stall · start: initial attributes are seeded on run creation (python) · at 00:49:06Z · abandoned wrun_41M3B0J0E60GV0BXZNHBDMQ573
run-pickup-stall · setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly (python) · at 00:49:07Z · abandoned wrun_41M3B0J4VK0GZFFBYFCRFDDSQB
run-pickup-stall · setAttributesInsideStepWorkflow: step-body calls append attributed native events (python) · at 00:49:20Z · abandoned wrun_41M3B0JXQP0GKD3HXDXPVAVTJC
run-pickup-stall · abortReasonWorkflow: abort reason preserved across boundaries (python) · at 00:49:21Z · abandoned wrun_41M3B0JYVT0GH5CSFX8H9MAQHX
run-pickup-stall · metadataFromHelperWorkflow - getWorkflowMetadata/getStepMetadata work from module-level helper (Vercel World Regression: getWorkflowMetadata() fully breaking runs #1577) (python) · at 00:49:23Z · abandoned wrun_41M3B0K2520GRT6DQ7QK0GSEJW
run-pickup-stall · Promise.all of disjoint-key writes: every key lands (python) · at 00:49:34Z · abandoned wrun_41M3B0KEWR0GS3909FNBKN37W1
run-pickup-stall · abortViaHookWorkflow: external hook triggers abort on in-flight step (python) · at 00:49:36Z · abandoned wrun_41M3B0KHYP0GQM138AM221JC1P
run-pickup-stall · start: initial attributes are seeded on run creation (python) · at 00:49:58Z · abandoned wrun_41M3B0M78P0GH3B5EBY5S2E8QS
run-pickup-stall · start: reserved-prefix initial attributes are seeded with allowReservedAttributes (python) · at 00:50:03Z · abandoned wrun_41M3B0MC680GKJ7RNDJBNEVT2K
run-pickup-stall · setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly (python) · at 00:50:16Z · abandoned wrun_41M3B0MSET0GWEE5BZ8H06KDFG
run-pickup-stall · setAttributesInsideStepWorkflow: step-body calls append attributed native events (python) · at 00:50:23Z · abandoned wrun_41M3B0MZBY0GHRRFV9EPXY7HS2
run-pickup-stall · Promise.all of disjoint-key writes: every key lands (python) · at 00:50:23Z · abandoned wrun_41M3B0N00F0GNK5B0FRJCRRAXV
run-pickup-stall · workflow throws after awaited setAttributes: attribute still persists on the failed run (python) · at 00:50:29Z · abandoned wrun_41M3B0N6050GXPFK5PJ8AHANRF
run-pickup-stall · workflow throws after awaited setAttributes: attribute still persists on the failed run (python) · at 00:51:00Z · abandoned wrun_41M3B0P41N0GNHA33Z2Y02XV05

E2E Test Summary

Summary

	Passed	Failed	Skipped	Total
❌ ▲ Vercel Production	3816	62	739	4617
✅ 💻 Local Development	4238	0	550	4788
✅ 📦 Local Production	4238	0	550	4788
✅ 🐘 Local Postgres	4238	0	550	4788
✅ 🪟 Windows	340	0	2	342
✅ vercel-http-transport	873	0	153	1026
✅ vercel-multi-region	27	0	0	27
✅ vercel-ws-transport	591	0	93	684
Total	18361	62	2637	21060

Details by Category

❌ ▲ Vercel Production

App	Passed	Failed	Skipped
✅ astro-node	141	0	30
✅ astro-quickjs	141	0	30
✅ example-node	141	0	30
✅ example-quickjs	141	0	30
✅ express-node	141	0	30
✅ express-quickjs	141	0	30
✅ fastify-node	141	0	30
✅ fastify-quickjs	141	0	30
✅ hono-node	141	0	30
✅ hono-quickjs	141	0	30
✅ nest-node	141	0	30
✅ nest-quickjs	141	0	30
✅ nextjs-turbopack-node	168	0	3
✅ nextjs-turbopack-quickjs	168	0	3
✅ nextjs-webpack-node	168	0	3
✅ nextjs-webpack-quickjs	168	0	3
✅ nitro-node	141	0	30
✅ nitro-quickjs	141	0	30
✅ nuxt-node	141	0	30
✅ nuxt-quickjs	141	0	30
❌ python-node	4	62	105
✅ sveltekit-node	160	0	11
✅ sveltekit-quickjs	160	0	11
✅ tanstack-start-node	141	0	30
✅ tanstack-start-quickjs	141	0	30
✅ vite-node	141	0	30
✅ vite-quickjs	141	0	30

✅ 💻 Local Development

App	Passed	Skipped
✅ astro-stable-node	142	29
✅ astro-stable-quickjs	142	29
✅ express-stable-node	142	29
✅ express-stable-quickjs	142	29
✅ fastify-stable-node	142	29
✅ fastify-stable-quickjs	142	29
✅ hono-stable-node	142	29
✅ hono-stable-quickjs	142	29
✅ nest-stable-node	142	29
✅ nest-stable-quickjs	142	29
✅ nextjs-turbopack-canary-node	170	1
✅ nextjs-turbopack-canary-quickjs	170	1
✅ nextjs-turbopack-stable-node	170	1
✅ nextjs-turbopack-stable-quickjs	170	1
✅ nextjs-webpack-canary-node	170	1
✅ nextjs-webpack-canary-quickjs	170	1
✅ nextjs-webpack-stable-node	170	1
✅ nextjs-webpack-stable-quickjs	170	1
✅ nitro-stable-node	142	29
✅ nitro-stable-quickjs	142	29
✅ nuxt-stable-node	142	29
✅ nuxt-stable-quickjs	142	29
✅ sveltekit-stable-node	161	10
✅ sveltekit-stable-quickjs	161	10
✅ tanstack-start-node	142	29
✅ tanstack-start-quickjs	142	29
✅ vite-stable-node	142	29
✅ vite-stable-quickjs	142	29

✅ 📦 Local Production

App	Passed	Skipped
✅ astro-stable-node	142	29
✅ astro-stable-quickjs	142	29
✅ express-stable-node	142	29
✅ express-stable-quickjs	142	29
✅ fastify-stable-node	142	29
✅ fastify-stable-quickjs	142	29
✅ hono-stable-node	142	29
✅ hono-stable-quickjs	142	29
✅ nest-stable-node	142	29
✅ nest-stable-quickjs	142	29
✅ nextjs-turbopack-canary-node	170	1
✅ nextjs-turbopack-canary-quickjs	170	1
✅ nextjs-turbopack-stable-node	170	1
✅ nextjs-turbopack-stable-quickjs	170	1
✅ nextjs-webpack-canary-node	170	1
✅ nextjs-webpack-canary-quickjs	170	1
✅ nextjs-webpack-stable-node	170	1
✅ nextjs-webpack-stable-quickjs	170	1
✅ nitro-stable-node	142	29
✅ nitro-stable-quickjs	142	29
✅ nuxt-stable-node	142	29
✅ nuxt-stable-quickjs	142	29
✅ sveltekit-stable-node	161	10
✅ sveltekit-stable-quickjs	161	10
✅ tanstack-start-node	142	29
✅ tanstack-start-quickjs	142	29
✅ vite-stable-node	142	29
✅ vite-stable-quickjs	142	29

✅ 🐘 Local Postgres

App	Passed	Skipped
✅ astro-stable-node	142	29
✅ astro-stable-quickjs	142	29
✅ express-stable-node	142	29
✅ express-stable-quickjs	142	29
✅ fastify-stable-node	142	29
✅ fastify-stable-quickjs	142	29
✅ hono-stable-node	142	29
✅ hono-stable-quickjs	142	29
✅ nest-stable-node	142	29
✅ nest-stable-quickjs	142	29
✅ nextjs-turbopack-canary-node	170	1
✅ nextjs-turbopack-canary-quickjs	170	1
✅ nextjs-turbopack-stable-node	170	1
✅ nextjs-turbopack-stable-quickjs	170	1
✅ nextjs-webpack-canary-node	170	1
✅ nextjs-webpack-canary-quickjs	170	1
✅ nextjs-webpack-stable-node	170	1
✅ nextjs-webpack-stable-quickjs	170	1
✅ nitro-stable-node	142	29
✅ nitro-stable-quickjs	142	29
✅ nuxt-stable-node	142	29
✅ nuxt-stable-quickjs	142	29
✅ sveltekit-stable-node	161	10
✅ sveltekit-stable-quickjs	161	10
✅ tanstack-start-node	142	29
✅ tanstack-start-quickjs	142	29
✅ vite-stable-node	142	29
✅ vite-stable-quickjs	142	29

✅ 🪟 Windows

App	Passed	Failed	Skipped
✅ nextjs-turbopack-node	170	0	1
✅ nextjs-turbopack-quickjs	170	0	1

✅ vercel-http-transport

App	Passed	Skipped
✅ example	141	30
✅ express	141	30
✅ hono	141	30
✅ nextjs-turbopack	168	3
✅ nitro	141	30
✅ vite	141	30

✅ vercel-multi-region

App	Passed	Failed	Skipped
✅ nextjs-turbopack	27	0	0

✅ vercel-ws-transport

App	Passed	Skipped
✅ example	141	30
✅ express	141	30
✅ nextjs-turbopack	168	3
✅ vite	141	30

📋 View full workflow run

github-actions · 2026-09-24T23:33:49Z

Sim World

Simulated world deterministic testing for races. Traces

🟠 world-sim scenario book — 1 fail of 42 total

fence=per-spec

scenario	outcome	events	virt	replay
✅ `smoke-no-steps`	completed	3	0ms	ok
✅ `smoke-one-step`	completed	6	0ms	ok
✅ `hook-at-step-started`	completed	12	0ms	ok
✅ `hook-at-step-completed`	completed	12	0ms	ok
✅ `hook-at-hook-created`	completed	12	0ms	ok
✅ `deadline-hook-wins`	completed	7	1.0h	ok
✅ `deadline-expires`	completed	7	1.0h	ok
✅ `step-vs-timer-early-settlement`	completed	8	1.0h	ok
✅ `long-sleep`	completed	11	30.0d	ok
✅ `hook-never-arrives`	stalled	3	0ms	skipped
✅ `step-retries-twice`	completed	10	2.0s	ok
✅ `parallel-steps`	completed	9	0ms	ok
✅ `hook-on-execution-state`	completed	12	0ms	ok
✅ `peek-hook-before-branch`	completed	12	0ms	ok
✅ `peek-hook-after-branch`	completed	12	0ms	ok
✅ `peek-hook-at-registration`	completed	12	0ms	ok
✅ `race-hook-before-probe`	completed	12	0ms	ok
✅ `race-hook-after-probe`	completed	12	0ms	ok
✅ `race-duplicate-delivery`	completed	13	0ms	ok
✅ `attr-hook-before-step`	completed	11	0ms	ok
✅ `attr-hook-after-step`	completed	11	0ms	ok
✅ `attr-from-step-body`	completed	13	0ms	ok
✅ `fork-hook-after-timeout`	completed	14	1.0m	ok
✅ `fork-hook-before-timeout`	completed	14	1.0m	ok
✅ `count-hook-after-timeout`	completed	17	1.0m	ok
✅ `count-hook-before-timeout`	completed	20	1.0m	ok
✅ `stale-read-step-count-fork`	completed	20	1.0m	ok
✅ `stale-read-equal-step-counts`	completed	14	1.0m	ok
✅ `step-vs-step-fork`	completed	12	0ms	ok
✅ `step-vs-step-fork-fenced`	completed	12	0ms	ok
✅ `fence-catches-benign-direction`	completed	12	5ms	ok
✅ `in-flight-before-decision`	completed	17	1.0m	ok
❌ `in-flight-before-decision-counted`	completed	17	1.0m	ok
✅ `in-flight-after-decision`	completed	19	2.0m	ok
✅ `stale-read-step-count-fork-fenced`	completed	20	1.0m	ok
✅ `fork-hook-wins`	completed	13	1.0m	ok
✅ `fork-timeout-wins`	completed	13	1.0m	ok
✅ `unclaimed-payload-under-fork`	completed	17	1.0m	ok
✅ `claimed-payload-under-fork`	completed	17	1.0m	ok
✅ `writers-independent-step-bodies`	completed	12	0ms	ok
✅ `writers-scripted-tempo`	completed	12	0ms	ok
✅ `cancel-mid-step`	cancelled	7	0ms	skipped

Full trace: world-sim.txt

…er's log tail A replay republished the victim's wake only while the forced hook_created was the last row the claimer itself wrote, so any own row landing between the creation and the wake (a step or wait terminal from another invocation, or, since #4392, a row the same suspension writes alongside the creation) hid the debt, and a victim parked only on `await hook` never woke. Every replay now republishes for each forced hook_created in the loaded log whose createdAt is within 24 hours (the queue's message retention and idempotency window), under the existing `hook-force-claim-<hookId>` key. The node:vm handler sends it alongside the suspension's writes and at most once per hook per invocation; QuickJS once per invocation on load. Both use the shared rule in hook-wake.ts. Closes #4393. Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Pranay Prakash <1797812+pranaygp@users.noreply.github.com>

Both engines used to create forced hooks one token at a time, each group waiting on its victim wake before the next started. That barrier is gone (#4392); these tests hold the first forced create until the second is issued, so a serial implementation deadlocks, and check that each victim is still woken once under its own key. Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Pranay Prakash <1797812+pranaygp@users.noreply.github.com>

github-actions · 2026-09-25T00:09:39Z

📊 Workflow Benchmarks

commit a866bbe · Fri, 25 Sep 2026 00:39:45 GMT · run logs

Backend: vercel · app: nextjs-turbopack

Metric	Scenario	Best (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	step	1996 (+783%) 🔻	2254 🔴 (+29%) 🔻	2286 🔴 (+15%) 🔻	2498 🔴 (+16%) 🔻	30
TTFS	stream	2100 (+895%) 🔻	2205 🔴 (+30%) 🔻	2243 🔴 (+28%) 🔻	2421 🔴 (+37%) 🔻	30
TTFS	hook + stream	559 (+12%)	2566 🔴 (+28%) 🔻	5956 🔴 (+190%) 🔻	7279 🔴 (+226%) 🔻	30
Fan-out TTFS	Promise.all(100 steps)	469 (-15%) 💚	636 (-16%) 💚	702 (-22%) 💚	2652 (+8.2%)	10
Fan-out TTLS	Promise.all(100 steps)	1458 (-40%) 💚	3207 (-1.1%)	3331 (-55%) 💚	10556 (+28%) 🔻	10
STSO	1020 steps (inline)	127 (+15%) 🔻	147 (-1.3%)	160 (-1.8%)	198 (-32%) 💚	1019
WO	1020 steps	147821 (-1.1%)	147821 (-1.1%)	147821 (-1.1%)	147821 (-1.1%)	1
CRTT	first chunk (pooled)	62 (-10%)	89 (-29%) 💚	143 (-15%)	158 (-25%) 💚	28

Streams

Scenario	CRTT 1st	p75	p90	p99	CDV max	iters
paced control (100/s, 60B)	80.5 (-26%)	118 (-46%)	137 (-67%)	226 (-79%)	108 (-37%)	10
size sweep (100/s, 160B-12KB)	85 (-21%)	134 (-31%)	245 (-22%)	828 (+70%)	96 (-51%)	10
replay gateway-gpt-5.4-nano-2000t (1x)	70 (-29%)	113 (-86%)	140 (-94%)	219 (-93%)	97 (-80%)	3
replay eve-gpt-5.6-sol-2000t (1x)	89.5 (-35%)	128 (-33%)	164 (-38%)	335 (-59%)	311 (-51%)	2
replay eve-gpt-5.6-sol-2000t (2x)	82 (-31%)	134 (-39%)	169 (-40%)	285 (-38%)	224 (-24%)	3

📈 STSO distribution vs main (inline / queue-hop histograms)

1020 steps (inline)

Cumulative STSO time: main 149219ms → this run 147656ms (Δ -1563ms, -1%)

100-150 ms  ███████████████████████┃  main 768  this 811   +43
150-200 ms  █████┃                    main 218  this 198   -20
200-250 ms  ┃                         main  10  this   7    -3
250-300 ms  ┃                         main  14  this   1   -13
300-350 ms  ┃                         main   4  this   1    -3
350-400 ms  ┃                         main   3  this   1    -2
400-450 ms  ┃                         main   2  this   0    -2

📈 CRTT drill-down vs main (RTT distributions & profiles)

variant  RTT 1ms→5s+             avg         p50         p90         p99     n
control  ······██▁····  103.6 (-40%)  101 (-27%)  137 (-67%)  226 (-79%)  3000
sweep    ······▆█▁▁···  124.9 (-20%)  105 (-23%)  245 (-22%)  828 (+70%)  3000
gw 1x    ·····▁█▅▁····   95.5 (-73%)   93 (-46%)  140 (-94%)  219 (-93%)  5295
eve 1x   ·····▁██▁▁···  107.2 (-35%)   93 (-33%)  164 (-38%)  335 (-59%)  5186
eve 2x   ·····▁▅█▁····    117 (-34%)  113 (-31%)  169 (-40%)  285 (-38%)  7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):

control  █▄▆▁▃▇▆▁▂▂  98–112ms
sweep    ▃▂▂▂▄█▆▅▁▁  103–171ms
gw 1x    █▄▅▅▆▂▁▄▃▄  89–104ms
eve 1x   █▁▄▆▅▆█▇▆▂  94–116ms
eve 2x   ▃▁▅▂▁▆▇█▆▄  103–132ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):

sweep  ▂▅█▇▄▄▁  123–127ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):

control  ▅▃▄▁▃█▅▂▂▂  27–38ms
sweep    ▁▂▂▃█▃▅▄▂▁  39–64ms
gw 1x    ▆▃█▆▇█▄▁▂█  25–28ms
eve 1x   ▅▁▁▅▄██▃▅▁  20–24ms
eve 2x   ▆▄▁▄▅▄▃▁▄█  18–25ms

ℹ️ Metric definitions & methodology

_{Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.}

_{The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.}

_{The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.}

_{Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.}

_{Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)}

_{Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles}

_{Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000t eaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t 6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041}

_{🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600}

_{All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).}

_{Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.}

Signed-off-by: Peter Wielander <mittgfu@gmail.com>

github-actions · 2026-09-25T00:26:19Z

No backport to stable for ff80d62 (AI decision).

This fixes the victim-wake recovery for the experimental_force hook takeover, which does not exist on stable: packages/core/src/runtime/hook-wake.ts and quickjs-entrypoint.ts are absent there, stable's suspension-handler.ts has no force-claim path, and docs/content/docs/v5/api-reference/workflow/create-hook.mdx is not present. It also builds directly on #4392, another main-only change, so there is no defect on the maintenance line for it to repair.

To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:

ff80d62a59c7bf0f45c4552d7b0cde82beb2ebc4

Copilot AI lite review requested due to automatic review settings September 24, 2026 23:29

pranaygp requested a review from a team as a code owner September 24, 2026 23:29

Copilot AI reviewed Sep 24, 2026

vercel Bot deployed to Preview – workflow-web September 24, 2026 23:30 View deployment

vercel Bot deployed to Preview – example-workflow September 24, 2026 23:30 View deployment

vercel Bot deployed to Preview – workbench-hono-workflow September 24, 2026 23:30 View deployment

vercel Bot deployed to Preview – workbench-fastify-workflow September 24, 2026 23:30 View deployment

vercel Bot deployed to Preview – workbench-express-workflow September 24, 2026 23:30 View deployment

vercel Bot deployed to Preview – workbench-nestjs-workflow September 24, 2026 23:30 View deployment

vercel Bot deployed to Preview – workbench-nitro-workflow September 24, 2026 23:30 View deployment

vercel Bot deployed to Preview – workbench-vite-workflow September 24, 2026 23:30 View deployment

vercel Bot deployed to Preview – workbench-tanstack-start-workflow September 24, 2026 23:30 View deployment

vercel Bot deployed to Preview – example-nextjs-workflow-turbopack September 24, 2026 23:30 View deployment

vercel Bot deployed to Preview – workbench-sveltekit-workflow September 24, 2026 23:31 View deployment

vercel Bot deployed to Preview – workflow-tarballs September 24, 2026 23:31 View deployment

vercel Bot deployed to Preview – workbench-astro-workflow September 24, 2026 23:31 View deployment

vercel Bot deployed to Preview – workbench-nuxt-workflow September 24, 2026 23:31 View deployment

vercel Bot deployed to Preview – example-nextjs-workflow-webpack September 24, 2026 23:31 View deployment

vercel Bot deployed to Preview – workflow-docs September 24, 2026 23:31 View deployment

vercel Bot deployed to Preview – workflow-swc-playground September 24, 2026 23:32 View deployment

Base automatically changed from process-hook-registrations-concurrently-with-ste to main September 25, 2026 00:04

vercel Bot and others added 2 commits September 25, 2026 00:09

pranaygp force-pushed the fix-force-claim-wake-recovery branch from 840df4f to 2db67e3 Compare September 25, 2026 00:09

vercel Bot deployed to Preview – workbench-python-workflow September 25, 2026 00:09 View deployment

VaguelySerious reviewed Sep 25, 2026

View reviewed changes

Comment thread .changeset/force-claim-wake-recovery.md Outdated

Apply suggestion from @VaguelySerious

a866bbe

Signed-off-by: Peter Wielander <mittgfu@gmail.com>

VaguelySerious approved these changes Sep 25, 2026

View reviewed changes

vercel Bot deployed to Preview – workbench-python-workflow September 25, 2026 00:20 View deployment

vercel Bot deployed to Preview – workflow-web September 25, 2026 00:20 View deployment

vercel Bot deployed to Preview – example-nextjs-workflow-turbopack September 25, 2026 00:21 View deployment

vercel Bot deployed to Preview – workbench-nestjs-workflow September 25, 2026 00:21 View deployment

vercel Bot deployed to Preview – workbench-nitro-workflow September 25, 2026 00:21 View deployment

vercel Bot deployed to Preview – workbench-express-workflow September 25, 2026 00:21 View deployment

vercel Bot deployed to Preview – workbench-hono-workflow September 25, 2026 00:21 View deployment

vercel Bot deployed to Preview – workbench-fastify-workflow September 25, 2026 00:21 View deployment

vercel Bot deployed to Preview – example-workflow September 25, 2026 00:21 View deployment

vercel Bot deployed to Preview – workflow-tarballs September 25, 2026 00:21 View deployment

vercel Bot deployed to Preview – workbench-astro-workflow September 25, 2026 00:21 View deployment

vercel Bot deployed to Preview – workbench-vite-workflow September 25, 2026 00:21 View deployment

vercel Bot deployed to Preview – workbench-tanstack-start-workflow September 25, 2026 00:21 View deployment

vercel Bot deployed to Preview – workbench-sveltekit-workflow September 25, 2026 00:21 View deployment

vercel Bot deployed to Preview – workbench-nuxt-workflow September 25, 2026 00:21 View deployment

vercel Bot deployed to Preview – example-nextjs-workflow-webpack September 25, 2026 00:21 View deployment

vercel Bot deployed to Preview – workflow-docs September 25, 2026 00:22 View deployment

vercel Bot deployed to Preview – workflow-swc-playground September 25, 2026 00:24 View deployment

pranaygp merged commit ff80d62 into main Sep 25, 2026
178 of 181 checks passed

pranaygp deleted the fix-force-claim-wake-recovery branch September 25, 2026 00:25

Navigation Menu

fix(core): repay a force-claim victim's wake independent of the claimer's log tail - #4398

fix(core): repay a force-claim victim's wake independent of the claimer's log tail#4398
pranaygp merged 3 commits into
mainfrom
fix-force-claim-wake-recovery

pranaygp commented Sep 24, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

vercel Bot commented Sep 24, 2026 •

edited

Loading

Uh oh!

changeset-bot Bot commented Sep 24, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 24, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 24, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 25, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Sep 25, 2026

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

pranaygp commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Fix (option 1 from the issue)

The window bound: 24 h from the row's createdAt

Why not options 2 or 3

Docs, tests, changeset

Assumptions / open points

Docs Preview

Checks

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

vercel Bot commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

changeset-bot Bot commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

github-actions Bot commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🧪 E2E Test Results

❌ Failed E2E Tests

⚠️ Flaky E2E Tests (passed on retry)

🛠 Infra Events (absorbed by the harness)

E2E Test Summary

Uh oh!

github-actions Bot commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Sim World

Uh oh!

github-actions Bot commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📊 Workflow Benchmarks

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Sep 25, 2026

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pranaygp commented Sep 24, 2026 •

edited

Loading

The window bound: 24 h from the row's `createdAt`

vercel Bot commented Sep 24, 2026 •

edited

Loading

changeset-bot Bot commented Sep 24, 2026 •

edited

Loading

github-actions Bot commented Sep 24, 2026 •

edited

Loading

github-actions Bot commented Sep 24, 2026 •

edited

Loading

github-actions Bot commented Sep 25, 2026 •

edited

Loading