fix(world-local,core): stop re-executing in-flight inline steps on redelivery - #4328
Conversation
…delivery World-local's queue defaulted to 30s undici headers/body deadlines (#3255), so a delivery running a longer inline step was declared failed and redelivered while the step was still executing. The redelivery kept attempt 1, re-entered turbo, and ran the lazy step body again concurrently, once per 35s of step runtime. - world-local: default WORKFLOW_LOCAL_{HEADERS,BODY}_TIMEOUT_MS to 0 (no deadline), mirroring world-postgres (#4114). The env vars now opt in. - core: route lazy and pre-claimed inline steps through runStepSingleFlight too, matching quickjs-entrypoint, so a redelivery of the same message waits for the running body instead of executing it again. - Add a workbench/vitest regression test that redelivers the start message mid-step and asserts the body is entered exactly once. Closes #3909 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-Authored-By: Peter Wielander <29887157+VaguelySerious@users.noreply.github.com>
🦋 Changeset detectedLatest commit: 2d2b2ab The changes in this PR will be included in the next version bump. This PR includes changesets to release 17 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results✅ All tests passed
|
| Passed | Failed | Skipped | Total | |
|---|---|---|---|---|
| ✅ ▲ Vercel Production | 3904 | 0 | 875 | 4779 |
| ✅ 💻 Local Development | 4406 | 0 | 550 | 4956 |
| ✅ 📦 Local Production | 4406 | 0 | 550 | 4956 |
| ✅ 🐘 Local Postgres | 4406 | 0 | 550 | 4956 |
| ✅ 🪟 Windows | 342 | 0 | 12 | 354 |
| ✅ 🌐 Cross-language Conformance | 68 | 0 | 84 | 152 |
| ✅ dynamic-runs | 0 | 0 | 0 | 0 |
| ✅ vercel-http-transport | 879 | 0 | 183 | 1062 |
| ✅ vercel-multi-region | 27 | 0 | 0 | 27 |
| ✅ vercel-ws-transport | 595 | 0 | 113 | 708 |
| Total | 19033 | 0 | 2917 | 21950 |
Details by Category
✅ ▲ Vercel Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-node | 142 | 0 | 35 |
| ✅ astro-quickjs | 142 | 0 | 35 |
| ✅ example-node | 142 | 0 | 35 |
| ✅ example-quickjs | 142 | 0 | 35 |
| ✅ express-node | 142 | 0 | 35 |
| ✅ express-quickjs | 142 | 0 | 35 |
| ✅ fastify-node | 142 | 0 | 35 |
| ✅ fastify-quickjs | 142 | 0 | 35 |
| ✅ hono-node | 142 | 0 | 35 |
| ✅ hono-quickjs | 142 | 0 | 35 |
| ✅ nest-node | 142 | 0 | 35 |
| ✅ nest-quickjs | 142 | 0 | 35 |
| ✅ nextjs-turbopack-node | 169 | 0 | 8 |
| ✅ nextjs-turbopack-quickjs | 169 | 0 | 8 |
| ✅ nextjs-webpack-node | 169 | 0 | 8 |
| ✅ nextjs-webpack-quickjs | 169 | 0 | 8 |
| ✅ nitro-node | 142 | 0 | 35 |
| ✅ nitro-quickjs | 142 | 0 | 35 |
| ✅ nuxt-node | 142 | 0 | 35 |
| ✅ nuxt-quickjs | 142 | 0 | 35 |
| ✅ python-node | 66 | 0 | 111 |
| ✅ sveltekit-node | 161 | 0 | 16 |
| ✅ sveltekit-quickjs | 161 | 0 | 16 |
| ✅ tanstack-start-node | 142 | 0 | 35 |
| ✅ tanstack-start-quickjs | 142 | 0 | 35 |
| ✅ vite-node | 142 | 0 | 35 |
| ✅ vite-quickjs | 142 | 0 | 35 |
✅ 💻 Local Development
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 148 | 0 | 29 |
| ✅ astro-stable-quickjs | 148 | 0 | 29 |
| ✅ express-stable-node | 148 | 0 | 29 |
| ✅ express-stable-quickjs | 148 | 0 | 29 |
| ✅ fastify-stable-node | 148 | 0 | 29 |
| ✅ fastify-stable-quickjs | 148 | 0 | 29 |
| ✅ hono-stable-node | 148 | 0 | 29 |
| ✅ hono-stable-quickjs | 148 | 0 | 29 |
| ✅ nest-stable-node | 148 | 0 | 29 |
| ✅ nest-stable-quickjs | 148 | 0 | 29 |
| ✅ nextjs-turbopack-canary-node | 176 | 0 | 1 |
| ✅ nextjs-turbopack-canary-quickjs | 176 | 0 | 1 |
| ✅ nextjs-turbopack-stable-node | 176 | 0 | 1 |
| ✅ nextjs-turbopack-stable-quickjs | 176 | 0 | 1 |
| ✅ nextjs-webpack-canary-node | 176 | 0 | 1 |
| ✅ nextjs-webpack-canary-quickjs | 176 | 0 | 1 |
| ✅ nextjs-webpack-stable-node | 176 | 0 | 1 |
| ✅ nextjs-webpack-stable-quickjs | 176 | 0 | 1 |
| ✅ nitro-stable-node | 148 | 0 | 29 |
| ✅ nitro-stable-quickjs | 148 | 0 | 29 |
| ✅ nuxt-stable-node | 148 | 0 | 29 |
| ✅ nuxt-stable-quickjs | 148 | 0 | 29 |
| ✅ sveltekit-stable-node | 167 | 0 | 10 |
| ✅ sveltekit-stable-quickjs | 167 | 0 | 10 |
| ✅ tanstack-start-node | 148 | 0 | 29 |
| ✅ tanstack-start-quickjs | 148 | 0 | 29 |
| ✅ vite-stable-node | 148 | 0 | 29 |
| ✅ vite-stable-quickjs | 148 | 0 | 29 |
✅ 📦 Local Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 148 | 0 | 29 |
| ✅ astro-stable-quickjs | 148 | 0 | 29 |
| ✅ express-stable-node | 148 | 0 | 29 |
| ✅ express-stable-quickjs | 148 | 0 | 29 |
| ✅ fastify-stable-node | 148 | 0 | 29 |
| ✅ fastify-stable-quickjs | 148 | 0 | 29 |
| ✅ hono-stable-node | 148 | 0 | 29 |
| ✅ hono-stable-quickjs | 148 | 0 | 29 |
| ✅ nest-stable-node | 148 | 0 | 29 |
| ✅ nest-stable-quickjs | 148 | 0 | 29 |
| ✅ nextjs-turbopack-canary-node | 176 | 0 | 1 |
| ✅ nextjs-turbopack-canary-quickjs | 176 | 0 | 1 |
| ✅ nextjs-turbopack-stable-node | 176 | 0 | 1 |
| ✅ nextjs-turbopack-stable-quickjs | 176 | 0 | 1 |
| ✅ nextjs-webpack-canary-node | 176 | 0 | 1 |
| ✅ nextjs-webpack-canary-quickjs | 176 | 0 | 1 |
| ✅ nextjs-webpack-stable-node | 176 | 0 | 1 |
| ✅ nextjs-webpack-stable-quickjs | 176 | 0 | 1 |
| ✅ nitro-stable-node | 148 | 0 | 29 |
| ✅ nitro-stable-quickjs | 148 | 0 | 29 |
| ✅ nuxt-stable-node | 148 | 0 | 29 |
| ✅ nuxt-stable-quickjs | 148 | 0 | 29 |
| ✅ sveltekit-stable-node | 167 | 0 | 10 |
| ✅ sveltekit-stable-quickjs | 167 | 0 | 10 |
| ✅ tanstack-start-node | 148 | 0 | 29 |
| ✅ tanstack-start-quickjs | 148 | 0 | 29 |
| ✅ vite-stable-node | 148 | 0 | 29 |
| ✅ vite-stable-quickjs | 148 | 0 | 29 |
✅ 🐘 Local Postgres
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 148 | 0 | 29 |
| ✅ astro-stable-quickjs | 148 | 0 | 29 |
| ✅ express-stable-node | 148 | 0 | 29 |
| ✅ express-stable-quickjs | 148 | 0 | 29 |
| ✅ fastify-stable-node | 148 | 0 | 29 |
| ✅ fastify-stable-quickjs | 148 | 0 | 29 |
| ✅ hono-stable-node | 148 | 0 | 29 |
| ✅ hono-stable-quickjs | 148 | 0 | 29 |
| ✅ nest-stable-node | 148 | 0 | 29 |
| ✅ nest-stable-quickjs | 148 | 0 | 29 |
| ✅ nextjs-turbopack-canary-node | 176 | 0 | 1 |
| ✅ nextjs-turbopack-canary-quickjs | 176 | 0 | 1 |
| ✅ nextjs-turbopack-stable-node | 176 | 0 | 1 |
| ✅ nextjs-turbopack-stable-quickjs | 176 | 0 | 1 |
| ✅ nextjs-webpack-canary-node | 176 | 0 | 1 |
| ✅ nextjs-webpack-canary-quickjs | 176 | 0 | 1 |
| ✅ nextjs-webpack-stable-node | 176 | 0 | 1 |
| ✅ nextjs-webpack-stable-quickjs | 176 | 0 | 1 |
| ✅ nitro-stable-node | 148 | 0 | 29 |
| ✅ nitro-stable-quickjs | 148 | 0 | 29 |
| ✅ nuxt-stable-node | 148 | 0 | 29 |
| ✅ nuxt-stable-quickjs | 148 | 0 | 29 |
| ✅ sveltekit-stable-node | 167 | 0 | 10 |
| ✅ sveltekit-stable-quickjs | 167 | 0 | 10 |
| ✅ tanstack-start-node | 148 | 0 | 29 |
| ✅ tanstack-start-quickjs | 148 | 0 | 29 |
| ✅ vite-stable-node | 148 | 0 | 29 |
| ✅ vite-stable-quickjs | 148 | 0 | 29 |
✅ 🪟 Windows
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack-node | 171 | 0 | 6 |
| ✅ nextjs-turbopack-quickjs | 171 | 0 | 6 |
✅ 🌐 Cross-language Conformance
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ python | 68 | 0 | 84 |
✅ dynamic-runs
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-vercel | 0 | 0 | 0 |
| ✅ example-vercel | 0 | 0 | 0 |
| ✅ express-vercel | 0 | 0 | 0 |
| ✅ fastify-vercel | 0 | 0 | 0 |
| ✅ hono-vercel | 0 | 0 | 0 |
| ✅ nest-vercel | 0 | 0 | 0 |
| ✅ nextjs-turbopack-vercel | 0 | 0 | 0 |
| ✅ nextjs-webpack-vercel | 0 | 0 | 0 |
| ✅ nitro-vercel | 0 | 0 | 0 |
| ✅ nuxt-vercel | 0 | 0 | 0 |
| ✅ sveltekit-vercel | 0 | 0 | 0 |
| ✅ tanstack-start-vercel | 0 | 0 | 0 |
| ✅ vite-vercel | 0 | 0 | 0 |
✅ vercel-http-transport
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ example | 142 | 0 | 35 |
| ✅ express | 142 | 0 | 35 |
| ✅ hono | 142 | 0 | 35 |
| ✅ nextjs-turbopack | 169 | 0 | 8 |
| ✅ nitro | 142 | 0 | 35 |
| ✅ vite | 142 | 0 | 35 |
✅ vercel-multi-region
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack | 27 | 0 | 0 |
✅ vercel-ws-transport
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ example | 142 | 0 | 35 |
| ✅ express | 142 | 0 | 35 |
| ✅ nextjs-turbopack | 169 | 0 | 8 |
| ✅ vite | 142 | 0 | 35 |
📊 Workflow Benchmarkscommit Backend:
Streams
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 131526ms → this run 191485ms (Δ +59959ms, +46%) 📈 CRTT drill-down vs main (RTT distributions & profiles)RTT over stream progress (avg per tenth of stream, bars scaled min→max): RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max): Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max): 📜 Previous results (1)5f56f85Wed, 23 Sep 2026 19:35:42 GMT · run logs
Streams
ℹ️ Metric definitions & methodologyStreams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach. The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000t 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor. |
Sim WorldSimulated world deterministic testing for races. Traces 🟠 world-sim scenario book — 1 fail of 42 total
Full trace: |
About these numbersSizes are gzip; parentheses show the change against
|
# Conflicts: # packages/world-local/src/queue.test.ts
karthikscale3
left a comment
There was a problem hiding this comment.
Reviewed the current diff. The single-flight change is focused, the regression test directly covers the duplicate inline-step execution race, and I found no blocking issues.
|
No backport to This is a genuine correctness fix (duplicate side effects from re-executing an in-flight inline step on redelivery), but it patches machinery that exists only on To override, re-run the Backport to stable workflow manually via |
closes #3909
Problem
Since #3255 (
@workflow/world-local@5.0.0-beta.33), world-local's queue gives each delivery a 30s undiciheadersTimeout/bodyTimeout. A delivery runs inline steps before it responds, so any"use step"longer than 30s had its live delivery declared failed and redelivered after a 5s backoff. That redelivery keepsx-vqs-message-attempt: 1, so the runtime re-enters turbo, skips the event log, and runs the same lazy inline step body again, concurrently. With the optimistic start, the body runs before the create-claim settles, so the losing claim discards only the result, not the side effects. The effect is one extra execution per ~35s of step runtime, and on world-local it leaves no trace in the durable log. I re-checked onmainat873b70b5cand both halves are still there.Fix
Two independent changes. Either one alone closes the report.
DEFAULT_HEADERS_TIMEOUT_MSandDEFAULT_BODY_TIMEOUT_MSgo from30_000to0, matching what fix(world-postgres): remove implicit queue HTTP deadlines #4114 did for world-postgres.WORKFLOW_LOCAL_HEADERS_TIMEOUT_MSandWORKFLOW_LOCAL_BODY_TIMEOUT_MSstill work, now as opt-in, and the docs warn that setting one below your longest inline step brings the duplication back.runStepSingleFlight. Before,runtime.tssent only owned-recovery steps through the in-process single-flight. Lazy and pre-claimed steps ran directly, on the assumption that their create-claim was the exactly-once gate.quickjs-entrypoint.tsalready sends every inline candidate through the single-flight, so this change makes the node:vm path match it. Fresh claims log atdebug, as they do in QuickJS. This covers any world whose transport can redeliver a live message, not just world-local.Tests
workbench/vitest/test/inline-step-ownership.test.tscase (Local world: 30s undici transport timeout redelivers a live queue message and re-executes an in-flight lazy-inline step #3909): it re-sends the run's start message 500ms into a 1.5s inline step and asserts the body is entered exactly once. Without the core change it fails (expected ['entered', 'entered'] to have a length of 1). With the change it passes.queue.test.tsnow asserts the defaults are0. The tests for a stalled handler and a stalled body already set their deadlines explicitly, so they now cover the opt-in path and I only renamed them.pnpm --filter @workflow/core test: 2622 passed.pnpm --filter @workflow/world-local test: 592 passed.workbench/vitest: 44 passed. Typecheck is clean.Not addressed
Turbo still treats
metadata.attempt === 1as proof that this is the first delivery (see the NOTE atruntime.ts:1237). With (2) in place the duplicate body is absorbed, but a redelivery still replays against an empty log. That deserves a separate decision: either count transport-failure redeliveries as attempts, or stop using the attempt number as that proof.🤖 Generated with Claude Code