[world-vercel] Report inline workflow step IDs per request - #4440
Conversation
Signed-off-by: Karthik Kalyanaraman <karthik.kalyanaraman@vercel.com>
🦋 Changeset detectedLatest commit: 685d976 The changes in this PR will be included in the next version bump. This PR includes changesets to release 21 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
📊 Workflow Benchmarkscommit Backend:
Streams
📈 STSO distribution (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: 197320ms over 1019 samples No 📈 CRTT drill-down (RTT distributions & profiles)No RTT over stream progress (avg per tenth of stream, bars scaled min→max): RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max): Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max): 📜 Previous results (2)43f297cSat, 26 Sep 2026 01:39:09 GMT · run logs
Streams
40b880bFri, 25 Sep 2026 22:57:23 GMT · run logs
Streams
ℹ️ Metric definitions & methodologyStreams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach. The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000t 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor. |
🧪 E2E Test Results✅ All tests passed 🛠 Infra Events (absorbed by the harness)Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
E2E Test SummarySummary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 🌐 Cross-language Conformance
✅ dynamic-runs
✅ vercel-http-transport
✅ vercel-multi-region
✅ vercel-ws-transport
|
Sim WorldSimulated world deterministic testing for races. Traces 🟠 world-sim scenario book — 1 fail of 42 total
Full trace: |
About these numbersSizes are gzip; parentheses show the change against
|
Signed-off-by: Karthik Kalyanaraman <karthik.kalyanaraman@vercel.com>
Signed-off-by: Karthik Kalyanaraman <karthik.kalyanaraman@vercel.com>
| * Worlds may use this synchronous hook to correlate step execution with the | ||
| * current platform invocation. Implementations must not throw. | ||
| */ | ||
| recordStepExecution?(stepId: string): void; |
There was a problem hiding this comment.
how worked up do we get about new World API surface area? should we have a write equivalent of analytics where we can namespace non-critical mutative calls?
There was a problem hiding this comment.
I think it's fine to add new methods if they're optional, though I wish this was at the end of the interface definition. Also, we could move this into analytics, but it's a bit different IMO, maybe if we have enough of these, we could do a telemetry or metrics sub-interface. Question is whether we should do this in advance assuming we'll have more
There was a problem hiding this comment.
I had the same thoughts as well. analytics is different. And, I am not sure if we will have more of these in the future - though it's possible. But, do we want to optimize for that future state now? I do not have strong opinions although I think we can run with this for now.
Co-authored-by: Peter Wielander <mittgfu@gmail.com> Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>
VaguelySerious
left a comment
There was a problem hiding this comment.
Approving assuming comments are addressed
… into codex/workflow-request-step-ids
|
No backport to This commit adds new capability rather than fixing a defect: it introduces a new optional To override, re-run the Backport to stable workflow manually via |
Description
Lets the Vercel World correlate a flow request with every workflow step whose body actually executes during that request.
Core exposes an optional World hook at the point immediately before step user code begins.
@workflow/world-vercelrecords those step IDs in request-scopedAsyncLocalStorageand returns the unique IDs, in execution order, throughx-vercel-internal-workflow-step-ids.The header is owned and emitted by the Vercel World, not the generic workflow entrypoint. Deliveries that already carry a scalar
stepIdkeep their existing behavior and do not receive the array header. Failed step bodies are included; skipped or unclaimed steps are not.How did you test your changes?
@workflow/worldand typechecked@workflow/coreand@workflow/world-vercelpnpm lintnode scripts/check-changesets.mjsPR Checklist - Required to merge
pnpm changesetwas run to create a changelog for this PRSigned-off-by)@vercel/workflowin a comment once the PR is ready, and the above checklist is complete