fix(world-postgres): leave job attempts past core's max-deliveries ceiling

[pranaygp]

Summary

Core writes run_failed (MAX_DELIVERIES_EXCEEDED) on delivery 49. Since #4421 it throws when that write fails transiently (429, 5xx, transport), so the queue can redeliver instead of acking and leaving the run running. world-postgres capped every Graphile job at exactly 49 attempts, so that throw used up the last attempt: the job was retired (attempts = max_attempts, never available again) and the run was stranded anyway.

This PR adds 24 attempts of headroom:

const CORE_MAX_DELIVERIES_EXCEEDED_ATTEMPT = 49;
const POST_CEILING_RETRY_ATTEMPTS = 24;
const MAX_GRAPHILE_JOB_ATTEMPTS =
  CORE_MAX_DELIVERIES_EXCEEDED_ATTEMPT + POST_CEILING_RETRY_ATTEMPTS;

The same constant already feeds the executor transfer's remaining-attempts math and the pg-boss migration floor, so those pick up the headroom without further changes.

Graphile's retry backoff is exp(least(attempts, 10)) seconds. From attempt 10 onward each retry waits about 6.1 h, so the 24 extra attempts keep retrying the terminal write for about 6 days. They cannot hot-loop, and nothing changes in steady state: core only uses these attempts once it is past its own ceiling, where it retries just the terminal write. A definitive failure (for example, a 400) still acks.

Without #4421 the change is harmless: delivery 49 acks either way, so the extra attempts go unused.

Fixes #4427.

Related, not fixed:

• world-postgres: step jobs are permanently retired by lost workers, not by failures #3992: attempts spent by lost workers. This is more budget, not a fix.
• world-postgres: maxAttempts is hardcoded to 3, so the retry budget cannot be tuned #3994: the retry cap is still not configurable, and stable still uses 3.

Tests

• queue.test.ts: new case leaves job attempts for redeliveries past core max deliveries. The two existing maxAttempts expectations are updated to 73. All three fail against main's queue.ts.
• pnpm --filter @workflow/world-postgres test: 314/314 pass.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
example-nextjs-workflow-turbopack	Ready	Preview, v0	Sep 29, 2026 6:18pm UTC
example-nextjs-workflow-webpack	Ready	Preview, v0	Sep 29, 2026 6:18pm UTC
example-workflow	Ready	Preview, v0	Sep 29, 2026 6:18pm UTC
workbench-astro-workflow	Ready	Preview, v0	Sep 29, 2026 6:18pm UTC
workbench-express-workflow	Ready	Preview, v0	Sep 29, 2026 6:18pm UTC
workbench-fastify-workflow	Ready	Preview, v0	Sep 29, 2026 6:18pm UTC
workbench-hono-workflow	Ready	Preview, v0	Sep 29, 2026 6:18pm UTC
workbench-nestjs-workflow	Ready	Preview, v0	Sep 29, 2026 6:18pm UTC
workbench-nitro-workflow	Ready	Preview, v0	Sep 29, 2026 6:18pm UTC
workbench-nuxt-workflow	Ready	Preview, v0	Sep 29, 2026 6:18pm UTC
workbench-python-workflow	Ready	Preview, v0	Sep 29, 2026 6:18pm UTC
workbench-sveltekit-workflow	Ready	Preview, v0	Sep 29, 2026 6:18pm UTC
workbench-tanstack-start-workflow	Ready	Preview, v0	Sep 29, 2026 6:18pm UTC
workbench-vite-workflow	Ready	Preview, v0	Sep 29, 2026 6:18pm UTC
workflow-docs	Ready	Preview, v0	Sep 29, 2026 6:18pm UTC
workflow-swc-playground	Ready	Preview, v0	Sep 29, 2026 6:18pm UTC
workflow-tarballs	Ready	Preview, v0	Sep 29, 2026 6:18pm UTC
workflow-web	Ready	Preview, v0	Sep 29, 2026 6:18pm UTC

[changeset-bot]

🦋 Changeset detected

Latest commit: ca100fd

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@workflow/world-postgres	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[github-actions]

📊 Workflow Benchmarks

commit ca100fd · Tue, 29 Sep 2026 18:35:07 GMT · run logs

Backend: vercel · app: nextjs-turbopack

Metric	Scenario	Best (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	step	2166 (+197%) 🔻	2436 🔴 (+123%) 🔻	2464 🔴 (+121%) 🔻	2859 🔴 (+137%) 🔻	30
TTFS	stream	2179 (+890%) 🔻	2415 🔴 (+130%) 🔻	2468 🔴 (+125%) 🔻	2576 🔴 (+81%) 🔻	30
TTFS	hook + stream	781 (-11%)	2766 🔴 (+108%) 🔻	3167 🔴 (+126%) 🔻	3227 🔴 (+107%) 🔻	30
Fan-out TTFS	Promise.all(100 steps)	704 (-19%) 💚	985 (-42%) 💚	1024 (-43%) 💚	7316 (+291%) 🔻	10
Fan-out TTLS	Promise.all(100 steps)	3167 (+38%) 🔻	3689 (+1.6%)	4642 (+23%) 🔻	15611 (+116%) 🔻	10
STSO	1020 steps (inline)	129 (+29%) 🔻	156 (+20%) 🔻	172 (+12%)	254 (+25%) 🔻	1019
WO	1020 steps	155545 (+18%) 🔻	155545 (+18%) 🔻	155545 (+18%) 🔻	155545 (+18%) 🔻	1
CRTT	first chunk (pooled)	74 (+2.8%)	100 (-30%) 💚	123 (-42%) 💚	140 (-61%) 💚	28

Streams

Scenario	CRTT 1st	p75	p90	p99	CDV max	iters
paced control (100/s, 60B)	84.5 (-14%)	124 (-35%)	150 (-50%)	300 (-39%)	108 (-52%)	10
size sweep (100/s, 160B-12KB)	86 (-17%)	123 (-46%)	161 (-54%)	358 (-43%)	119 (-44%)	10
replay gateway-gpt-5.4-nano-2000t (1x)	93 (-32%)	117 (-38%)	132 (-59%)	216 (-62%)	164 (-62%)	3
replay eve-gpt-5.6-sol-2000t (1x)	108 (+3%)	124 (-37%)	147 (-52%)	254 (-54%)	203 (-55%)	2
replay eve-gpt-5.6-sol-2000t (2x)	87 (-39%)	157 (-52%)	194 (-60%)	260 (-72%)	187 (-60%)	3

📈 STSO distribution vs main (inline / queue-hop histograms)

1020 steps (inline)

Cumulative STSO time: main 131526ms → this run 155171ms (Δ +23645ms, +18%)

100-150 ms  █████████████████┃██████  main 899  this 665  -234
150-200 ms  ███░░░░░┃                 main 108  this 325  +217
200-250 ms  ┃                         main   8  this  17    +9
250-300 ms  ┃                         main   2  this   6    +4
300-350 ms  ┃                         main   0  this   1    +1
350-400 ms  ┃                         main   1  this   2    +1
400-450 ms  ┃                         main   1  this   1    +0
700-750 ms  ┃                         main   0  this   1    +1
750-800 ms  ┃                         main   0  this   1    +1

📈 CRTT drill-down vs main (RTT distributions & profiles)

variant  RTT 1ms→5s+             avg         p50         p90         p99     n
control  ······▆█▁····  109.4 (-31%)  105 (-24%)  150 (-50%)  300 (-39%)  3000
sweep    ······▇█▁····  108.4 (-38%)  101 (-35%)  161 (-54%)  358 (-43%)  3000
gw 1x    ·····▁█▇▁····   99.9 (-37%)   98 (-28%)  132 (-59%)  216 (-62%)  5295
eve 1x   ·····▁▇█▁····    107 (-34%)  101 (-25%)  147 (-52%)  254 (-54%)  5186
eve 2x   ·····▁▃█▁····  132.4 (-43%)  126 (-42%)  194 (-60%)  260 (-72%)  7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):

control  █▇▂▂▁▇▃▁▅▃  101–120ms
sweep    ▅▅▂▂▃▁▃█▃▁  99–127ms
gw 1x    ▇▁██▁▅▂▆▁▁  96–105ms
eve 1x   ▁▃▅▄▃▅▆█▃▂  99–118ms
eve 2x   ▂▁▄▂▁▄▅█▃▃  118–164ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):

sweep  █▇██▆▂▁  105–110ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):

control  ▆▄▃▃▂█▃▁▄▅  29–41ms
sweep    ▁▅▂▆▃▁▆█▃▁  38–52ms
gw 1x    █▅▄▃▄▇▆█▁▅  25–32ms
eve 1x   ▂▅▁▃▇▄██▇█  18–24ms
eve 2x   ▆▃▂▂▄▁▄▁▂█  20–27ms

ℹ️ Metric definitions & methodology

_{Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.}

_{The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.}

_{The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.}

_{Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.}

_{Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)}

_{Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles}

_{Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000t eaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t 6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041}

_{🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600}

_{All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).}

_{Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.}

[github-actions]

🧪 E2E Test Results

✅ All tests passed

⚠️ Flaky E2E Tests (passed on retry)

These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.

• applies prepareCall before streaming (nextjs-webpack · vercel-prod / vercel / node / production)
• experimental_retention: 0 purges the run payloads once the run finishes (nitro · vercel-prod / vercel / node / production)

🛠 Infra Events (absorbed by the harness)

Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.

10 infra events

• run-pickup-stall · runs app-generated source against a registered step (tanstack-start) · at 18:19:33Z · abandoned wrun_01M3Q68ZB8HMHZMJK3S0CC10N8
• cold-start-warmup · suite warmup (tanstack-start) · at 18:19:40Z · abandoned wrun_01M3Q68ZCB2C62J5MHGT6H83VS
• run-pickup-stall · hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (nuxt) · at 18:22:13Z · abandoned wrun_01M3Q6DVC4JJY0K9WNWMFMGX3D
• run-pickup-stall · hookCleanupTestWorkflow - hook token reuse after workflow completion (nextjs-webpack) · at 18:27:34Z · abandoned wrun_01M3Q6QMZY56057TTEY2YQN4Z5
• run-pickup-stall · hookGetConflictWorkflow - awaiting hook.getConflict() registers hook without payload (nextjs-webpack) · at 18:27:43Z · abandoned wrun_01M3Q6QXS6GN0TXX4WVEWESYDS
• run-pickup-stall · 'hookGetConflictWithPriorStepWorkflow' - hook.getConflict() does not block step execution (nextjs-webpack) · at 18:27:43Z · abandoned wrun_01M3Q6QYB838KKQ8T4333WDJWY
• run-pickup-stall · customSerializationWorkflow - custom class serialization with WORKFLOW_SERIALIZE/WORKFLOW_DESERIALIZE (nextjs-webpack) · at 18:29:03Z · abandoned wrun_01M3Q6TC4QXMYNC1CX0GQJM1MY
• run-pickup-stall · instanceMethodStepWorkflow - instance methods with "use step" directive (nextjs-webpack) · at 18:29:04Z · abandoned wrun_01M3Q6TDNFBW6RBRS810H30902
• run-pickup-stall · crossContextSerdeWorkflow - classes defined in step code are deserializable in workflow context (nextjs-webpack) · at 18:29:05Z · abandoned wrun_01M3Q6TDVDEDDHVFN8ZHJJ0MD1
• run-pickup-stall · errorSubclassRoundTripWorkflow - first-class Error subclasses survive every serialization boundary (nextjs-webpack) · at 18:29:05Z · abandoned wrun_01M3Q6TEEEVMY2DCR32SKFJJ5C

E2E Test Summary

Summary

	Passed	Failed	Skipped	Total
✅ ▲ Vercel Production	3904	0	875	4779
✅ 💻 Local Development	4406	0	550	4956
✅ 📦 Local Production	4406	0	550	4956
✅ 🐘 Local Postgres	4406	0	550	4956
✅ 🪟 Windows	342	0	12	354
✅ 🌐 Cross-language Conformance	68	0	84	152
✅ dynamic-runs	0	0	0	0
✅ vercel-http-transport	879	0	183	1062
✅ vercel-multi-region	27	0	0	27
✅ vercel-ws-transport	595	0	113	708
Total	19033	0	2917	21950

Details by Category

✅ ▲ Vercel Production

App	Passed	Failed	Skipped
✅ astro-node	142	0	35
✅ astro-quickjs	142	0	35
✅ example-node	142	0	35
✅ example-quickjs	142	0	35
✅ express-node	142	0	35
✅ express-quickjs	142	0	35
✅ fastify-node	142	0	35
✅ fastify-quickjs	142	0	35
✅ hono-node	142	0	35
✅ hono-quickjs	142	0	35
✅ nest-node	142	0	35
✅ nest-quickjs	142	0	35
✅ nextjs-turbopack-node	169	0	8
✅ nextjs-turbopack-quickjs	169	0	8
✅ nextjs-webpack-node	169	0	8
✅ nextjs-webpack-quickjs	169	0	8
✅ nitro-node	142	0	35
✅ nitro-quickjs	142	0	35
✅ nuxt-node	142	0	35
✅ nuxt-quickjs	142	0	35
✅ python-node	66	0	111
✅ sveltekit-node	161	0	16
✅ sveltekit-quickjs	161	0	16
✅ tanstack-start-node	142	0	35
✅ tanstack-start-quickjs	142	0	35
✅ vite-node	142	0	35
✅ vite-quickjs	142	0	35

✅ 💻 Local Development

App	Passed	Failed	Skipped
✅ astro-stable-node	148	0	29
✅ astro-stable-quickjs	148	0	29
✅ express-stable-node	148	0	29
✅ express-stable-quickjs	148	0	29
✅ fastify-stable-node	148	0	29
✅ fastify-stable-quickjs	148	0	29
✅ hono-stable-node	148	0	29
✅ hono-stable-quickjs	148	0	29
✅ nest-stable-node	148	0	29
✅ nest-stable-quickjs	148	0	29
✅ nextjs-turbopack-canary-node	176	0	1
✅ nextjs-turbopack-canary-quickjs	176	0	1
✅ nextjs-turbopack-stable-node	176	0	1
✅ nextjs-turbopack-stable-quickjs	176	0	1
✅ nextjs-webpack-canary-node	176	0	1
✅ nextjs-webpack-canary-quickjs	176	0	1
✅ nextjs-webpack-stable-node	176	0	1
✅ nextjs-webpack-stable-quickjs	176	0	1
✅ nitro-stable-node	148	0	29
✅ nitro-stable-quickjs	148	0	29
✅ nuxt-stable-node	148	0	29
✅ nuxt-stable-quickjs	148	0	29
✅ sveltekit-stable-node	167	0	10
✅ sveltekit-stable-quickjs	167	0	10
✅ tanstack-start-node	148	0	29
✅ tanstack-start-quickjs	148	0	29
✅ vite-stable-node	148	0	29
✅ vite-stable-quickjs	148	0	29

✅ 📦 Local Production

App	Passed	Failed	Skipped
✅ astro-stable-node	148	0	29
✅ astro-stable-quickjs	148	0	29
✅ express-stable-node	148	0	29
✅ express-stable-quickjs	148	0	29
✅ fastify-stable-node	148	0	29
✅ fastify-stable-quickjs	148	0	29
✅ hono-stable-node	148	0	29
✅ hono-stable-quickjs	148	0	29
✅ nest-stable-node	148	0	29
✅ nest-stable-quickjs	148	0	29
✅ nextjs-turbopack-canary-node	176	0	1
✅ nextjs-turbopack-canary-quickjs	176	0	1
✅ nextjs-turbopack-stable-node	176	0	1
✅ nextjs-turbopack-stable-quickjs	176	0	1
✅ nextjs-webpack-canary-node	176	0	1
✅ nextjs-webpack-canary-quickjs	176	0	1
✅ nextjs-webpack-stable-node	176	0	1
✅ nextjs-webpack-stable-quickjs	176	0	1
✅ nitro-stable-node	148	0	29
✅ nitro-stable-quickjs	148	0	29
✅ nuxt-stable-node	148	0	29
✅ nuxt-stable-quickjs	148	0	29
✅ sveltekit-stable-node	167	0	10
✅ sveltekit-stable-quickjs	167	0	10
✅ tanstack-start-node	148	0	29
✅ tanstack-start-quickjs	148	0	29
✅ vite-stable-node	148	0	29
✅ vite-stable-quickjs	148	0	29

✅ 🐘 Local Postgres

App	Passed	Failed	Skipped
✅ astro-stable-node	148	0	29
✅ astro-stable-quickjs	148	0	29
✅ express-stable-node	148	0	29
✅ express-stable-quickjs	148	0	29
✅ fastify-stable-node	148	0	29
✅ fastify-stable-quickjs	148	0	29
✅ hono-stable-node	148	0	29
✅ hono-stable-quickjs	148	0	29
✅ nest-stable-node	148	0	29
✅ nest-stable-quickjs	148	0	29
✅ nextjs-turbopack-canary-node	176	0	1
✅ nextjs-turbopack-canary-quickjs	176	0	1
✅ nextjs-turbopack-stable-node	176	0	1
✅ nextjs-turbopack-stable-quickjs	176	0	1
✅ nextjs-webpack-canary-node	176	0	1
✅ nextjs-webpack-canary-quickjs	176	0	1
✅ nextjs-webpack-stable-node	176	0	1
✅ nextjs-webpack-stable-quickjs	176	0	1
✅ nitro-stable-node	148	0	29
✅ nitro-stable-quickjs	148	0	29
✅ nuxt-stable-node	148	0	29
✅ nuxt-stable-quickjs	148	0	29
✅ sveltekit-stable-node	167	0	10
✅ sveltekit-stable-quickjs	167	0	10
✅ tanstack-start-node	148	0	29
✅ tanstack-start-quickjs	148	0	29
✅ vite-stable-node	148	0	29
✅ vite-stable-quickjs	148	0	29

✅ 🪟 Windows

App	Passed	Failed	Skipped
✅ nextjs-turbopack-node	171	0	6
✅ nextjs-turbopack-quickjs	171	0	6

✅ 🌐 Cross-language Conformance

App	Passed	Failed	Skipped
✅ python	68	0	84

✅ dynamic-runs

App	Passed	Failed	Skipped
✅ astro-vercel	0	0	0
✅ example-vercel	0	0	0
✅ express-vercel	0	0	0
✅ fastify-vercel	0	0	0
✅ hono-vercel	0	0	0
✅ nest-vercel	0	0	0
✅ nextjs-turbopack-vercel	0	0	0
✅ nextjs-webpack-vercel	0	0	0
✅ nitro-vercel	0	0	0
✅ nuxt-vercel	0	0	0
✅ sveltekit-vercel	0	0	0
✅ tanstack-start-vercel	0	0	0
✅ vite-vercel	0	0	0

✅ vercel-http-transport

App	Passed	Failed	Skipped
✅ example	142	0	35
✅ express	142	0	35
✅ hono	142	0	35
✅ nextjs-turbopack	169	0	8
✅ nitro	142	0	35
✅ vite	142	0	35

✅ vercel-multi-region

App	Passed	Failed	Skipped
✅ nextjs-turbopack	27	0	0

✅ vercel-ws-transport

App	Passed	Failed	Skipped
✅ example	142	0	35
✅ express	142	0	35
✅ nextjs-turbopack	169	0	8
✅ vite	142	0	35

---

📋 View full workflow run

[github-actions]

Sim World

Simulated world deterministic testing for races. Traces

🟠 world-sim scenario book — 1 fail of 42 total

fence=per-spec

scenario	outcome	events	virt	replay	violations
✅ smoke-no-steps	completed	3	0ms	ok	0
✅ smoke-one-step	completed	6	0ms	ok	0
✅ hook-at-step-started	completed	12	0ms	ok	0
✅ hook-at-step-completed	completed	12	0ms	ok	0
✅ hook-at-hook-created	completed	12	0ms	ok	0
✅ deadline-hook-wins	completed	7	1.0h	ok	0
✅ deadline-expires	completed	7	1.0h	ok	0
✅ step-vs-timer-early-settlement	completed	8	1.0h	ok	0
✅ long-sleep	completed	11	30.0d	ok	0
✅ hook-never-arrives	stalled	3	0ms	skipped	0
✅ step-retries-twice	completed	10	2.0s	ok	0
✅ parallel-steps	completed	9	0ms	ok	0
✅ hook-on-execution-state	completed	12	0ms	ok	0
✅ peek-hook-before-branch	completed	12	0ms	ok	0
✅ peek-hook-after-branch	completed	12	0ms	ok	0
✅ peek-hook-at-registration	completed	12	0ms	ok	0
✅ race-hook-before-probe	completed	12	0ms	ok	0
✅ race-hook-after-probe	completed	12	0ms	ok	0
✅ race-duplicate-delivery	completed	13	0ms	ok	0
✅ attr-hook-before-step	completed	11	0ms	ok	0
✅ attr-hook-after-step	completed	11	0ms	ok	0
✅ attr-from-step-body	completed	13	0ms	ok	0
✅ fork-hook-after-timeout	completed	14	1.0m	ok	0
✅ fork-hook-before-timeout	completed	14	1.0m	ok	0
✅ count-hook-after-timeout	completed	17	1.0m	ok	0
✅ count-hook-before-timeout	completed	20	1.0m	ok	0
✅ stale-read-step-count-fork	completed	20	1.0m	ok	0
✅ stale-read-equal-step-counts	completed	14	1.0m	ok	0
✅ step-vs-step-fork	completed	12	0ms	ok	0
✅ step-vs-step-fork-fenced	completed	12	0ms	ok	0
✅ fence-catches-benign-direction	completed	12	5ms	ok	0
✅ in-flight-before-decision	completed	17	1.0m	ok	0
❌ in-flight-before-decision-counted	completed	17	1.0m	ok	0
✅ in-flight-after-decision	completed	19	2.0m	ok	0
✅ stale-read-step-count-fork-fenced	completed	20	1.0m	ok	0
✅ fork-hook-wins	completed	13	1.0m	ok	0
✅ fork-timeout-wins	completed	13	1.0m	ok	0
✅ unclaimed-payload-under-fork	completed	17	1.0m	ok	0
✅ claimed-payload-under-fork	completed	17	1.0m	ok	0
✅ writers-independent-step-bodies	completed	12	0ms	ok	0
✅ writers-scripted-tempo	completed	12	0ms	ok	0
✅ cancel-mid-step	cancelled	7	0ms	skipped	0

Full trace: world-sim.txt

[github-actions]

Framework	Flow route	Step reg.	Framework output
hono	265.9 KiB (+562 B)	97.2 KiB (+1.0 KiB)	1.98 MiB (+56.3 KiB)
nextjs-turbopack	273.3 KiB (+557 B)	426 B (±0)	979.1 KiB (+56.4 KiB)

About these numbers

Sizes are gzip; parentheses show the change against main.
Flow route and Step reg. gate this job, on raw bytes rather than the gzip shown, at max(2%, 50.0 KiB). Framework output is informational.

ca100fd · run

[TooTallNate]

LGTM. Reproduced #4427 against real Postgres 16 + graphile-worker 0.16.6: with maxAttempts: 49 a throw on delivery 49 retires the job (attempts=49/max_attempts=49); with 73 it stays retryable (next run ~6.1h out). All MAX_GRAPHILE_JOB_ATTEMPTS consumers (default addJob, executor transfer math, invoke's add_job, pg-boss migration floor) stay consistent. World-postgres tests pass.

Non-blocking follow-ups:

• Jobs enqueued before this release keep their stored max_attempts = 49 (and transfers carry it forward), so worth a note in the changeset.
• The 49 duplicates core's MAX_QUEUE_DELIVERIES + 1 across packages; a cross-reference comment in core/src/runtime/constants.ts (or a shared constant) would prevent silent drift.
• The new test only asserts > 49; a transfer test on the default cap (legacy job at attempt 49 → maxAttempts: 25, attemptOffset: 48) would cover more.
• Please rebase onto main so CI runs this together with #4421.

[pranaygp]

Thanks @TooTallNate. I addressed the follow-ups in 4b080c4:

• Stored caps on existing jobs: the changeset now says that jobs enqueued before this release keep max_attempts = 49, and that executor transfers carry that cap forward.
• Drift: core/src/runtime/constants.ts (MAX_QUEUE_DELIVERIES) and world-postgres' CORE_MAX_DELIVERIES_EXCEEDED_ATTEMPT now point at each other. I didn't make it a shared constant because world-postgres doesn't depend on @workflow/core.
• Transfer coverage: new test keeps post-ceiling headroom when transferring a job on the default cap. A legacy job at attempt 49 with no stored cap becomes an executor job with maxAttempts: 25 and attemptOffset: 48. It fails against the pre-PR queue.ts.
• Rebase: the branch already contains main with fix(core): don't strand runs on throttled writes past the delivery ceiling or in QuickJS inline claims #4421 (merged in 8625608), so CI runs the two together. I merged instead of rebasing to avoid a force push.

World-postgres: 316/316 tests pass locally.

[pranaygp]

CI status at 6c8f763, after one retrigger:

• Green: Unit Tests (ubuntu and windows; the previous run hit a pg-teardown flake and a world-local flake), all 28 E2E Local Postgres jobs, lint, and build.
• Red, and also red on main: 36 E2E Vercel Prod/HTTP/WS jobs, and therefore E2E Required Check. Every one fails on the same single test, createHook({ experimental_force: true }) › declines to take a token from a run whose runtime predates involuntary disposal. main's Tests runs fail on the same test at 54b48ef, 6ac572c (fix(core): don't strand runs on throttled writes past the delivery ceiling or in QuickJS inline claims #4421), 20aa656 and 35c5b32, which is the commit this branch merged. It is a world-vercel hook force-claim test (feat(hooks): createHook({ experimental_force: true }) takes a held token over #4193), and this PR touches only world-postgres.
• Tarballs Preview Smoke Checks: hit the 10-minute job timeout. It timed out the same way on fix(core): don't strand runs on throttled writes past the delivery ceiling or in QuickJS inline claims #4421.

[github-actions]

No backport to stable for efb58a0 (AI decision).

The defect this fixes does not exist on stable: I verified that packages/world-postgres/src/queue.ts on stable has no MAX_GRAPHILE_JOB_ATTEMPTS constant and caps Graphile jobs at 3 attempts (maxAttempts: 3, job.retry_limit ?? 3), so there is no 49-attempt cap colliding with core's max-deliveries ceiling. The fix is also only meaningful in combination with #4421's throw-on-transient-terminal-write behavior (the PR itself notes it is a no-op without it, and that "stable still uses 3"), so cherry-picking it would change stable's retry budget without fixing anything there.

To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:

efb58a0523d9dae11051729958af474237ab7d7e