Shared Postgres world: queue namespaces don't isolate claiming, and startup recovery re-enqueues other applications' active runs

[mechiland]

Summary

When multiple independent applications share one @workflow/world-postgres database, they execute each other's workflow runs. Queue namespaces (WORKFLOW_QUEUE_NAMESPACE) do not prevent this, for two independent reasons:

1. Claiming ignores namespaces. The graphile-worker task identifiers are ${jobPrefix}flows / ${jobPrefix}steps — the namespace only prefixes the topic string inside the message. Any consumer polling the same database claims any namespace's jobs, and createTaskHandler then reconstructs the queue name using the consumer's own prefix (const queueName = ${queue}${messageData.id}``), so the foreign job is executed locally instead of being skipped.
2. Startup recovery is unscoped. reenqueueActiveRuns lists all pending/running runs with no namespace/deployment filter and re-enqueues each under the booting application's namespace prefix. So every application boot takes over every other application's in-flight runs. Fix namespaced active run recovery #2888 fixed which namespace the recovered runs are enqueued under, and [world-local] Scope untagged startup recovery to untagged runs #2667 scoped world-local recovery by tag — but the Postgres world still passes its raw, unfiltered storage.runs into recovery (packages/world-postgres's start()), so cross-application takeover remains. Notably, workflow_runs.deployment_id exists in the schema but world-postgres hardcodes getDeploymentId() to 'postgres', so it can't be used for scoping today.

Code references (checked against @workflow/world@5.0.0-beta.21 and @workflow/world-postgres@5.0.0-beta.27, the latest betas as of 2026-07-17):

• packages/world/src/recovery.ts — runs.list({ status, resolveData: 'none', ... }) with no scoping parameter.
• packages/world-postgres/src/queue.ts — getJobQueueName() derives the task identifier from jobPrefix only; createTaskHandler() rebuilds the queue name from the local prefix without validating the producer's namespace.
• packages/world-postgres/src/index.ts — start() → reenqueueActiveRuns(storage.runs, queue.queue, 'world-postgres', config.namespace).

How we hit this

We run a self-hosted platform that hosts many independent Eve agents (each is its own app/process), and initially pointed all of them at one WORKFLOW_POSTGRES_URL. Eve already derives a distinct WORKFLOW_QUEUE_NAMESPACE per agent, so topics were namespaced — yet agents intermittently executed each other's turns with the wrong application's code and credentials (surfacing as flaky provider/API-key errors, depending on which runner won the claim). Tracing it led to the two mechanisms above.

Repro sketch

1. Two apps A and B (any two workflow apps with same-named workflows — e.g. two instances of the same framework), same WORKFLOW_POSTGRES_URL, distinct WORKFLOW_QUEUE_NAMESPACE.
2. Start a long-running workflow in A.
3. While it is running, boot B. B logs [world-postgres] Re-enqueued N active run(s) on startup including A's run, under B's namespace.
4. B's runner claims and executes A's run (same workflow_flows task id; handler reconstructs the topic with B's prefix). With distinct WORKFLOW_POSTGRES_JOB_PREFIX values the takeover becomes deterministic: the re-enqueued foreign run can only be claimed by B.

Suggested directions

Either would resolve this class of issue:

• Scope claiming and recovery to the world's identity — e.g. include the namespace in the graphile task identifier (or validate the producer namespace in the handler and re-enqueue/skip on mismatch), and give runs.list a filter that reenqueueActiveRuns passes through (the deployment_id column looks like a natural hook, similar in spirit to the world-local tag scoping in [world-local] Scope untagged startup recovery to untagged runs #2667).
• Or, if a Postgres world database is intended to be exclusive to a single application, state that explicitly in the world-postgres docs — today nothing fails loudly when two applications share one database; they just silently cross-execute.

Workaround

We moved to one Postgres database per application (derived from a base URL, created + workflow-postgres-setup bootstrapped on demand). That fully isolates storage, queue, and recovery, and has been working well — sharing it here for other self-hosters who hit the same symptom.

[hikaru-212]

I dug into this locally and was able to reproduce both failure modes described here.

For the claim path, App A can create a run/job and App B can still claim it when both applications share the same Graphile task.

The job is then dispatched using B’s locally reconstructed queue namespace.

For startup recovery, starting App B can also cause reenqueueActiveRuns() to pick up pending/running runs created by App A, because the current Postgres run storage has no application-level ownership filter.

I added a local regression covering both cases.

Against the unpatched implementation I tested locally, both checks fail in the expected way:

• B dispatches an A-created run.
• B startup recovery re-enqueues A-created active runs.

I also built a local prototype to test one possible fix direction.

With that prototype, the same regression passes without changing the assertions, and the existing @workflow/world-postgres test suite still passes.

What seems to be missing is a durable application identity carried consistently across storage, queueing, dispatch, and recovery.

In the prototype, I currently:

• persist an application owner on workflow_runs
• scope startup recovery by that owner
• include the owner and the original full queue name in the Postgres queue message
• partition Graphile tasks and keyed jobs by owner
• validate the owner before dispatch
• validate the persisted run owner again before executing an existing run
• fail closed for legacy rows/jobs where ownership cannot be determined

Before turning this into a PR, I wanted to check what direction you would prefer for the ownership identity and migration shape.

Would a Postgres-specific applicationId be reasonable here, or would you prefer to extend/reuse an existing concept such as namespace or jobPrefix?

Persisting the owner on workflow_runs would also require a schema migration, so I’d rather align on that before polishing the patch.

For scope, this prototype is only addressing the cross-application isolation problem in #2978.

It is not intended to solve:

• parked-vs-runnable recovery selection: world-postgres: startup recovery re-enqueues parked runs and accumulates duplicate jobs #3119
• multi-replica boot coordination: Owner-aware / single-flight boot recovery for multi-instance self-hosted deployments #3138
• concurrent replay of a live run: world-postgres: startup recovery re-enqueues runs a LIVE peer process is executing — concurrent replay corrupts the event log #3758

[mechiland]

Thanks for digging in and reproducing both paths — that matches what we traced when we filed this.

A caveat up front: I'm a user, not a maintainer, so I can't tell you which direction they'd accept. What I can share is what we ended up doing, plus one thing I noticed re-reading main today that might save you a migration.

What we did. We couldn't wait on a fix, so we went two ways. Short term, one database per application, which is still what runs in production for us. Longer term we wrote our own World for the multi-tenant case (evelandhq/workflow-world), where a tenant id is persisted on every row and leads every query, and the embedded runner's Graphile task name is suffixed per tenant. One lesson from that which applies to your prototype: don't try to express "only claim my jobs" with Graphile flags. forbiddenFlags is a deny-list and can't say that. Partitioning the task identifier is what actually held for us.

What's already in main (as of 17bd649). This may be relevant to your "which identity" question:

• workflow_runs.deployment_id already exists and is NOT NULL. Every existing row holds 'postgres', because packages/world-postgres/src/queue.ts hardcodes getDeploymentId() to that string.
• Since fix(core): re-route runs delivered to the wrong deployment #2960 (August), core has guardDeploymentAffinity in packages/core/src/runtime/deployment-guard.ts. It compares run.deploymentId with world.getDeploymentId() before every delivery, re-routes a misrouted message to the pinned deployment with a bounded retry budget, and records run_failed with DEPLOYMENT_MISMATCH when the budget is spent. That looks like the fail-closed validation your prototype does by hand.
• It's gated on capabilities.deploymentAffinity === true, which world-postgres doesn't declare, so it's inert there today.
• QueueOptions.deploymentId exists at the interface level; world-postgres ignores it.

So a possible shape with no schema change: make the deployment id configurable (config.deploymentId / WORKFLOW_POSTGRES_DEPLOYMENT_ID), add a deploymentId filter to runs.list (#3707 is touching that param right now), pass it from reenqueueActiveRuns, suffix the task identifier with it, honour opts.deploymentId in queue(), and declare the capability. Whether the maintainers prefer that over a separate applicationId column I honestly don't know; it just seemed worth pointing out that the column and the guard already exist.

Two things from the operator side. Not objections, just what I'd worry about running it:

• Failing closed on legacy rows by default would strand in-flight runs for every single-app install on upgrade, which is most installs. Keeping today's behaviour when no id is configured, plus an explicit one-time adoption step for existing 'postgres' rows, would have been what I wanted as a user.
• I'd avoid the word "owner" for this. Owner-aware / single-flight boot recovery for multi-instance self-hosted deployments #3138 and world-postgres: startup recovery re-enqueues runs a LIVE peer process is executing — concurrent replay corrupts the event log #3758 need a dynamic owner (which live process holds the run). This issue needs a static one (which application the run belongs to). Same word for both will make those threads harder to follow.

[hikaru-212]

Thanks — the deploymentId suggestion and the upgrade concerns were really helpful. I went back and checked the deployment-affinity path and its existing runtime tests in more detail.

With deploymentAffinity enabled, the contract treats a deployment ID as one concrete/immutable build. The runtime tests enforce that pin: a run persisted for deployment A but delivered under deployment B is rerouted toward A, and if rerouting is exhausted the run is failed with DEPLOYMENT_MISMATCH.

That makes me still lean toward a separate, stable applicationId for the cross-redeployment application boundary in #2978, rather than reusing deploymentId for both concepts.

I do agree with your upgrade concern, though. I’d change the rollout shape from what I originally prototyped:

• preserve the existing behavior when no application ID is configured
• enable strict application isolation only with an explicit applicationId
• provide an explicit adoption/compatibility path for legacy rows/jobs rather than failing them closed by default
• coordinate the transition so old/unconfigured workers can’t continue doing unscoped recovery while strict isolation is being enabled

Also agreed on terminology — I’ll use “application identity/scope” rather than “owner” here, to keep it separate from the live-process ownership discussed in #3138/#3758.

This is still just the direction I’m leaning after checking the current implementation/tests; maintainer feedback on the identity and migration shape would still be very welcome.