Add dynamic workflows, backed by encrypted ref storage

.changeset/dynamic-workflow-source.md

@@ -0,0 +1,12 @@
+---
+'@workflow/core': minor
+'@workflow/world': minor
+'@workflow/world-vercel': minor
+'@workflow/world-local': minor
+'@workflow/world-postgres': minor
+'@workflow/web-shared': minor
+'@workflow/cli': minor
+'workflow': minor
+---
+
+Add experimental dynamic workflows: `start()` accepts workflow source as a string, compiles and stores it with the run through the run-payload serialization pipeline, and replays from that stored code. Steps are exposed to the source through an explicit `experimental_dynamic.steps` map, which is not a security boundary: dynamic source runs with the deployment's full privileges. Off by default; a deployment opts in with `WORKFLOW_EXPERIMENTAL_DYNAMIC_WORKFLOWS=1`, and dynamic runs can only start on the current deployment.

.github/actions/report-vercel-e2e/action.yml

@@ -72,6 +72,7 @@ runs:
           e2e-${{ inputs.results-name }}.json
           e2e-${{ inputs.results-name }}.flaky.json
           e2e-metadata-${{ inputs.app }}-vercel.json
+          e2e-dynamic-runs-${{ inputs.app }}-vercel.json
           e2e-failures-${{ inputs.app }}-vercel.json
           e2e-infra-${{ inputs.app }}-vercel.json
           e2e-diagnostics-${{ inputs.app }}-vercel.json

.github/workflows/tests.yml

@@ -844,7 +844,7 @@ jobs:
         run: |
           export DEV_SERVER_LOG_PATH="$GITHUB_WORKSPACE/dev-server-${{ matrix.app.name }}-${{ matrix.app.artifactSuffix }}.log"
           rm -f "$DEV_SERVER_LOG_PATH"
-          (cd "$WORKBENCH_APP_PATH" && pnpm dev 2>&1 | tee "$DEV_SERVER_LOG_PATH") &
+          (cd "$WORKBENCH_APP_PATH" && WORKFLOW_EXPERIMENTAL_DYNAMIC_WORKFLOWS=1 pnpm dev 2>&1 | tee "$DEV_SERVER_LOG_PATH") &
           echo "starting tests in 10 seconds" && sleep 10
           pnpm vitest run packages/core/e2e/dev.test.ts; sleep 10
           pnpm run test:e2e --reporter=verbose --reporter=json --reporter=./packages/core/e2e/github-reporter.ts --outputFile=e2e-local-dev-${{ matrix.app.name }}-${{ matrix.app.artifactSuffix }}.json
@@ -857,6 +857,8 @@ jobs:
           WORKFLOW_DEV_HMR_LOGS: "1"
           NEXT_CANARY: ${{ matrix.app.canary && '1' || '' }}
           WORKFLOW_VM: ${{ matrix.app.vm || '' }}
+          # The server above opts in, so a dynamic E2E opt-in refusal fails.
+          WORKFLOW_E2E_EXPECT_DYNAMIC_WORKFLOWS: "1"
 
       - name: Generate E2E summary
         if: always()
@@ -946,7 +948,7 @@ jobs:
       - name: Run E2E Tests
         run: |
           export PROD_SERVER_LOG_PATH="$GITHUB_WORKSPACE/prod-server-${{ matrix.app.name }}-${{ matrix.app.artifactSuffix }}.log"
-          (cd "$WORKBENCH_APP_PATH" && pnpm start 2>&1 | tee "$PROD_SERVER_LOG_PATH") &
+          (cd "$WORKBENCH_APP_PATH" && WORKFLOW_EXPERIMENTAL_DYNAMIC_WORKFLOWS=1 pnpm start 2>&1 | tee "$PROD_SERVER_LOG_PATH") &
           echo "starting tests in 10 seconds" && sleep 10
           pnpm run test:e2e --reporter=verbose --reporter=json --reporter=./packages/core/e2e/github-reporter.ts --outputFile=e2e-local-prod-${{ matrix.app.name }}-${{ matrix.app.artifactSuffix }}.json
         env:
@@ -956,6 +958,8 @@ jobs:
           DEPLOYMENT_URL: "http://localhost:${{ matrix.app.name == 'sveltekit' && '4173' || (matrix.app.name == 'astro' && '4321' || '3000') }}"
           NEXT_CANARY: ${{ matrix.app.canary && '1' || '' }}
           WORKFLOW_VM: ${{ matrix.app.vm || '' }}
+          # The server above opts in, so a dynamic E2E opt-in refusal fails.
+          WORKFLOW_E2E_EXPECT_DYNAMIC_WORKFLOWS: "1"
 
       - name: Generate E2E summary
         if: always()
@@ -1055,7 +1059,7 @@ jobs:
       - name: Run E2E Tests
         run: |
           export PROD_SERVER_LOG_PATH="$GITHUB_WORKSPACE/prod-server-postgres-${{ matrix.app.name }}-${{ matrix.app.artifactSuffix }}.log"
-          (cd "$WORKBENCH_APP_PATH" && pnpm start 2>&1 | tee "$PROD_SERVER_LOG_PATH") &
+          (cd "$WORKBENCH_APP_PATH" && WORKFLOW_EXPERIMENTAL_DYNAMIC_WORKFLOWS=1 pnpm start 2>&1 | tee "$PROD_SERVER_LOG_PATH") &
           echo "starting tests in 10 seconds" && sleep 10
           pnpm run test:e2e --reporter=verbose --reporter=json --reporter=./packages/core/e2e/github-reporter.ts --outputFile=e2e-local-postgres-${{ matrix.app.name }}-${{ matrix.app.artifactSuffix }}.json
         env:
@@ -1065,6 +1069,8 @@ jobs:
           DEPLOYMENT_URL: "http://localhost:${{ matrix.app.name == 'sveltekit' && '4173' || (matrix.app.name == 'astro' && '4321' || '3000') }}"
           NEXT_CANARY: ${{ matrix.app.canary && '1' || '' }}
           WORKFLOW_VM: ${{ matrix.app.vm || '' }}
+          # The server above opts in, so a dynamic E2E opt-in refusal fails.
+          WORKFLOW_E2E_EXPECT_DYNAMIC_WORKFLOWS: "1"
 
       - name: Generate E2E summary
         if: always()

docs/content/docs/v5/advanced/dynamic-workflows.mdx

@@ -0,0 +1,224 @@
+---
+title: Dynamic Workflows
+description: Start a workflow run from source code that was not part of your build.
+type: conceptual
+summary: Pass workflow source to start() to run orchestration whose shape is only known after deployment.
+prerequisites:
+  - /docs/foundations/starting-workflows
+  - /docs/how-it-works/code-transform
+related:
+  - /docs/api-reference/workflow-api/start
+  - /docs/how-it-works/encryption
+  - /docs/configuration/runtime-tuning
+---
+
+<Callout type="warning">
+Dynamic workflows are **experimental** and **off by default**. The API may change without a major version bump. A deployment must opt in with `WORKFLOW_EXPERIMENTAL_DYNAMIC_WORKFLOWS=1`, dynamic runs can only start on the current deployment, and the World must support dynamic-source storage. See [Enabling dynamic workflows](#enabling-dynamic-workflows) and [World support](#world-support).
+</Callout>
+
+<Callout type="error">
+Dynamic source runs with the **full privileges of your deployment's functions**. It can read every environment variable, use the network and the filesystem, and call any step in the deployment. `experimental_dynamic.steps` is not a security boundary. Only pass source you would merge into your codebase. See [Security](#security).
+</Callout>
+
+Normally a workflow function is compiled into your build: the [code transform](/docs/how-it-works/code-transform) rewrites every `"use workflow"` function, the build bundles them, and `start()` names one by importing it.
+
+A dynamic workflow skips that. You hand `start()` a string of JavaScript, and it runs — no build, no deploy:
+
+```ts
+import { start } from 'workflow/api';
+import { fetchUser, sendEmail } from './steps';
+
+const run = await start(
+  `
+async function workflow(input) {
+  "use workflow";
+
+  const user = await steps.fetchUser(input.userId);
+  await steps.sendEmail(user.email);
+
+  return { ok: true };
+}
+`,
+  [{ userId: 'user_123' }],
+  {
+    experimental_dynamic: {
+      steps: { fetchUser, sendEmail },
+    },
+  }
+);
+
+console.log(await run.status); // 'running'
+```
+
+Only the *orchestration* is dynamic. Every step the source calls was deployed with your app, and `experimental_dynamic.steps` names the ones it calls by alias. That map does not stop source from reaching other steps; see [Security](#security). There is no way to define a new step from source.
+
+## When to use this
+
+Reach for dynamic workflows when the **shape** of the orchestration is only known after you deploy, and the source comes from code you trust as much as your own:
+
+- **Orchestration your application assembles** from reviewed templates, over a fixed set of deployed steps.
+- **Experiments** — try a new composition of existing steps without shipping a build.
+
+Dynamic workflows are not a way to run code written by your end users or generated by a model from their input. That source would run with your deployment's privileges; see [Security](#security).
+
+If your workflows are known at build time, use a normal workflow function. It has better types, better errors, no source validation, and no size limits.
+
+## Enabling dynamic workflows
+
+Dynamic workflows are off unless the deployment sets:
+
+```bash
+WORKFLOW_EXPERIMENTAL_DYNAMIC_WORKFLOWS=1
+```
+
+Only `1` or `true` (case-insensitive) enables them; any other value, or no value, leaves them off. The runtime reads the variable where workflows execute, when it needs it, so set it on the deployment or dev server rather than at build time. It controls three things:
+
+- **Starting.** `start()` with source throws before it contacts the World, creates a run, or enqueues anything unless this process has opted in.
+- **Delivery.** When a dynamic run reaches a deployment that has not opted in, the runtime does not execute its stored code. It fails the run with a `RUNTIME_ERROR` rather than retrying it.
+- **Health check.** A deployment advertises dynamic support in its [health check](/docs/api-reference/workflow-runtime/health-check) only when it has opted in.
+
+### Same deployment only
+
+A dynamic run must execute on the deployment that started it. `start()` rejects a dynamic start whose target differs from the current deployment. That includes an explicit `deploymentId` for another deployment, `deploymentId: 'latest'` when it resolves to a different deployment, and any concrete target when the current deployment cannot be determined. The rejection happens before any capability probe, key lookup, upload, run creation, or queue message.
+
+## What the source can use
+
+Dynamic source has no imports. Instead, the generated code predefines a small runtime surface:
+
+| Binding | What it is |
+| --- | --- |
+| `steps` | Frozen object of the aliases you passed in `experimental_dynamic.steps`. Calling one dispatches that registered step. |
+| `sleep` | The [durable sleep](/docs/api-reference/workflow/sleep) primitive. |
+| `createHook` | The [hook](/docs/foundations/hooks) primitive, for waiting on an external signal. |
+
+The source also runs inside the normal deterministic workflow VM, so the usual [workflow globals](/docs/api-reference/workflow-globals) — `Date`, `Math.random`, `crypto`, `URL`, `TextEncoder`, `structuredClone`, and the rest — are available with the same determinism guarantees as a static workflow.
+
+Dynamic source exposes only the small set of primitives injected by its generated wrapper. `createWebhook()` also needs the static workflow module's URL and metadata helper, and `getWritable()` needs its workflow-stream helper, so neither is currently injected into dynamic source. Use `createHook()` with server-side `resumeHook()`, and perform streaming through registered steps or a statically compiled workflow.
+
+Here is a longer example using a timer and a hook to wait for an approval:
+
+```ts
+import { start } from 'workflow/api';
+import { sendEmail } from './steps';
+
+const run = await start(
+  `
+async function workflow(input) {
+  "use workflow";
+
+  await sleep("15m");
+
+  const approval = createHook({ token: input.approvalToken });
+  const result = await Promise.race([
+    approval,
+    sleep("1d").then(() => ({ approved: false, timedOut: true })),
+  ]);
+
+  if (result.approved) {
+    await steps.sendEmail(input.email);
+  }
+
+  return result;
+}
+`,
+  [{
+    userId: 'user_123',
+    email: 'ada@example.com',
+    approvalToken: 'approval-req_01J...',
+  }],
+  {
+    experimental_dynamic: {
+      steps: { sendEmail },
+    },
+  }
+);
+```
+
+Supply a unique, deterministic approval token from the caller. The workflow must recreate the same token during replay, while the external service needs that token to call `resumeHook()`; do not use a tenant or user ID alone when concurrent runs can overlap.
+
+## Rules for the source
+
+`start()` validates the source before it writes anything, so a definition that could never run fails at the call site rather than on a queue delivery:
+
+- It must declare `async function workflow(...)`. Pass `experimental_dynamic.exportName` to use a different name; export names may contain letters, digits, and `_`, and cannot start with a digit.
+- The function's first statement must be the `"use workflow"` directive.
+- No `import` or `export`. Reach steps through `steps`, not through modules.
+- JavaScript only — no TypeScript syntax, no npm dependencies, no bundling.
+- No inline `"use step"` functions. Steps come from `experimental_dynamic.steps`.
+- At most 128 KB of source.
+- On Vercel, the run's execution context is limited to 2,048 bytes of JSON, and the `dynamicWorkflow` metadata below counts against it. That leaves room for roughly 30 step aliases, depending on how long the aliases and step IDs are. A start that exceeds it fails before anything is written.
+
+Everything a static workflow must obey still applies: the body has to be [deterministic](/docs/foundations/workflows-and-steps), and any side effect belongs in a step.
+
+## Workflow IDs
+
+You do not choose the workflow ID. It is derived from the source and its step bindings:
+
+```
+workflow//dynamic/<source-hash>//<exportName>
+```
+
+Two consequences worth knowing:
+
+- **The same definition always gets the same ID.** Runs of one generated workflow group together in [observability](/docs/observability) and share a queue topic, even across processes.
+- **A caller cannot claim an ID.** Because the hash covers the source *and* the step bindings, arbitrary source cannot be made to run under a static workflow's name — or under another definition's.
+
+Changing the source, or pointing an alias at a different step, produces a different workflow.
+
+## How the code is stored
+
+A dynamic run's workflow function is not in your deployment's bundle, so the run carries its own compiled workflow code — and replaying the run means replaying *that* code, not whatever your deployment contains now.
+
+That code uses the same serialization path as workflow inputs. It is compressed when the run protocol supports compression and compression is worthwhile, and encrypted when the World supplies run key material (see [Encryption](/docs/how-it-works/encryption)). Vercel's supported configuration provides encrypted storage; the Local and Postgres Worlds store it in plaintext. Retention and deletion apply whether the stored bytes are plaintext or ciphertext.
+
+When a run has key material, or was started with encryption, a delivery only executes code stored in the run's symmetric `encr` envelope. It refuses plaintext and sealed (`encp`) payloads and fails the run. Encryption keeps the code confidential; it does not prove who wrote it. See [Security](#security).
+
+On Vercel, durable workflow code storage is ref-backed on the run. The definition's size changes only how those bytes reach the backend:
+
+- **Small definitions** (the overwhelming majority) ride inline in the `run_created` request frame. The backend materializes those bytes into the run's ref-backed storage, with no upload request from `start()`.
+- **Larger definitions** are uploaded first, and `run_created` carries the resulting reference. This costs one extra request at `start()`.
+
+Both paths are transparent — there is nothing to configure. Here, “inline” describes request transport, not a second durable storage shape.
+
+Alongside the serialized code, the run records small plaintext metadata on `executionContext.dynamicWorkflow`: the source hash, the export name, and the alias-to-step-ID map. That is what lets a run be identified as dynamic without decoding the source. It is plaintext even when the code is encrypted, so anyone who can read the run can see which step IDs it was given and the aliases they were given under.
+
+## World support
+
+Dynamic workflows need a World that can store the run's workflow code.
+
+| World | Support |
+| --- | --- |
+| [Vercel](/worlds/vercel) | Encrypted, ref-backed storage (small definitions transported inline; large definitions uploaded first). Requires a backend that advertises dynamic-source storage. |
+| [Local](/worlds/local) | Stored in plaintext on the run record in the local filesystem store. |
+| [Postgres](/worlds/postgres) | Stored in plaintext on the run row. |
+| Others | Whatever the World explicitly attests through the versioned dynamic-workflow storage capability. |
+
+After the opt-in and same-deployment checks, `start()` checks the backend's dynamic-workflow storage capability. A backend that does not advertise it, including one that predates the capability endpoint, fails the start. On Vercel, `start()` then validates the final execution context against the 2,048-byte limit. All of this happens before serializing or uploading code, creating an event, or publishing a queue message.
+
+## Security
+
+<Callout type="warning">
+Dynamic source is **trusted application code** with the full privileges of your deployment's functions. The workflow VM is a determinism sandbox, not a security sandbox. Code in it can reach the host process: it can read every environment variable, use the network and the filesystem, and call any step registered in the deployment with any arguments.
+</Callout>
+
+- **`steps` is not a boundary.** The `steps` object contains only the aliases you passed and is frozen, so ordinary code that calls `steps.somethingElse()` fails the run instead of dispatching a step it was not given. Code that is trying to reach other steps, or the host, can.
+- **Only start source you would merge.** Do not build source from end-user input, and do not run model output generated from untrusted input. Either one gives whoever controls that input your deployment's privileges.
+- **Opting in is a deployment decision.** A deployment that sets `WORKFLOW_EXPERIMENTAL_DYNAMIC_WORKFLOWS` executes stored code for any dynamic run it receives. Anyone who can start a workflow on it can run code with its privileges.
+- **Encryption gives confidentiality only.** Where the World encrypts stored code, it cannot be read at rest without the run's key, and a delivery refuses code that is not encrypted with that key. Anyone who can obtain the run's key can still write valid code, so encryption does not replace the opt-in.
+- **Plaintext Worlds turn storage write access into code execution.** The Local and Postgres Worlds store the code in plaintext. On an opted-in deployment, anyone who can write to the Postgres database or the local data directory can make every worker execute code of their choosing.
+- **The step map is readable.** `executionContext.dynamicWorkflow.steps` stores the alias-to-step-ID map in plaintext, so anyone with read access to the run sees the step IDs the source was given.
+
+Treat dynamic source the way you would treat code in a pull request: written or reviewed by someone you trust with the deployment.
+
+## Limitations
+
+- Experimental — the API may change without a major version bump.
+- Off unless the deployment sets `WORKFLOW_EXPERIMENTAL_DYNAMIC_WORKFLOWS=1`.
+- Same-deployment starts only.
+- JavaScript only. No TypeScript syntax, npm dependencies, or bundling.
+- Steps must already be registered in the deployment; no runtime step registration.
+- No inline `"use step"` functions, `createWebhook`, or `getWritable`.
+- No caller-provided workflow IDs.
+- Parser-based validation checks JavaScript syntax and the required source/wrapper shape without executing it. It does not validate behavior, determinism, or intent.
+- On Vercel, roughly 30 step aliases fit the 2,048-byte execution-context limit.
+- Requires a World with dynamic-source storage.