Backport #4179: fix: retry backend connection failures and improve error logs

.changeset/transport-failure-not-user-error.md

@@ -0,0 +1,6 @@
+---
+'@workflow/core': patch
+'@workflow/world-vercel': patch
+---
+
+Route unrecognized backend connection and stream failures through existing retry policies, rebuilding shared event connections after repeated HTTP/2 failures. Keep invalid backend URLs, blocked ports, and unsupported request headers out of those retries. Include error cause chains in run-failure logs to expose underlying socket, DNS, and TLS failures.

docs/content/docs/foundations/errors-and-retries.mdx

@@ -164,6 +164,16 @@ export async function myWorkflow(input: unknown) {
 
 Uncaught, the run fails immediately with the `USER_ERROR` code — without retrying. See [serialization-failed](/docs/errors/serialization-failed) for common causes and fixes.
 
+## Backend Connection Failures
+
+On Vercel, backend connection failures and interrupted event streams use the existing retry policies, even when their error codes are unrecognized. This lets workflows recover from network failures instead of immediately failing with `USER_ERROR`. Persistent failures can still exhaust the retry budget.
+
+The SDK also replaces its shared events connection pool after repeated HTTP/2 session failures. Invalid backend URLs, including unsupported protocols and embedded credentials, fail immediately. Fetch requests to blocked ports or with unsupported headers (such as `Expect`) also fail without retrying.
+
+A connection failure does not prove that the backend rejected a write: it may have accepted it before the response was lost. Continue to make step side effects [idempotent](/docs/foundations/idempotency).
+
+Failed-run logs include the underlying error causes and their codes, exposing socket, DNS, or TLS errors behind messages such as `TypeError: fetch failed`. If a cause cannot be read, the log includes `[unavailable cause]` and the run can still be recorded as failed.
+
 ## Error Codes
 
 When a workflow run fails, the error may include a `code` that classifies the failure. You can access it programmatically via the `Run` class:

packages/core/README.md

@@ -1,3 +1,5 @@
 # @workflow/core
 
 Core runtime package for [Workflow SDK](https://useworkflow.dev).
+
+Failed-run logs include underlying error causes and codes to help diagnose failures such as socket, DNS, and TLS errors. Unreadable causes are marked without preventing the run from being recorded as failed.

packages/core/src/runtime.ts

@@ -56,7 +56,12 @@ import {
   withTraceContext,
   withWorkflowBaggage,
 } from './telemetry.js';
-import { getErrorName, getErrorStack, normalizeUnknownError } from './types.js';
+import {
+  formatErrorCauseChain,
+  getErrorName,
+  getErrorStack,
+  normalizeUnknownError,
+} from './types.js';
 import { buildWorkflowSuspensionMessage } from './util.js';
 import { runWorkflow } from './workflow.js';
 
@@ -919,6 +924,15 @@ export function workflowEntrypoint(
                     errorCode,
                     errorName,
                     errorStack,
+                    // Neither the message nor the stack reaches a wrapped
+                    // error's reason: `TypeError: fetch failed` carries an
+                    // empty message by design and a stack of pure
+                    // `node:internal/` frames, and the world layer's own
+                    // wrappers name the request that failed rather than what
+                    // failed about it. Undefined when there is no cause, so
+                    // the field disappears for an ordinary user throw.
+                    errorCause:
+                      formatErrorCauseChain(terminalError) || undefined,
                   });
 
                   // Fail the workflow run via event (event-sourced architecture)

packages/core/src/types.test.ts

@@ -0,0 +1,135 @@
+import { describe, expect, it } from 'vitest';
+import { formatErrorCauseChain } from './types.js';
+
+describe('formatErrorCauseChain', () => {
+  it.each([
+    'cause',
+    'name',
+    'message',
+    'code',
+    'errors',
+  ])('tolerates a throwing %s getter in the cause chain', (property) => {
+    const cause = new Error('inner');
+    Object.defineProperty(cause, property, {
+      get() {
+        throw new Error('getter failed');
+      },
+    });
+
+    expect(formatErrorCauseChain(new Error('outer', { cause }))).toContain(
+      '[unavailable cause]'
+    );
+  });
+
+  it('preserves readable links before an inaccessible cause', () => {
+    const { proxy, revoke } = Proxy.revocable({}, {});
+    revoke();
+    const middle = new Error('middle', { cause: proxy });
+
+    expect(formatErrorCauseChain(new Error('outer', { cause: middle }))).toBe(
+      'Error: middle\n[unavailable cause]'
+    );
+  });
+
+  it('tolerates a throwing getter on the outer cause', () => {
+    const error = new Error('outer');
+    Object.defineProperty(error, 'cause', {
+      get() {
+        throw new Error('getter failed');
+      },
+    });
+
+    expect(formatErrorCauseChain(error)).toBe('[unavailable cause]');
+  });
+
+  it('returns an empty string when there is no cause', () => {
+    expect(formatErrorCauseChain(new Error('boom'))).toBe('');
+    expect(formatErrorCauseChain('not an error')).toBe('');
+    expect(formatErrorCauseChain(undefined)).toBe('');
+  });
+
+  it('renders the wrapped reason a `fetch failed` hides', () => {
+    // The whole point: the outer error says nothing, the cause says
+    // everything. The outer link is skipped — the log already prints it.
+    const cause = Object.assign(new Error('other side closed'), {
+      name: 'SocketError',
+      code: 'UND_ERR_SOCKET',
+    });
+    const wrapper = new TypeError('fetch failed', { cause });
+
+    expect(formatErrorCauseChain(wrapper)).toBe(
+      'SocketError: other side closed (UND_ERR_SOCKET)'
+    );
+  });
+
+  it('renders each link of a multi-level chain, outermost first', () => {
+    const inner = Object.assign(
+      new Error('getaddrinfo ENOTFOUND ai-gateway.vercel.sh'),
+      { code: 'ENOTFOUND' }
+    );
+    const wrapper = new Error('POST /v4/… transport failure (ENOTFOUND)', {
+      cause: new TypeError('fetch failed', { cause: inner }),
+    });
+
+    expect(formatErrorCauseChain(wrapper)).toBe(
+      [
+        'TypeError: fetch failed',
+        // The code is already in the message, so it is not repeated.
+        'Error: getaddrinfo ENOTFOUND ai-gateway.vercel.sh',
+      ].join('\n')
+    );
+  });
+
+  it('summarizes the attempts an AggregateError collects', () => {
+    // A happy-eyeballs connect reports every address it tried on `errors`
+    // and leaves the AggregateError itself blank.
+    const aggregate = new AggregateError(
+      [
+        Object.assign(new Error('connect ECONNREFUSED 10.0.0.1:443'), {
+          code: 'ECONNREFUSED',
+        }),
+        Object.assign(new Error('connect ECONNREFUSED [::1]:443'), {
+          code: 'ECONNREFUSED',
+        }),
+      ],
+      ''
+    );
+
+    expect(
+      formatErrorCauseChain(new TypeError('fetch failed', { cause: aggregate }))
+    ).toBe(
+      [
+        'AggregateError',
+        'Error: connect ECONNREFUSED 10.0.0.1:443',
+        'Error: connect ECONNREFUSED [::1]:443',
+      ].join('\n')
+    );
+  });
+
+  it('caps a long chain', () => {
+    let error = new Error('innermost');
+    for (let i = 0; i < 8; i++) {
+      error = new Error(`level ${i}`, { cause: error });
+    }
+
+    const lines = formatErrorCauseChain(error).split('\n');
+    expect(lines).toHaveLength(5);
+    expect(lines.at(-1)).toBe('…');
+  });
+
+  it('stops on a cyclic chain', () => {
+    const inner = new Error('inner') as Error & { cause?: unknown };
+    const outer = new Error('outer', { cause: inner });
+    inner.cause = outer;
+
+    expect(formatErrorCauseChain(outer)).toBe(
+      ['Error: inner', 'Error: outer'].join('\n')
+    );
+  });
+
+  it('renders a non-error cause', () => {
+    expect(
+      formatErrorCauseChain(new Error('boom', { cause: 'a string' }))
+    ).toBe('a string');
+  });
+});

packages/core/src/types.ts

@@ -14,6 +14,92 @@ export function getErrorStack(v: unknown): string {
   return '';
 }
 
+/** Upper bound on the links {@link formatErrorCauseChain} renders. */
+const MAX_CAUSE_LINKS = 4;
+
+/** One `Name: message (CODE)` line for a link in a cause chain. */
+function describeErrorLink(value: unknown): string {
+  if (typeof value !== 'object' || value === null) {
+    return String(value);
+  }
+  const { name, message, code } = value as {
+    name?: unknown;
+    message?: unknown;
+    code?: unknown;
+  };
+  const label = typeof name === 'string' && name ? name : 'Error';
+  const text =
+    typeof message === 'string' && message ? `${label}: ${message}` : label;
+  return typeof code === 'string' && code && !text.includes(code)
+    ? `${text} (${code})`
+    : text;
+}
+
+/**
+ * Summarize the `cause` chain hanging off a thrown value, one link per line,
+ * outermost first. The value itself is skipped: whatever logs this already
+ * states it, in the message or in the stack header.
+ *
+ * `util.inspect` renders `[cause]` when Node prints an error, but the
+ * structured logs read `name` / `message` / `stack` and drop everything else
+ * — exactly the wrong half for errors that arrive pre-wrapped.
+ * `TypeError: fetch failed` is the canonical one: undici's wrapper says
+ * nothing on its own and its stack is all `node:internal/` frames, so the DNS,
+ * socket or TLS failure that actually happened is only readable one or two
+ * `cause` hops down. Same for the world layer's own wrapping, where the
+ * request that failed is on the wrapper and the reason it failed is on the
+ * cause.
+ *
+ * `AggregateError` also gets its `errors` summarized, because a
+ * happy-eyeballs connect reports every attempt there and leaves the
+ * `AggregateError` itself blank.
+ *
+ * Returns `''` when there is no cause, so callers can drop the field.
+ */
+export function formatErrorCauseChain(value: unknown): string {
+  const lines: string[] = [];
+  const seen = new Set<unknown>();
+
+  try {
+    for (
+      let current = causeOf(value);
+      current != null && lines.length <= MAX_CAUSE_LINKS;
+      current = causeOf(current)
+    ) {
+      if (typeof current !== 'object') {
+        lines.push(String(current));
+        break;
+      }
+      // A cause chain can loop (`err.cause = err`) or repeat a shared error.
+      if (seen.has(current)) break;
+      seen.add(current);
+      lines.push(describeErrorLink(current), ...aggregatedLinks(current));
+    }
+  } catch {
+    // Causes can contain getters or proxies that throw. Logging must not
+    // replace the original error or prevent the run_failed event from being written.
+    lines.push('[unavailable cause]');
+  }
+
+  return lines.length > MAX_CAUSE_LINKS
+    ? [...lines.slice(0, MAX_CAUSE_LINKS), '…'].join('\n')
+    : lines.join('\n');
+}
+
+function causeOf(value: unknown): unknown {
+  return typeof value === 'object' && value !== null
+    ? (value as { cause?: unknown }).cause
+    : undefined;
+}
+
+/** The attempts an `AggregateError` collected, if this link is one. */
+function aggregatedLinks(value: object): string[] {
+  const errors = (value as { errors?: unknown }).errors;
+  return Array.isArray(errors)
+    ? errors.slice(0, MAX_CAUSE_LINKS).map(describeErrorLink)
+    : [];
+}
+
 export interface NormalizedUnknownError {
   name: string;
   message: string;

packages/world-vercel/README.md

@@ -6,6 +6,12 @@ Integrates with Vercel's infrastructure for storage, queuing, and authentication
 
 Used by default for deployments on Vercel. Authentication and API endpoints are configured automatically in Vercel deployments.
 
+## Connection failures
+
+Backend connection failures and interrupted event streams follow existing retry policies, including failures with unrecognized error codes. Repeated HTTP/2 session failures rebuild the shared events connection pool. Invalid backend URL protocols, embedded credentials, Fetch-blocked ports, and unsupported request headers fail immediately.
+
+See [Backend Connection Failures](https://useworkflow.dev/docs/foundations/errors-and-retries#backend-connection-failures) for retry behavior and diagnostics.
+
 ## Custom dispatcher
 
 HTTP requests (including the queue) default to a shared undici `RetryAgent` that handles connection pooling and retries. Pass a custom `dispatcher` to override it — e.g. to tune undici on newer Node runtimes: