[DEV-106] Automatic failover across multiple agent harnesses when usage limits are hit

docs/code/configuration.md

@@ -392,6 +392,32 @@ Common `AGENT_HARNESS` values include `claude-code`, `opencode`, `codex`, `curso
 
 **Qwen note:** Qwen Code accepts a model via its `--model` flag (e.g. `qwen3-coder-plus`) — set it with `AGENT_MODEL`; you can also keep the model in `~/.qwen/settings.json`.
 
+### Failover across multiple harnesses
+
+`AGENT_HARNESS` accepts a comma-separated, priority-ordered list so the unattended worker keeps processing when an agent hits its usage limit:
+
+```bash
+# .devintern-code/.env
+AGENT_HARNESS=claude-code,codex
+```
+
+The first entry is your preferred harness; later entries are fallbacks in priority order. A single value behaves exactly as before.
+
+**Failover behavior (worker mode):**
+
+Applies to every unattended worker surface — fleet task polling, PR review addressing, `@mention` runs, conflict resolution, scheduled automations, estimations, dashboard retries, relay-driven tasks, and `devintern webhook serve` — not only the webhook queue.
+
+- At startup every entry is checked against the harness registry and your machine: unknown or not-installed entries produce a clear warning and are skipped, and the effective chain is logged (e.g. `Agent harness: claude-code → codex (failover enabled)`).
+- When the active harness reports a usage/rate limit, the worker records its reset window (parsed from the limit output; a 1-hour cooldown applies when no timer is parseable, e.g. monthly spend limits) and immediately retries the same work on the highest-priority harness that still has capacity.
+- When the primary harness's window elapses, the worker automatically fails back to it and logs the switch. Fallback agents hitting their own limits mid-run advance the chain again.
+- If every harness in the chain is limited at once, new agent work is deferred until the earliest window ends (the webhook queue pauses; polling/review/automation runs return to their next tick).
+- Failover state (active harness + per-harness windows) persists in the queue database, so restarting the worker resumes on the right harness instead of retrying a still-limited agent.
+- Which harness executed each run is recorded in run records, and `/health` on the webhook server reports the active harness, the chain, and open limit windows.
+
+Interactive one-shot runs you start yourself (`devintern TASK-123` in a terminal) always use the first (priority) entry; the worker pins each subprocess to the active harness so failover can switch the next attempt.
+
+**Per-harness overrides inside a list:** `<HARNESS>_CLI_PATH` (e.g. `CODEX_CLI_PATH`) resolves per active harness at spawn time. The global `AGENT_CLI_PATH` applies to the first entry only, so a stale global override cannot leak onto a fallback agent. `AGENT_MODEL` applies to whichever harness is active (the string is harness-specific).
+
 ### Model selection
 
 Set the model the agent harness runs with using `AGENT_MODEL` in `.devintern-code/.env`:

docs/code/worker.md

@@ -38,6 +38,10 @@ devintern worker
 devintern webhook serve
 ```
 
+## Agent failover
+
+Set `AGENT_HARNESS=codex,grok` (comma-separated, priority first) in the workspace `.env` so the worker keeps going when one agent hits a usage limit. Failover applies to every worker job: tracker tasks, PR review addressing, `@mention` runs, conflict resolution, scheduled automations, estimations, dashboard retries, and relay-driven work. Details: [Failover across multiple harnesses](./configuration.md#failover-across-multiple-harnesses).
+
 ## Recurring automations
 
 Put recurring work in `workspace.toml`. Set `repo` when the workspace has multiple repositories; it is optional for a one-repo workspace:

packages/agent-harness/src/detect-usage-limit.ts

@@ -20,10 +20,14 @@
  *   AI_RetryError: Failed after 4 attempts. Last error: Too Many Requests
  *   Too Many Requests: {"error":{"code":"1302","message":"Rate limit reached for req..."}}
  *   rate_limit_error / quota exceeded
+ *   Free usage exceeded, subscribe to Go
+ *   5 hour usage limit reached. It will reset in 5 hours 23 minutes. To continue
+ *     using this model now, enable usage from your available balance
  *
- * which the provider-rate-limit patterns below cover.
+ * which the subscription and provider-rate-limit patterns below cover.
  * Ref (opencode API rate-limit reporting):
  *   https://github.com/sst/opencode/issues/2398  (AI_RetryError: ... Too Many Requests)
+ *   https://github.com/anomalyco/opencode/blob/dev/packages/opencode/src/session/retry.ts
  *
  * OpenCode may keep `run` alive after a provider error and only expose the
  * diagnostic through `--print-logs`, as a timestamped line containing an
@@ -43,19 +47,30 @@ const USAGE_LIMIT_PATTERNS = [
   /^(?:(?:AI_(?:APICall|Retry)Error|error):\s*)?(?:(?:\d+[- ]hour\s+)?(?:usage|session|account|fast|opus|sonnet|fable 5|usage credit) limit reached|claude (?:ai )?usage limit(?: reached)?)(?:\s*(?:[.·—-]\s*)?(?:resets?|try again|available again|retry[- ]after)\b[^\n]*)?[.!]?$/i,
   // Claude Code 2.1.218 exports these as USAGE_LIMIT_ERROR_PREFIXES.
   /^(?:error:\s*)?(?:you(?:'|’)re out of (?:usage credits|extra usage)|your org is out of usage\s*·\s*(?:add funds to continue|contact your admin)|your seat type doesn(?:'|’)t include usage credits|your usage allocation has been disabled by your admin|your group(?:'|’)s usage limit is set to \$0)(?:\s*(?:[.·—-]\s*)?(?:resets?|try again|available again|retry[- ]after)\b[^\n]*)?[.!]?$/i,
-  // Codex UsageLimitReachedError variants. Keep these whole-line anchored:
-  // Codex writes its complete tool transcript to stderr.
-  /^you(?:'|’)ve hit your usage limit\.\s+(?:(?:upgrade to (?:pro|plus)|visit https:\/\/chatgpt\.com\/codex\/settings\/usage|contact your admin)\b[^\n]*?)(?:or\s+)?try again at\s+[^\n.]+[.]?$/i,
-  /^you(?:'|’)ve hit your usage limit for [^.]+\.\s+switch to another model now, or try again at\s+[^\n.]+[.]?$/i,
-  /^your workspace is out of credits\.\s+(?:add credits to continue|ask your workspace owner to add credits)[.]?$/i,
-  /^you hit your spend cap set in your workspace\.\s+(?:increase your spend cap to continue|ask your workspace owner to increase the spend cap)[.]?$/i,
+  // Codex UsageLimitReachedError Display (openai/codex protocol/src/error.rs).
+  // Plan promo copy and the reset suffix (`try again at 4:27 PM` vs `later`)
+  // vary; keep the distinctive prefix and allow the rest of the line. Codex
+  // writes its complete tool transcript to stderr, often with an `ERROR:` prefix.
+  /^(?:error:\s*)?you(?:'|’)ve hit your usage limit(?:\s+for [^.]+)?\.(?:\s+[^\n]*)?$/i,
+  /^your workspace is out of credits\.\s+(?:add credits to continue|ask your workspace owner to (?:add credits|refill in order to continue))[.]?$/i,
+  /^you hit your spend cap set (?:in your workspace|by the owner of your workspace)\.\s+(?:increase your spend cap to continue|ask (?:your workspace owner|an owner) to increase (?:your |the )?spend cap(?: to continue)?)[.]?$/i,
   /^quota exceeded\.\s+check your plan and billing details[.]?$/i,
   /^to use codex with your chatgpt plan, upgrade to plus\b[^\n]*$/i,
+  // OpenCode Go (anomalyco/opencode session/retry.ts): FreeUsageLimitError and
+  // GoUsageLimitError are rewritten into these headless retry messages.
+  /^(?:error:\s*)?free usage exceeded(?:,\s*subscribe to go)?[.]?$/i,
+  /^(?:(?:AI_(?:APICall|Retry)Error|error):\s*)?(?:(?:\d+[-\s]?hour|weekly|daily|monthly)\s+)?usage limit reached\.(?:\s+it will reset in [^\n.]+(?:\.|$))?(?:\s+to continue using this model now, enable usage from your available balance\b[^\n]*)?$/i,
+  /^subscription quota exceeded\.\s+you can continue using free models[.]?$/i,
   // Grok Build translates account/team exhaustion to these headless messages.
   /^you(?:'|’)ve hit the rate limit for your plan\.\s+upgrade your account or try again later[.]?$/i,
   /^you(?:'|’)ve hit your team(?:'|’)s api rate limit\.\s+ask a team admin to purchase more credits for higher limits, or try again later\.\s+see https:\/\/docs\.x\.ai\/developers\/rate-limits#rate-limit-tiers$/i,
   /^you(?:'|’)ve reached your free grok build usage limit for now\.\s+get supergrok for much higher limits, or try again later:\s+https:\/\/grok\.com\/supergrok\?referrer=grok-build$/i,
   /^resource-exhausted:\s+too many requests for team [^.]+\.\s+see https:\/\/console\.x\.ai\/team\/default\/rate-limits[.]?$/i,
+  // Grok Build paid-balance exhaustion: headless CLI prints a pretty-printed
+  // JSON Internal error (`"message": "API error (status 402 Payment Required): ..."`).
+  // Allow a JSON `"...": "` prefix so the diagnostic still matches inside that object.
+  /(?:^|:\s*")(?:api error \(status 402 payment required\):\s*)?grok build usage balance exhausted\b/i,
+  /(?:^|:\s*")api error \(status 402 payment required\)/i,
   // Goose maps provider HTTP 402 responses to this error and can exit zero.
   /^(?:error:\s*)?credits exhausted:\s+[^\n]+$/i,
   /^(?:⚠\s*)?individual quota reached\.\s+please upgrade your subscription to increase your limits[.]?$/i,
@@ -138,7 +153,10 @@ function isLikelyProviderDiagnostic(line: OutputLine): boolean {
     /^\[API Error:\s*429\b[^\n]*\]$/i.test(trimmed) ||
     /^(?:AI_RetryError|Too Many Requests)\b/i.test(trimmed) ||
     /^HTTP\s*429\b/i.test(trimmed) ||
-    /^\s*[{"[].*(?:rate_limit|quota|insufficient balance|add credits).*[}\]]\s*$/i.test(trimmed) ||
+    /^(?:internal error|error:\s*internal error)\b/i.test(trimmed) ||
+    /^\s*[{"[].*(?:rate_limit|quota|insufficient balance|add credits|payment required|usage balance exhausted).*[}\]]\s*$/i.test(
+      trimmed,
+    ) ||
     /\b(?:last error|provider (?:error|response)|response status|returned (?:an? )?(?:error|status)|request failed|retrying)\b/i.test(
       trimmed,
     )
@@ -198,11 +216,17 @@ function findUsageLimitLine(stdout: string, stderr: string): OutputLine | undefi
 
   return lines.find((line) => {
     const normalized = line.normalized.trim();
-    if (!normalized || isSourceOrDiffLine(normalized)) {
+    // Codex (and some other CLIs) prefix diagnostics with `ERROR:`. Strip it
+    // only for subscription-limit matching so a prefixed Codex message is
+    // still detected without treating arbitrary transcript lines as errors.
+    const diagnostic = normalized.replace(/^(?:error:\s*)/i, "");
+    if (!normalized || isSourceOrDiffLine(normalized) || isSourceOrDiffLine(diagnostic)) {
       return false;
     }
 
-    if (USAGE_LIMIT_PATTERNS.some((pattern) => pattern.test(normalized))) {
+    if (
+      USAGE_LIMIT_PATTERNS.some((pattern) => pattern.test(diagnostic) || pattern.test(normalized))
+    ) {
       return true;
     }
 

packages/agent-harness/src/harness-chain.ts

@@ -0,0 +1,242 @@
+/**
+ * Priority-ordered harness chain (`AGENT_HARNESS=claude-code,codex`).
+ *
+ * A comma-separated `AGENT_HARNESS` value names a failover chain: the first
+ * entry is the preferred harness, later entries are fallbacks used when an
+ * earlier one hits its usage/rate limit. Parsing and resolution live here so
+ * every consumer (worker failover, readiness probe, one-shot runs) sees the
+ * same list semantics:
+ *
+ *   - Entries are split on commas and trimmed; empty entries are dropped.
+ *   - Aliases (e.g. `agy`, deprecated `gemini`) resolve to canonical names.
+ *   - Duplicate canonical names collapse to the first occurrence, preserving
+ *     the requested priority.
+ *   - An empty/unset value defaults to `["claude-code"]`.
+ *
+ * Resolution validates each name against the registry, resolves its CLI path
+ * per harness (so `AGENT_CLI_PATH` cannot leak across harnesses — harness
+ * specific `<HARNESS>_CLI_PATH` overrides and defaults apply instead), and
+ * checks installability. Unknown or not-installed entries are reported as
+ * issues for the caller to warn about and skipped rather than fatal, unless
+ * that would leave an empty chain (then the full list is kept and the spawn
+ * itself surfaces the real error, matching single-harness behavior).
+ */
+
+import { DEFAULT_HARNESS_NAME, getHarness, HARNESS_ALIASES, listHarnesses } from "./registry.js";
+import { getHarnessCliCommand, isHarnessCliAvailable } from "./resolver.js";
+import type { AgentHarness } from "./types.js";
+
+/**
+ * Parse a raw (possibly comma-separated) harness list into canonical names.
+ *
+ * Applies registry aliases, drops empty entries, and de-duplicates canonical
+ * names keeping the first occurrence. Returns `[DEFAULT_HARNESS_NAME]` when
+ * nothing usable remains.
+ *
+ * @param raw - Raw `AGENT_HARNESS` value (already env-resolved), may be undefined.
+ * @returns Ordered canonical harness names (priority first).
+ */
+export function parseHarnessList(raw: string | undefined): string[] {
+  const names: string[] = [];
+  const seen = new Set<string>();
+  for (const part of (raw ?? "").split(",")) {
+    const requested = part.trim();
+    if (!requested) {
+      continue;
+    }
+    const canonical = getHarness(requested)?.name ?? requested;
+    if (seen.has(canonical)) {
+      continue;
+    }
+    seen.add(canonical);
+    names.push(canonical);
+  }
+  return names.length > 0 ? names : [DEFAULT_HARNESS_NAME];
+}
+
+/** One ordered, resolved entry of the harness chain. */
+export interface HarnessChainEntry {
+  /** Canonical registry name (after alias resolution). */
+  readonly name: string;
+  readonly harness: AgentHarness;
+  /** Resolved CLI command/path for this harness. */
+  readonly path: string;
+  /** Whether the resolved CLI is installed/reachable on this machine. */
+  readonly installed: boolean;
+}
+
+/** A chain entry that could not be used, with a warning message. */
+export interface HarnessChainIssue {
+  /** Name as written in the list (after alias resolution). */
+  readonly requested: string;
+  readonly reason: "unknown" | "not-installed";
+  readonly message: string;
+}
+
+/** Result of resolving a harness chain. */
+export interface ResolvedHarnessChain {
+  /** Ordered usable entries (priority first). */
+  readonly entries: HarnessChainEntry[];
+  /** Entries dropped from the chain, with warning messages. */
+  readonly issues: HarnessChainIssue[];
+  /** All canonical names as parsed from the raw value (before dropping). */
+  readonly parsed: string[];
+  /** True when the raw value listed more than one harness. */
+  readonly multiHarness: boolean;
+}
+
+/** Options for {@link resolveHarnessChain}. */
+export interface HarnessChainOptions {
+  /** Raw list value; defaults to `AGENT_HARNESS` env (or the default harness). */
+  raw?: string;
+  /**
+   * Probe installability and drop not-installed entries (default true).
+   * When false, every known entry is kept with `installed` reported as-is.
+   */
+  checkInstalled?: boolean;
+  /**
+   * When false, suppress deprecation warnings for aliased harness names
+   * (e.g. `gemini` → `antigravity`). Defaults to true.
+   */
+  warnDeprecated?: boolean;
+  /** Installability predicate override (tests). Defaults to PATH probing. */
+  isInstalled?: (entry: { name: string; path: string }) => boolean;
+}
+
+/**
+ * Resolve the CLI path for one parsed chain entry.
+ *
+ * Harness-specific `<HARNESS>_CLI_PATH` overrides and the harness default
+ * apply; the global `AGENT_CLI_PATH` only applies to the first (priority)
+ * entry so a single-value configuration keeps resolving exactly like
+ * {@link resolveHarness} does today, while a stale global override cannot
+ * stick to a fallback harness selected during failover.
+ *
+ * @param harness - Registered harness for the entry.
+ * @param isPrimary - Whether this is the first (priority) chain entry.
+ * @param warnDeprecated - Whether legacy path env vars may warn.
+ * @returns The CLI command or path to spawn/probe for this entry.
+ */
+function resolveEntryPath(
+  harness: AgentHarness,
+  isPrimary: boolean,
+  warnDeprecated?: boolean,
+): string {
+  if (isPrimary && process.env.AGENT_CLI_PATH) {
+    return process.env.AGENT_CLI_PATH;
+  }
+  return getHarnessCliCommand(harness, { warnDeprecated });
+}
+
+/**
+ * Resolve the priority-ordered harness chain from a comma-separated value.
+ *
+ * Unknown names and (when `checkInstalled`) not-installed CLIs are reported in
+ * `issues` and skipped; if nothing survives, the full parsed list is kept so
+ * spawning fails with the familiar actionable error instead of an empty chain.
+ *
+ * @param options - Raw value override, installability toggles, and test hooks.
+ * @returns The resolved chain with usable entries and dropped-entry issues.
+ */
+export function resolveHarnessChain(options: HarnessChainOptions = {}): ResolvedHarnessChain {
+  const raw = options.raw ?? process.env.AGENT_HARNESS;
+  const parsed = parseHarnessList(raw);
+  const checkInstalled = options.checkInstalled ?? true;
+  const warnDeprecated = options.warnDeprecated;
+
+  const entries: HarnessChainEntry[] = [];
+  const issues: HarnessChainIssue[] = [];
+
+  const buildEntry = (
+    canonical: string,
+    isPrimary: boolean,
+    reportIssues: boolean,
+    enforceInstalled: boolean,
+  ): HarnessChainEntry | null => {
+    const harness = getHarness(canonical);
+    if (!harness) {
+      if (reportIssues) {
+        const availableNames = listHarnesses()
+          .map((h) => `"${h.name}"`)
+          .join(", ");
+        issues.push({
+          requested: canonical,
+          reason: "unknown",
+          message: `Unknown agent harness "${canonical}" in AGENT_HARNESS; skipping it. Available harnesses: ${availableNames}.`,
+        });
+      }
+      return null;
+    }
+
+    const path = resolveEntryPath(harness, isPrimary, warnDeprecated);
+    const installed = options.isInstalled
+      ? options.isInstalled({ name: harness.name, path })
+      : isHarnessCliAvailable(path);
+
+    if (enforceInstalled && checkInstalled && !installed) {
+      if (reportIssues) {
+        const envKey = harness.name.toUpperCase().replace(/-/g, "_");
+        issues.push({
+          requested: canonical,
+          reason: "not-installed",
+          message: `Harness "${canonical}" is not installed (looked for "${path}"); skipping it. Install its CLI or set ${envKey}_CLI_PATH to the executable.`,
+        });
+      }
+      return null;
+    }
+
+    return { name: harness.name, harness, path, installed };
+  };
+
+  // Emit deprecation warnings for requested alias names (e.g. `gemini`),
+  // once per distinct alias, mirroring resolveHarness's behavior.
+  if (warnDeprecated !== false) {
+    const warnedAliases = new Set<string>();
+    for (const token of (raw ?? "").split(",")) {
+      const requested = token.trim();
+      const alias = requested ? HARNESS_ALIASES[requested] : undefined;
+      if (alias?.deprecated && alias.warning && !warnedAliases.has(requested)) {
+        warnedAliases.add(requested);
+        console.warn(`⚠️  ${alias.warning}`);
+      }
+    }
+  }
+
+  parsed.forEach((canonical, index) => {
+    const entry = buildEntry(canonical, index === 0, true, true);
+    if (entry) {
+      entries.push(entry);
+    }
+  });
+
+  if (entries.length === 0) {
+    if (parsed.every((canonical) => !getHarness(canonical))) {
+      // Nothing but unknown names: a configuration error, same as handing
+      // resolveHarness a single invalid name.
+      const availableNames = listHarnesses()
+        .map((h) => `"${h.name}"`)
+        .join(", ");
+      throw new Error(
+        `Unknown agent harness "${parsed.join(",")}" in AGENT_HARNESS. ` +
+          `Available harnesses: ${availableNames}. ` +
+          `Set AGENT_HARNESS to a comma-separated list of registered harnesses.`,
+      );
+    }
+    // Everything was dropped as not installed: keep the full parsed list so
+    // the spawn path reports the familiar "CLI not found" error instead of
+    // failing startup with an empty chain.
+    parsed.forEach((canonical, index) => {
+      const entry = buildEntry(canonical, index === 0, false, false);
+      if (entry) {
+        entries.push(entry);
+      }
+    });
+  }
+
+  return {
+    entries,
+    issues,
+    parsed,
+    multiHarness: parsed.length > 1,
+  };
+}