[DEV-106] Automatic failover across multiple agent harnesses when usage limits are hit

[danii1]

Task: DEV-106

Summary: Automatic failover across multiple agent harnesses when usage limits are hit

Implementation Details

I'll start by exploring the existing codebase to understand the structures mentioned in the task.
Now let me look at supporting files: agent-model, run-recorder, and the docs/env example.
Now let me check a few more call sites and existing test patterns.
I now have a full picture. Let me set up a task list and implement.
Now creating the harness-chain module in agent-harness:
Let me refine that module — the fallback path is convoluted:
Now update resolveHarness in resolver.ts to handle comma-separated AGENT_HARNESS:
parseHarnessList canonicalizes aliases, which would break the deprecation-warning path in resolveHarness (it checks the raw name). Let me use the raw first token there instead:
There's now a circular import (resolver ↔ harness-chain), and import/no-cycle is an error in this repo. Let me move DEFAULT_HARNESS_NAME into registry.ts:
Now export the new module from the agent-harness index:
Now the failover state machine:
Now add active-harness persistence to webhook-queue.ts:
Before wiring the server, let me check how the webhook integration test drives this module:
Let me check how processReviewAsync's internal catch interacts with UsageLimitError:
The UsageLimitError thrown in processReviewAsync appears swallowed by its own catch. Let me check git history to understand:
Confirmed: the internal catch swallows UsageLimitError, making the wrapper's pause logic unreachable — I'll need to re-throw it there for failover to work. Now let me wire the server. First, imports and module state:
Now replace the module state and the pause/resume functions:
Edge case: resolveHarnessChain can return empty entries when every name is unknown, which would make HarnessFailover throw an unhelpful error. An all-unknown chain should throw the familiar config error instead:
Now update the catch blocks and spawn sites:
Now the re-throw in processReviewAsync's catch so the wrapper can fail over:
Now update the startup logic in startWebhookServer:
Update the health endpoint to surface failover state:
Now fix the run-record harness name in review-polling-acquirer:
Dependencies aren't installed in this worktree. Let me install:
Typecheck passes. Now let me check harness defaultPath values for tests, and the webhook-queue test pattern:
Now write the harness-chain tests:
Two issues: the all-not-installed fallback still applies the installability drop, and the alias deprecation warning is never emitted by the chain resolver. Let me fix both:
All 23 pass. Now the failover tests:
Two real issues: my test helper captures active by value (closure bug), and restore doesn't clear stale persisted rows. Fixing the helper first:
Now fix restore to clear stale/expired persisted rows:
All pass. Now run the full suites for both packages:
All green. Now docs. Let me view the relevant sections:
Now the .env.example files:
Docs done. Let me now review the full webhook-server diff for correctness:
One stale comment references initializeFailover (I named it ensureFailover). Also I need to verify the pruning semantics in windows()/earliestResetMs() — the timer callback iterates windows() and expects expired entries to still be there:
windows() prunes expired entries, but the timer callback needs to see them to log the failback. Let me switch to pure time-comparison semantics (no pruning):
Fix the stale comment in webhook-server:
The only lint error is in my new test file (inline type specifier). Fixing:
Lint exits 0 now. Let me check that my files add no new warnings and re-run everything from root:
Let me verify run-record harness metadata in the CLI flow, and do a final review of the new modules:
The CLI task pipeline doesn't record the harness in run records yet — the AC asks for this. Let me find how the workflow resolves its harness:
All pass. Final review of the complete new modules:
Two small cleanups — a misleading doc phrase in restore and a redundant re-read in reportUsageLimit:
The startup block is solid. Final full verification across the monorepo:
DEV-106 is implemented and verified (typecheck, lint, format, and all package test suites pass — 40 new tests added).

What was built

@devintern/agent-harness

• src/harness-chain.ts (new): parseHarnessList (comma-split, trim, alias resolution, canonical dedupe, claude-code default) and resolveHarnessChain (registry validation, per-harness CLI path resolution, installability checks with injectable predicate). Unknown/not-installed entries become warnings + are skipped; an all-unknown chain throws the familiar config error; an all-not-installed chain is kept so the spawn surfaces the real "CLI not found" error. AGENT_CLI_PATH applies to the primary entry only so a stale global override can't leak onto a fallback.
• src/resolver.ts: resolveHarness() now takes the first entry of a comma-separated AGENT_HARNESS, keeping one-shot/review-helper flows exactly as today. DEFAULT_HARNESS_NAME moved to registry.ts (avoids an import/no-cycle error).

@getdevintern/code worker mode (webhook server)

• src/lib/harness-failover.ts (new): deterministic state machine — the active harness is always the highest-priority non-limited entry. reportUsageLimit extends windows (furthest reset wins), switches/fails-over, or reports exhausted; windowElapsed clears windows and logs failback to the primary; persistence/clock/log hooks are injected.
• src/webhook-server.ts: startup resolves the chain, warns + skips bad entries, logs Agent harness: a → b (failover enabled), and recovers persisted windows/active harness (stale state dropped with warnings). handleUsageLimit fails over and keeps processing; the queue only pauses when all harnesses are limited, resuming at the earliest reset. Failback timers re-arm automatically. All spawn sites use resolveActiveHarness() so <HARNESS>_CLI_PATH resolves per active harness; /health reports active harness, chain, and open windows. Also fixed a latent bug: processReviewAsync's catch swallowed UsageLimitError, making queue pause/failover unreachable.
• src/lib/webhook-queue.ts: getAllRateLimits, setActiveHarness/getActiveHarness/clearActiveHarness (persisted per-harness in webhook_meta, surviving restarts).
• Run records now record harness for task, pr_mention, and base-sync runs; the review acquirer parses the primary name instead of storing the raw list.

Docs: docs/code/configuration.md gains a "Failover across multiple harnesses" section (syntax, priority semantics, failover/failback, edge cases, env overrides); both .env.example files document the list form.

Tests: packages/agent-harness/tests/harness-chain.test.ts (23) and packages/code/tests/harness-failover.test.ts (17, including restart round-trips through a real SQLite queue DB).

---

This PR was automatically created by @devintern/code

[danii1]

⚠️ devintern resolved this branch's merge conflicts with main but could not publish the merge to this PR (push rejected: Failed to push branch: + '[' '' = 0 ']' + call_lefthook run pre-push origin https://github.com/getdevintern/devintern + test -n '' + lefthook -h + [path] -h + [path] run pre-push origin https://github.com/getdevintern/devintern error: failed to push some refs to 'https://github.com/ge…). No changes landed on the PR; manual action needed.

[danii1]

⚠️ devintern resolved this branch's merge conflicts with main but could not publish the merge to this PR (push rejected: Failed to push branch: + '[' '' = 0 ']' + call_lefthook run pre-push origin https://github.com/getdevintern/devintern + test -n '' + lefthook -h + [path] -h ++ git rev-parse --show-toplevel + dir=[path] ++ uname ++ tr '[:upper:]' '[:lower:]' + osArch=linux ++ uname -m ++ sed 's/aarch6…). No changes landed on the PR; manual action needed.

[danii1]

🔀 This branch had merge conflicts with main; devintern resolved them and pushed the merge. Please double-check the resolution.

[danii1]

⚠️ devintern resolved this branch's merge conflicts with main but could not publish the merge to this PR (push rejected: Failed to push branch: + '[' '' = 0 ']' + call_lefthook run pre-push origin https://github.com/getdevintern/devintern + test -n '' + lefthook -h + [path] -h + [path] run pre-push origin https://github.com/getdevintern/devintern error: failed to push some refs to 'https://github.com/ge…). No changes landed on the PR; manual action needed.

[danii1]

🔀 This branch had merge conflicts with main; devintern resolved them and pushed the merge. Please double-check the resolution.

[danii1]

⚠️ devintern resolved this branch's merge conflicts with main but could not publish the merge to this PR (push rejected: Failed to push branch: error: failed to push some refs to 'https://github.com/getdevintern/devintern' �[38;2;0;0;0m╭�[38;2;6;6;6m─�[38;2;12;12;12m─�[38;2;17;17;17m─�[38;2;20;20;20m─�[38;2;24;24;24m─�[38;2;27;27;27m─�[38;2;31;31;31m─�[38;2;34;34;34m─�[38;2;38;38;38m─�[38;2;41;41;41m─�[…). No changes landed on the PR; manual action needed.

[danii1]

🔀 This branch had merge conflicts with main; devintern resolved them and pushed the merge. Please double-check the resolution.

[danii1]

Failover now covers the fleet worker, not just webhook serve.

What broke on DEV-33

1. Codex printed ERROR: You've hit your usage limit... — the detector required the unprefixed line, so the CLI treated it as a generic exit 1 and posted a failure comment (which then stranded the ticket behind the incomplete-attempt gate).
2. devintern worker (polling/fleet) spawned one-shot CLI children that always used the first AGENT_HARNESS entry. Failover only lived in the webhook server.

Fix

• Detect Codex's ERROR: prefix.
• Shared WorkerFailover pins each worker child to the active harness and retries on exit 75: tracker tasks, PR reviews, @mentions, conflict resolution, automations, estimations, dashboard retries, and relay.
• Exhausted chains defer until a window elapses; state still persists in the queue DB.

Restart the worker daemon to pick this up (systemctl --user restart devintern-worker.service).

[danii1]

🔀 This branch had merge conflicts with main; devintern resolved them and pushed the merge. Please double-check the resolution.