From 8e86077457a8007f3ec31c9bdc6a63722a2fcb3c Mon Sep 17 00:00:00 2001 From: "Robert E. Lee" Date: Wed, 29 Jul 2026 16:22:59 -0700 Subject: [PATCH] feat(run): bounded per-worker escalation in ak run (ADR-0019) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The #76 closure punch list requires ordered, bounded cross-host escalation in the canonical ak run path, tested with an OpenCode-qualified route — the legacy whole-pipeline retry in deprecated ak dual does not satisfy it. Design (ADR-0019): - The resolved route's escalate array travels with each worker at materializeRunPlan; self-equal rungs are dropped (legacy L4 rule) and unroutable rungs fail materialization exactly like an unroutable primary. - On --escalate, a cleanly-failed worker (failed/timed_out) advances ONE rung per attempt in ladder order, bounded by the ladder's length, with per-attempt timeout. Never advanced: blocked/cancelled (dependency state), permission_required (a consent boundary — escalating around it would violate the supervised-host contract), orphaned (uncertain state). - The final result carries attempts[] (host/model/status/category/ duration/reason per attempt) ONLY when more than one attempt ran — no fabricated trail; the final host/model are the rung that executed; attempts is schema-validated like every other result field. A succeeded-after-escalation unblocks dependents normally. A rung with no adapter records cli_unavailable and continues to the next. - ak dual keeps its legacy semantics untouched until #83 removes the wrapper; the semantic difference is named in UPGRADING.md. Tests: 13 new (engine ordering/boundaries/trail/schema, plan materialization + rejection, CLI flag flow + --dry-run ladder display). pnpm run check exit 0 (1080 kit + cjs), pnpm run test:surface exit 0 (25). Live proof (OpenCode-qualified route, sandboxed repo + real CLIs): implementation:opencode with a bogus provider/model → the opencode attempt fails with the serve's 400 (worker_error), advances to the claude rung, which succeeds and writes the mutation — trail recorded verbatim: coder: opencode:failed(worker_error) -> claude:succeeded(success) Dependents unblocked; permission/consent surfaces untouched. --- README.md | 2 +- docs/UPGRADING.md | 2 +- docs/adr/0019-escalation-in-ak-run.md | 94 +++++++++++++++++++ docs/adr/README.md | 1 + src/commands/run.mjs | 20 +++- src/lib/execution/runner.mjs | 63 +++++++++++-- src/lib/execution/schema.mjs | 15 +++ src/lib/routing.mjs | 14 +++ tests/kit/execution-runner.test.mjs | 126 ++++++++++++++++++++++++++ tests/kit/routing.test.mjs | 4 +- tests/kit/run-command.test.mjs | 56 ++++++++++++ 11 files changed, 382 insertions(+), 15 deletions(-) create mode 100644 docs/adr/0019-escalation-in-ak-run.md diff --git a/README.md b/README.md index 5ea28a2..f4a9a0c 100644 --- a/README.md +++ b/README.md @@ -93,7 +93,7 @@ What the verbs cover: | **sync** | The one convergence verb: upgrades first when a new release exists, then re-heals everything an upgrade wipes, then re-checks and reports. Included in that heal: it **installs any enabled frontier host** (claude/codex/opencode) that's entirely absent — never touching an external (mise/brew/native) install — and **re-applies provider wiring** (the `ENABLE_*` host env, OpenCode's native configuration, the aqe fallback chain, and ruflo API providers) whenever it has drifted — and, on a dual-host project, **seeds/heals the Claude/Codex default routing policy** (materializing eligible routes into agentic-qe's `agentOverrides`, e.g. after an aqe upgrade first makes it eligible). It also **installs/repins the standalone `agentdb` CLI** to ruflo's bundled version (keeping the shared cognitive store coherent) and appends a **health-history snapshot** so `status` can flag regressions across syncs. It also **re-runs the RuvNet Brain installer** to pull the latest release when the on-disk KB has drifted (or installs it if absent, when enabled). It also **self-updates the kit**: when a newer `@pacphi/agentic-kit` exists it installs it as the *last* step (the new code applies from the next `ak` run, never mid-sync). Prerelease installs (`4.0.0-alpha.*`) track the `next` npm dist-tag as well as `latest`, so alphas see their successors; stable installs only ever follow `latest`. `--no-upgrade` skips the self-update along with the package upgrades. | | **dashboard** | Opens a read-only local web dashboard (`127.0.0.1:7431`, localhost-only, never detaches) with seven tabs: **Overview · Hosts & Routing · Providers · Runtime · Intelligence · Usage · Live**. The first five render `ak status` health and routing; Usage indexes local Claude/Codex transcripts on demand. Live groups work by project, then host-branded root sessions with nested agent/worker threads and independently evidenced provider/model metadata, and pairs an interactive agent/tool execution canvas with a rich, server-masked transcript stream. Active sessions can be followed live or reviewed with synchronized play/pause/seek; completed sessions remain available for bounded playback. Live contains no chat or control plane. Ruflo, agentic-qe, and dual-run stores are not auto-discovered; register each trusted structured JSONL file with repeatable `--live-source 'surface=path'` (`surface` is `ruflo`, `aqe`, or `dual-run`). The page is self-contained and offline-first (no internet fetches; local files and loopback subprocesses/endpoints only). A fresh **per-session token** is minted at startup and required by every `/api/*` route (`x-dash-token` header; the launch URL carries it in the `#` fragment, same contract as `ak admin` — ADR-0007, ADR-0014) — this page serves full transcript text, so it is gated the same way admin already gated GitHub/npm stats. See [Live Sessions](docs/LIVE-SESSIONS.md) for coverage, syntax, and privacy limits. **Auto-opens your browser** (`--no-open` for headless/SSH); `--port N` changes the port; tabs deep-link (`#live`) and persist. Stop with Ctrl-C. (Also available as `ak x dashboard`.) | | **admin** | Opens the **maintainer admin** (`127.0.0.1:7432`, localhost-only, foreground) — the project-telemetry sibling of `dashboard`, with the same dark/light visual theme and persisted theme preference: unique repo visitors and cloners (GitHub traffic API, needs a push-access token via `GITHUB_TOKEN`/`GH_TOKEN`/`gh auth token` — panels degrade honestly without one), contributors and watchers, npm download momentum (last 7d vs prior 7d, sparklines — shown as trend only, never an absolute reach number, since mirrors/CI inflate the raw count), latest CI run status and open Dependabot alerts, a **"since you last looked"** delta strip over a local baseline, open issues/PRs from others (oldest first), and external humans ranked by recency (bots excluded). Access is gated by a **per-session token** carried in the URL fragment and sent header-only; the page makes **zero external fetches** (the server proxies GitHub/npm; your credential never reaches the page or the payload — ADR-0007, ADR-0013). Where `dashboard` is offline-first, `admin` does deliberate GitHub/npm egress — that contract split is why they're siblings, not tabs. `--port N`, `--no-open`; Ctrl-C stops. (Also available as `ak x admin`.) | -| **run** | **Canonical execution surface.** Executes the template vocabulary through host-neutral supervised adapters. It accepts an explicit OpenCode route (persisted or `--route`) alongside Claude/Codex; `--dry-run` prints the exact plan. An OpenCode worker runs an isolated loopback server with ephemeral basic authentication, returns only normalized observed facts, and aborts instead of approving a permission request. `ak run` does not turn OpenCode into an AQE provider or primary host. | +| **run** | **Canonical execution surface.** Executes the template vocabulary through host-neutral supervised adapters. It accepts an explicit OpenCode route (persisted or `--route`) alongside Claude/Codex; `--dry-run` prints the exact plan (with each worker's escalation ladder); `--escalate` advances a failed worker one rung of its route's ladder per attempt (bounded by the ladder; permission/consent and uncertain results are never escalated — ADR-0019). An OpenCode worker runs an isolated loopback server with ephemeral basic authentication, returns only normalized observed facts, and aborts instead of approving a permission request. `ak run` does not turn OpenCode into an AQE provider or primary host. | | **dual** | **Deprecated compatibility wrapper** for existing Claude+Codex `claude-flow-codex` scripts, including its legacy escalation behavior. New execution work should use `ak run`; `ak dual` warns on stderr and will be removed before the stable release. | | **host** | Canonical alpha namespace for execution-host status, selection, primary-host choice, activity routing, and reversible teardown: `ak host status\|pick\|refresh\|off`. The plumbing spelling is `ak x host`. `ak provider` and `ak x provider` are deprecated compatibility aliases that warn on stderr and will be removed before the stable release. | | **uninstall** | Removes the kit's footprint (and any legacy shell-kit install); project data is never touched; `--purge` also offers to remove the global packages. | diff --git a/docs/UPGRADING.md b/docs/UPGRADING.md index 0a5fdda..c8b7b72 100644 --- a/docs/UPGRADING.md +++ b/docs/UPGRADING.md @@ -28,7 +28,7 @@ family on your behalf. | Migration surface | What to know | | ------------------- | ----------------------------------------------- | -| `ak dual` → `ak run` | `ak dual` is a deprecated compatibility wrapper — existing scripts keep working (it warns on stderr and will be removed before the stable release); use `ak run` for new execution work. OpenCode routes require the current release — remove them before downgrading. | +| `ak dual` → `ak run` | `ak dual` is a deprecated compatibility wrapper — existing scripts keep working (it warns on stderr and will be removed before the stable release); use `ak run` for new execution work. OpenCode routes require the current release — remove them before downgrading. `--escalate` exists on both, with deliberately different semantics: the wrapper retries the *whole pipeline* once on any failure; `ak run` advances only the *failed worker* one rung of its route's ladder per attempt (ADR-0019) and records the attempt trail in the result. | A **host** runs the work; a **provider** serves inference. A binding can connect one provider to several hosts through separate native configuration **projections**, while **observability** diff --git a/docs/adr/0019-escalation-in-ak-run.md b/docs/adr/0019-escalation-in-ak-run.md new file mode 100644 index 0000000..be936be --- /dev/null +++ b/docs/adr/0019-escalation-in-ak-run.md @@ -0,0 +1,94 @@ +# ADR-0019 — Bounded per-worker escalation in `ak run` + +- **Status:** Accepted +- **Date:** 2026-07-29 +- **Deciders:** agentic-kit maintainers + +## Context + +The #76 closure punch list requires that ordered, bounded cross-host escalation reach +the canonical `ak run` path — or be explicitly retired with an approved migration +decision. The legacy behavior lives only in the deprecated `ak dual` wrapper (ADR-0004): +on a non-zero pipeline exit and an explicit `--escalate` flag, the *entire* pipeline +re-runs once with `escalatePolicy(policy)` — every activity bumped to its ladder's first +non-self rung. That shape has three faults the canonical runner should not inherit: + +1. **Whole-pipeline retry for a single worker's failure.** A failing `coder` re-ran the + successful `architect` too — spend and wall-clock doubled for no benefit. +2. **No attempt evidence.** The escalated run presented as a fresh run; the first + attempt's failure left no trace in the result contract. +3. **No consent boundary.** A permission-refused or uncertain worker would be retried + like any other failure. + +## Decision + +Escalation moves into the runner as **bounded per-worker ladder attempts**, opt-in per +invocation via `ak run --escalate`: + +1. **The ladder travels with the worker.** `materializeRunPlan` attaches the resolved + route's `escalate` array to each worker. Self-equal rungs are dropped at + materialization (re-running the identical host+model changes nothing — the legacy L4 + rule), and every rung must be a routable host or materialization fails exactly the + way an unroutable primary route does. +2. **Ordered and bounded.** A worker whose result is escalatable advances **one rung at + a time**, in ladder order, and stops when a rung succeeds or the ladder exhausts. + There is no unbounded retry: the ladder's length is the bound, and per-worker + `--timeout` applies per attempt. +3. **Escalatable means *cleanly failed*.** `failed` and `timed_out` may advance. + Never advanced: `blocked`/`cancelled` (dependency state, not a worker failure), + `permission_required` (a consent boundary — escalating around it would violate the + supervised-host contract that the OpenCode permission-abort implements), and + `orphaned` (execution state uncertain; a retry risks a double run). +4. **The trail is evidence, not noise.** The final result carries `attempts[]` — each + attempt's host, model, status, exitCategory, durationMs, and (on failure) bounded + reason — but ONLY when more than one attempt ran. A single attempt is + indistinguishable from escalation being off, and emitting a trail there would + fabricate an event that did not happen. The final result's host/model are the rung + that actually executed (observed truth), a succeeded-after-escalation unblocks + dependents normally, and `attempts` is schema-validated like every other result + field. +5. **Explicit opt-in, same as legacy.** Escalation doubles attempted work by design; + it stays behind `--escalate` rather than becoming the default posture. + +This satisfies the punch list's "ordered, bounded cross-host escalation into the +canonical path, tested with an OpenCode-qualified route": an +`implementation:opencode` route with a claude ladder rung escalates through the real +OpenCode adapter (`permission_required` from its consent boundary excluded by rule 3). + +The legacy `escalatePolicy` + `dual --escalate` stay untouched inside the deprecated +wrapper until #83 removes the wrapper; nothing here changes their behavior. + +## Consequences + +- A failing worker no longer re-runs successful siblings — escalation cost is scoped to + the failure. +- `ak run --json` results now carry `attempts[]` on escalated workers; consumers see + where a worker started and where it landed, and the consent/uncertainty exclusions + are visible in what is *absent*. +- The rung must have an execution adapter to advance: a ladder naming a host with no + adapter records `cli_unavailable` for that rung and continues to the next. +- The deprecated wrapper's whole-pipeline semantics and the canonical per-worker + semantics differ *deliberately*; the migration note in UPGRADING.md names that as an + intended improvement, not a drift. + +## Alternatives considered + +- **Port dual's whole-pipeline retry.** Rejected for the three faults above: it wastes + successful work, hides the first attempt, and ignores consent boundaries. +- **Always-on escalation.** Rejected: doubling attempted spend must remain a + per-invocation choice, exactly as it was in the legacy wrapper. +- **Retire escalation entirely** (the punch list's alternative). Rejected: the ladder + data already exists in the routing policy and the per-worker shape is strictly + better than retiring a capability users have. + +## References + +- `src/lib/execution/runner.mjs` (`executeWorkerWithEscalation`, escalatable rules), + `src/lib/execution/schema.mjs` (`attempts` validation), + `src/lib/routing.mjs` (`materializeRunPlan` ladder attach), + `src/commands/run.mjs` (`--escalate`). +- ADR-0004 (legacy escalation semantics), ADR-0018 (canonical execution contract), + #76 punch list item 1, #83 (deprecated-wrapper removal that consumes this migration). +- Tests: `tests/kit/execution-runner.test.mjs` (engine, boundaries, trail, schema), + `tests/kit/run-command.test.mjs` (materialization, CLI flow), plus the live + OpenCode-qualified smoke recorded on #76. diff --git a/docs/adr/README.md b/docs/adr/README.md index 8f81f68..e86954f 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -27,6 +27,7 @@ Consequences**, and cites the grounded source it rests on where relevant. | [0016](0016-capability-driven-integration-adapters.md) | Capability-driven host, provider, binding, projection, and observability adapters | Accepted | | [0017](0017-opencode-host.md) | OpenCode as a managed, observable host through native surfaces | Accepted | | [0018](0018-generalized-host-worker-execution.md) | Generalized host-worker execution; `ak run` canonical | Accepted | +| [0019](0019-escalation-in-ak-run.md) | Bounded per-worker escalation in `ak run` | Accepted | Theme: ADRs **0001–0006** define **dual-host LLM routing and leadership** — how `ak` lets ruflo route each development activity (architecture, implementation, testing, review, …) to the right host (Claude diff --git a/src/commands/run.mjs b/src/commands/run.mjs index 2746c0c..1209326 100644 --- a/src/commands/run.mjs +++ b/src/commands/run.mjs @@ -7,6 +7,7 @@ import { DUAL_RUN_TEMPLATE_NAMES, materializeRunPlan, parseRouteSpecs } from '.. export const options = { route: { type: 'string', multiple: true }, + escalate: { type: 'boolean', default: false }, 'dry-run': { type: 'boolean', default: false }, 'max-concurrent': { type: 'string' }, timeout: { type: 'string' }, @@ -25,14 +26,19 @@ Templates: ${DUAL_RUN_TEMPLATE_NAMES.join(', ')} Options: --route 'act:host[:model]' per-run routing override (repeatable; not persisted) + --escalate on failure, advance the worker one rung of its + route's escalation ladder (bounded by the ladder; + permission/consent and uncertain results are + never escalated) --dry-run print the host-neutral execution plan only --max-concurrent max concurrent workers (default 4) - --timeout per-worker timeout (default 120000) + --timeout per-worker timeout, per attempt (default 120000) --json emit machine-readable plan/results Examples: ak run feature "add token-bucket rate limiting" --dry-run - ak run security "src/auth/" --route 'security-scan:opencode'`; + ak run security "src/auth/" --route 'security-scan:opencode' + ak run feature "fix the flaky parser" --escalate`; function positiveInt(value, name) { if (value === undefined) return undefined; @@ -54,14 +60,18 @@ function printPlan(plan) { console.log(bold(`run: ${plan.template}`)); for (const worker of plan.workers) { const dependency = worker.dependsOn?.length ? `after ${worker.dependsOn.join(', ')}` : 'start'; - console.log(` ${worker.id.padEnd(12)} ${worker.host.padEnd(9)} ${(worker.configuredModel ?? '').padEnd(24)} ${dim(dependency)}`); + const ladder = worker.escalate?.length ? dim(` ↑ ${worker.escalate.map((rung) => rung.host).join('→')}`) : ''; + console.log(` ${worker.id.padEnd(12)} ${worker.host.padEnd(9)} ${(worker.configuredModel ?? '').padEnd(24)} ${dim(dependency)}${ladder}`); } } function printResults(results) { for (const result of results) { const detail = result.failure?.reason ? ` — ${result.failure.reason}` : ''; - console.log(` ${result.workerId.padEnd(12)} ${result.host.padEnd(9)} ${result.status} (${result.exitCategory})${dim(detail)}`); + // The escalation trail is visible, not silent: a success after advancing + // rungs says where it started (ADR-0019). + const trail = result.attempts?.length > 1 ? dim(` (escalated from ${result.attempts[0].host})`) : ''; + console.log(` ${result.workerId.padEnd(12)} ${result.host.padEnd(9)} ${result.status} (${result.exitCategory})${trail}${dim(detail)}`); } } @@ -94,7 +104,7 @@ export async function run({ flags, positionals, executePlan = executeRunPlan, cf timeoutMs = positiveInt(flags.timeout, 'timeout'); } catch (error) { fail(error.message); return 2; } if (!flags.json) printPlan(plan); - const results = await executePlan(plan, { maxConcurrent, timeoutMs }); + const results = await executePlan(plan, { maxConcurrent, timeoutMs, escalate: !!flags.escalate }); if (flags.json) console.log(JSON.stringify({ plan, results }, null, 2)); else printResults(results); // The human status line is gated on !json — a trailing "✓ run complete" diff --git a/src/lib/execution/runner.mjs b/src/lib/execution/runner.mjs index 360b7fc..1d70a4b 100644 --- a/src/lib/execution/runner.mjs +++ b/src/lib/execution/runner.mjs @@ -92,11 +92,63 @@ function adapterFor(adapters, host) { return adapter ? validateExecutionAdapter(adapter) : null; } +// ── bounded escalation (ADR-0019) ─────────────────────────────────────────── +// A non-success worker result may advance ONE rung of the route's ladder +// (host+model), bounded by the ladder's length — never the whole-pipeline +// retry the legacy dual wrapper did. What may NOT advance, ever: +// - blocked/cancelled results (dependency state, not a worker failure); +// - permission_required (a consent boundary — escalating around it would be +// a safety violation, exactly the opencode abort contract); +// - orphaned (execution state is uncertain; a retry risks a double run). +const ESCALATABLE_STATUSES = new Set(['failed', 'timed_out']); +const BLOCKING_CATEGORIES = new Set(['permission_required', 'cancelled', 'orphaned']); + +const escalatable = (result) => ESCALATABLE_STATUSES.has(result.status) && !BLOCKING_CATEGORIES.has(result.exitCategory); + +/** Compact one attempt for the result's trail: which rung, what verdict, how + * long, and (on failure) why — bounded like every other failure surface. */ +function compactAttempt(worker, result) { + const attempt = { + host: worker.host, model: worker.configuredModel ?? null, + status: result.status, exitCategory: result.exitCategory, + durationMs: result.durationMs, + }; + const reason = result.failure?.reason; + if (result.status !== 'succeeded' && typeof reason === 'string') attempt.reason = reason; + return attempt; +} + +/** Execute a worker, advancing one ladder rung per escalatable failure when + * `escalate` is on. The final result carries an `attempts` trail ONLY when + * more than one attempt ran — a single attempt is indistinguishable from a + * run with escalation off, and emitting one would fabricate an event that + * did not happen. */ +async function executeWorkerWithEscalation(worker, adapters, { cwd, timeoutMs, clock, escalate = false }) { + const ladder = escalate ? [...(worker.escalate ?? [])] : []; + const attempts = []; + let current = worker; + let result; + for (;;) { + const adapter = adapterFor(adapters, current.host); + result = adapter + ? await executeWorker(current, adapter, { cwd, timeoutMs, clock }) + : workerFailure(current, { exitCategory: 'cli_unavailable', failure: { reason: `no execution adapter for host "${current.host}"` }, clock }); + attempts.push(compactAttempt(current, result)); + if (!escalatable(result)) break; + const rung = ladder.shift(); + if (!rung) break; + current = { ...current, host: rung.host, configuredModel: rung.model ?? null }; + } + if (attempts.length <= 1) return result; + return validateWorkerResult({ ...result, attempts }); +} + /** Run a materialized routing plan as a bounded-concurrency dependency DAG. - * A failed dependency blocks descendants; independent branches keep running. */ + * A failed dependency blocks descendants; independent branches keep running. + * `escalate: true` enables bounded per-worker ladder retries (ADR-0019). */ export async function executeRunPlan(plan, { - adapters = EXECUTION_ADAPTERS, cwd = process.cwd(), maxConcurrent = 4, timeoutMs, clock = nowIso, -} = /** @type {{adapters?:Record|Map, cwd?:string, maxConcurrent?:number, timeoutMs?:number, clock?:()=>string}} */ ({})) { + adapters = EXECUTION_ADAPTERS, cwd = process.cwd(), maxConcurrent = 4, timeoutMs, clock = nowIso, escalate = false, +} = /** @type {{adapters?:Record|Map, cwd?:string, maxConcurrent?:number, timeoutMs?:number, clock?:()=>string, escalate?:boolean}} */ ({})) { validatePlan(plan); if (!Number.isInteger(maxConcurrent) || maxConcurrent < 1) throw new TypeError('maxConcurrent must be a positive integer'); const pending = new Map(plan.workers.map((worker) => [worker.id, worker])); @@ -104,10 +156,7 @@ export async function executeRunPlan(plan, { const running = new Map(); const start = (worker) => { - const adapter = adapterFor(adapters, worker.host); - const promise = adapter - ? executeWorker(worker, adapter, { cwd, timeoutMs, clock }) - : Promise.resolve(workerFailure(worker, { exitCategory: 'cli_unavailable', failure: { reason: `no execution adapter for host "${worker.host}"` }, clock })); + const promise = executeWorkerWithEscalation(worker, adapters, { cwd, timeoutMs, clock, escalate }); running.set(worker.id, promise.then((result) => ({ id: worker.id, result }))); pending.delete(worker.id); }; diff --git a/src/lib/execution/schema.mjs b/src/lib/execution/schema.mjs index e33e5c2..c46da3b 100644 --- a/src/lib/execution/schema.mjs +++ b/src/lib/execution/schema.mjs @@ -59,5 +59,20 @@ export function validateWorkerResult(value) { if (value.provider === null && value.providerProvenance !== 'unknown') { throw new TypeError('workerResult.providerProvenance must be unknown without a provider'); } + // attempts (ADR-0019): the escalation trail — present only when a worker + // actually advanced rungs. Each entry is one executed attempt's compact + // verdict, never a fabricated one. + if (value.attempts !== undefined) { + if (!Array.isArray(value.attempts)) throw new TypeError('workerResult.attempts must be an array when present'); + for (const [i, a] of value.attempts.entries()) { + assertRecord(a, `workerResult.attempts[${i}]`); + if (typeof a.host !== 'string' || !a.host) throw new TypeError(`workerResult.attempts[${i}].host must be a non-empty string`); + if (a.model !== null && typeof a.model !== 'string') throw new TypeError(`workerResult.attempts[${i}].model must be string|null`); + assertEnum(a.status, WORKER_STATUSES, `workerResult.attempts[${i}].status`); + assertEnum(a.exitCategory, EXIT_CATEGORIES, `workerResult.attempts[${i}].exitCategory`); + if (!Number.isFinite(a.durationMs) || a.durationMs < 0) throw new TypeError(`workerResult.attempts[${i}].durationMs must be a non-negative number`); + if (a.reason !== undefined && typeof a.reason !== 'string') throw new TypeError(`workerResult.attempts[${i}].reason must be a string when present`); + } + } return immutable(structuredClone(value)); } diff --git a/src/lib/routing.mjs b/src/lib/routing.mjs index a83a485..2ea8749 100644 --- a/src/lib/routing.mjs +++ b/src/lib/routing.mjs @@ -441,6 +441,19 @@ export function materializeRunPlan(policy = {}, { template = 'feature', task = ' if (!eligibility.ok) { throw new Error(`route for "${n.activity}" cannot materialize: host "${r.host}" requires canRouteActivities`); } + // The escalation ladder travels with the worker (ADR-0019). Self-equal + // rungs are dropped here (escalating to the same host+model would re-run + // the identical attempt — the legacy L4 rule), and every rung must be a + // routable host or materialization fails the same way the primary does. + const ladder = (r.escalate ?? []) + .filter((rung) => rung && (rung.host !== r.host || (rung.model ?? null) !== (r.model ?? null))) + .map((rung) => { + const rungEligibility = validateActivityHost(rung.host); + if (!rungEligibility.ok) { + throw new Error(`escalation rung for "${n.activity}" cannot materialize: host "${rung.host}" requires canRouteActivities`); + } + return { host: rung.host, model: rung.model ?? null }; + }); return { id: n.id, activity: n.activity, @@ -450,6 +463,7 @@ export function materializeRunPlan(policy = {}, { template = 'feature', task = ' prompt: n.prompt(task), ...(n.dependsOn ? { dependsOn: n.dependsOn } : {}), ...(n.maxTurns ? { maxTurns: n.maxTurns } : {}), + ...(ladder.length ? { escalate: ladder } : {}), }; }), }; diff --git a/tests/kit/execution-runner.test.mjs b/tests/kit/execution-runner.test.mjs index 7f8ce3e..ee47346 100644 --- a/tests/kit/execution-runner.test.mjs +++ b/tests/kit/execution-runner.test.mjs @@ -33,6 +33,132 @@ function adapter({ observation = { type: 'idle' }, events = [] } = {}) { }; } +// ── bounded escalation (ADR-0019) ─────────────────────────────────────────── + +/** An adapter that fails (or succeeds) per host with a canned script. */ +function scriptedAdapter(events, script) { + return { + id: 'scripted', + async readiness() { return { ready: true }; }, + async prepare({ worker: w }) { return { worker: w }; }, + async launch(state) { return state; }, + async observe(state) { events.push(`observe:${state.worker.host}`); return script.observe?.() ?? { type: 'idle' }; }, + interpret(state, _observed) { + const fail = script.fail; + return { + workerId: state.worker.id, activity: state.worker.activity, role: state.worker.role, host: state.worker.host, + status: fail ? (script.status ?? 'failed') : 'succeeded', + exitCategory: fail ? (script.exitCategory ?? 'worker_error') : 'success', + startedAt: clock(), endedAt: clock(), durationMs: 1, + provider: null, providerProvenance: 'unknown', configuredModel: state.worker.configuredModel ?? null, + observedModel: null, sessionId: null, transcriptRefs: [], + failure: fail ? { reason: script.reason ?? 'scripted failure' } : null, usage: null, + }; + }, + async cancel() {}, + async cleanup() {}, + }; +} + +const escalatableWorker = (id, host, ladder) => ({ + id, activity: 'implementation', role: 'coder', host, configuredModel: `${host}-model`, + prompt: id, ...(ladder ? { escalate: ladder } : {}), +}); + +test('escalation advances one rung on failure and records the full trail (OpenCode-qualified route)', async () => { + const events = []; + const adapters = { + opencode: scriptedAdapter(events, { fail: true, reason: 'serve 400' }), + claude: scriptedAdapter(events, { fail: false }), + }; + const plan = { workers: [ + escalatableWorker('coder', 'opencode', [{ host: 'claude', model: 'claude-opus-5' }]), + { ...escalatableWorker('reviewer', 'claude'), dependsOn: ['coder'], role: 'reviewer', activity: 'review' }, + ] }; + const results = await executeRunPlan(plan, { adapters, escalate: true, clock }); + const coder = results.find((r) => r.workerId === 'coder'); + assert.equal(coder.status, 'succeeded', 'the claude rung carried the worker'); + assert.equal(coder.host, 'claude', 'the final result reports the rung that actually ran'); + assert.equal(coder.attempts.length, 2); + assert.deepEqual( + coder.attempts.map((a) => [a.host, a.status]), + [['opencode', 'failed'], ['claude', 'succeeded']], + 'ordered, bounded trail — opencode first, then the claude rung', + ); + assert.match(coder.attempts[0].reason, /serve 400/); + assert.equal(results.find((r) => r.workerId === 'reviewer').status, 'succeeded', + 'an escalated success unblocks dependents'); +}); + +test('escalation is bounded by the ladder and records every attempt when it exhausts', async () => { + const events = []; + const adapters = { + opencode: scriptedAdapter(events, { fail: true, reason: 'first' }), + claude: scriptedAdapter(events, { fail: true, reason: 'second' }), + codex: scriptedAdapter(events, { fail: true, reason: 'third' }), + }; + const plan = { workers: [escalatableWorker('coder', 'opencode', [{ host: 'claude' }, { host: 'codex' }])] }; + const [result] = await executeRunPlan(plan, { adapters, escalate: true, clock }); + assert.equal(result.status, 'failed'); + assert.equal(result.attempts.length, 3, 'three attempts, then it stops — no unbounded retry'); + assert.deepEqual(result.attempts.map((a) => a.host), ['opencode', 'claude', 'codex']); + assert.equal(result.host, 'codex', 'the final result is the last rung actually executed'); +}); + +test('consent and uncertain states are never escalated (permission, orphaned, blocked)', async () => { + for (const [exitCategory, status] of [['permission_required', 'failed'], ['orphaned', 'failed'], ['worker_error', 'blocked']]) { + const adapters = { + opencode: scriptedAdapter([], { fail: true, exitCategory, status }), + claude: scriptedAdapter([], { fail: false }), + }; + const plan = { workers: [escalatableWorker('coder', 'opencode', [{ host: 'claude' }])] }; + const [result] = await executeRunPlan(plan, { adapters, escalate: true, clock }); + assert.equal(result.attempts ?? undefined, undefined, + `${exitCategory}/${status} must not escalate (attempts absent — single attempt, no trail fabricated)`); + assert.equal(result.host, 'opencode', 'no rung was attempted past the boundary'); + } +}); + +test('without --escalate a ladder-carrying worker makes exactly one attempt', async () => { + const events = []; + const adapters = { opencode: scriptedAdapter(events, { fail: true }), claude: scriptedAdapter(events, { fail: false }) }; + const plan = { workers: [escalatableWorker('coder', 'opencode', [{ host: 'claude' }])] }; + const [result] = await executeRunPlan(plan, { adapters, escalate: false, clock }); + assert.equal(result.status, 'failed'); + assert.equal(result.attempts ?? undefined, undefined); + assert.deepEqual(events.filter((e) => e.startsWith('observe:')), ['observe:opencode'], 'no rung attempted'); +}); + +test('a successful first attempt leaves no escalation trail (nothing fabricated)', async () => { + const adapters = { opencode: scriptedAdapter([], { fail: false }) }; + const plan = { workers: [escalatableWorker('coder', 'opencode', [{ host: 'claude' }])] }; + const [result] = await executeRunPlan(plan, { adapters, escalate: true, clock }); + assert.equal(result.status, 'succeeded'); + assert.equal(result.attempts ?? undefined, undefined, 'one attempt is indistinguishable from escalation off'); +}); + +test('a rung with no execution adapter records cli_unavailable and advances to the next rung', async () => { + const adapters = { codex: scriptedAdapter([], { fail: false }) }; + const plan = { workers: [escalatableWorker('coder', 'opencode', [{ host: 'claude' }, { host: 'codex' }])] }; + const [result] = await executeRunPlan(plan, { adapters, escalate: true, clock }); + assert.equal(result.status, 'succeeded'); + assert.equal(result.attempts.length, 3); + assert.deepEqual(result.attempts.map((a) => [a.host, a.exitCategory]), + [['opencode', 'cli_unavailable'], ['claude', 'cli_unavailable'], ['codex', 'success']]); +}); + +test('the attempts trail is schema-validated end to end', async () => { + const adapters = { + opencode: scriptedAdapter([], { fail: true }), + claude: scriptedAdapter([], { fail: false }), + }; + const plan = { workers: [escalatableWorker('coder', 'opencode', [{ host: 'claude', model: 'claude-opus-5' }])] }; + const [result] = await executeRunPlan(plan, { adapters, escalate: true, clock }); + const { validateWorkerResult } = await import('../../src/lib/execution/schema.mjs'); + assert.doesNotThrow(() => validateWorkerResult(result), 'the escalated result passes the same schema'); + assert.equal(result.attempts[1].model, 'claude-opus-5', 'the rung model is recorded'); +}); + // qe-court A2: the timeout branch must schema-validate interpret() too — a // malformed timeout result becomes a bounded protocol_error, never garbage // shipped raw into `ak run --json`. diff --git a/tests/kit/routing.test.mjs b/tests/kit/routing.test.mjs index cffa6e0..309fbde 100644 --- a/tests/kit/routing.test.mjs +++ b/tests/kit/routing.test.mjs @@ -197,7 +197,9 @@ test('host-neutral run plan preserves every legacy dual worker assignment', () = const plan = materializeRunPlan(policy, { template: 'feature', task: 'add auth' }); const dual = policyToDualRunConfig(policy, { template: 'feature', task: 'add auth' }); assert.equal(plan.template, 'feature'); - assert.deepEqual(plan.workers.map(({ host, configuredModel, activity: _activity, ...worker }) => ({ + // `escalate` is run-only ladder metadata (ADR-0019); the legacy projection + // has its own escalatePolicy overlay instead, so it is excluded from parity. + assert.deepEqual(plan.workers.map(({ host, configuredModel, activity: _activity, escalate: _ladder, ...worker }) => ({ ...worker, platform: host, model: configuredModel ?? undefined, })), dual.workers); assert.ok(plan.workers.every((worker) => worker.activity && worker.host && !('platform' in worker))); diff --git a/tests/kit/run-command.test.mjs b/tests/kit/run-command.test.mjs index 71218ed..205816e 100644 --- a/tests/kit/run-command.test.mjs +++ b/tests/kit/run-command.test.mjs @@ -62,3 +62,59 @@ test('ak run without --json still prints the human status line', async () => { assert.equal(result, 0); assert.match(out, /run complete/); }); + +// ── bounded escalation: plan materialization + CLI surface (ADR-0019) ──────── + +test('materializeRunPlan attaches the route ladder, dropping self-equal rungs', () => { + const cfg = { providers: { dualRouting: { implementation: { + host: 'opencode', model: 'opencode/kimi-k3', source: 'user', + escalate: [ + { host: 'opencode', model: 'opencode/kimi-k3' }, // self-equal: re-running this changes nothing + { host: 'claude', model: 'claude-sonnet-5' }, + ], + } } } }; + const { plan } = buildRunPlan(cfg, 'feature', 'probe'); + const coder = plan.workers.find((w) => w.activity === 'implementation'); + assert.deepEqual(coder.escalate, [{ host: 'claude', model: 'claude-sonnet-5' }], + 'only the non-self rung survives, with its model'); +}); + +test('materializeRunPlan rejects an escalation rung to a non-routable host', () => { + const cfg = { providers: { dualRouting: { implementation: { + host: 'claude', source: 'user', escalate: [{ host: 'not-a-host' }], + } } } }; + assert.throws(() => buildRunPlan(cfg, 'feature', 'probe'), + /escalation rung for "implementation" cannot materialize: host "not-a-host" requires canRouteActivities/); +}); + +test('--dry-run shows the escalation ladder on ladder-carrying workers', async () => { + const cfg = { providers: { dualRouting: { implementation: { + host: 'opencode', model: 'opencode/kimi-k3', source: 'user', + escalate: [{ host: 'claude', model: 'claude-sonnet-5' }], + } } } }; + const { out } = await captureLog(() => run({ + flags: { 'dry-run': true }, positionals: ['feature', 'probe'], cfg, + })); + assert.match(out, /coder\s+opencode\s+opencode\/kimi-k3\s+after architect\s+↑ claude/, 'ladder visible in the plan'); +}); + +test('--escalate flows into the executor opts; results print the escalation trail', async () => { + let seen; + const executePlan = async (plan, opts) => { + seen = opts; + return plan.workers.map((w) => ({ + ...succeededResult(w), + attempts: [ + { host: 'opencode', model: 'opencode/kimi-k3', status: 'failed', exitCategory: 'worker_error', durationMs: 1, reason: 'serve 400' }, + { host: 'claude', model: 'claude-sonnet-5', status: 'succeeded', exitCategory: 'success', durationMs: 2 }, + ], + })); + }; + const { result, out } = await captureLog(() => run({ + flags: { escalate: true }, positionals: ['feature', 'probe'], executePlan, cfg: testCfg(), + })); + assert.equal(result, 0); + assert.equal(seen.escalate, true, 'the flag reaches executeRunPlan'); + assert.ok(out.includes('succeeded (success)') && out.includes('escalated from opencode'), + `the trail is visible, not silent:\n${out}`); +});