fix: eagerly warm extension host stores from engine TaskStores - #3340
Conversation
Pre-populate setHostTaskStore for all registered projects from already-running ProjectEngine TaskStores, so extension API tools (fn_task_update, fn_task_archive, fn_agent_show) find a cached store and never fall through to createTaskStoreForBackend, which can time out creating a second connection pool. Uses each engine's existing TaskStore directly via engine.getTaskStore() — no new PG boot, no schema advisory-lock contention, no connection-pool exhaustion. cwd store is already injected; non-cwd projects are warmed from the running engines. Per-project failures are non-fatal warnings.
📝 WalkthroughWalkthroughDashboard startup now warms extension-host task stores for registered non-CWD projects by reusing each project engine’s existing ChangesExtension host task-store warmup
Estimated code review effort: 2 (Simple) | ~10 minutes Sequence Diagram(s)sequenceDiagram
participant Dashboard
participant RegisteredProjects
participant ProjectEngine
participant setHostTaskStore
Dashboard->>RegisteredProjects: enumerate registered projects
loop non-CWD projects
Dashboard->>ProjectEngine: get existing engine task store
ProjectEngine-->>Dashboard: return TaskStore or no engine
Dashboard->>setHostTaskStore: register existing TaskStore
end
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/cli/src/commands/dashboard.ts`:
- Line 2284: Update the comment near the cwd skip in the dashboard command to
remove the stale numeric line reference, replacing it with “already injected
above” or an equivalent location-independent description while preserving its
meaning.
- Around line 2296-2298: Update the extension host warmup flow in
packages/cli/src/commands/dashboard.ts around the setHostTaskStore calls to
count only successful registrations and report that successful count in the
warmup log, rather than using all non-CWD candidates; update
.changeset/extension-host-store-warmup.md lines 5-7 to describe the behavior as
best-effort for running non-CWD engines.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 92133bb3-ec72-4eff-9b9e-8d550053c40b
📒 Files selected for processing (2)
.changeset/extension-host-store-warmup.mdpackages/cli/src/commands/dashboard.ts
| void (async () => { | ||
| try { | ||
| const projects = await centralCoreForEngine.listProjects(); | ||
| // Skip cwd — its store is already injected at line 928. |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Remove the stale line reference.
The comment points to "line 928", but setHostTaskStore(cwd, store) is at Line 1002. Use “already injected above” or update the reference.
As per coding guidelines, keep comments updated as requirements change.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/cli/src/commands/dashboard.ts` at line 2284, Update the comment near
the cwd skip in the dashboard command to remove the stale numeric line
reference, replacing it with “already injected above” or an equivalent
location-independent description while preserving its meaning.
Source: Coding guidelines
| if (nonCwd.length > 0) { | ||
| logSink.log(`Warmed extension host stores for ${nonCwd.length} project(s)`, "extension"); | ||
| } |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Make the reported warmup coverage match actual success.
The implementation can skip projects or catch registration failures, but the status log and changeset describe all candidates as warmed.
packages/cli/src/commands/dashboard.ts#L2296-L2298: count successfulsetHostTaskStorecalls..changeset/extension-host-store-warmup.md#L5-L7: describe the behavior as best-effort for running non-CWD engines.
📍 Affects 2 files
packages/cli/src/commands/dashboard.ts#L2296-L2298(this comment).changeset/extension-host-store-warmup.md#L5-L7
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/cli/src/commands/dashboard.ts` around lines 2296 - 2298, Update the
extension host warmup flow in packages/cli/src/commands/dashboard.ts around the
setHostTaskStore calls to count only successful registrations and report that
successful count in the warmup log, rather than using all non-CWD candidates;
update .changeset/extension-host-store-warmup.md lines 5-7 to describe the
behavior as best-effort for running non-CWD engines.
…sion#3340) ## Summary Warm the extension-host task stores **up front** at dashboard startup instead of letting the first `fn_task_*` call lazily boot a second PostgreSQL pool per project. ## What changed `packages/cli/src/commands/dashboard.ts`: - After the dashboard boots, iterate every registered project (from `centralCoreForEngine.listProjects()`) and call `setHostTaskStore(p.path, engine.getTaskStore())` for each non-cwd project that already has a running `ProjectEngine`. - Reuses each engine's **existing** `TaskStore` directly — no new backend connection, no schema advisory-lock contention, no extra connection-pool exhaustion. - `cwd` is skipped because its store is already injected at startup. - Per-project failures are non-fatal (warn) and a failed project listing logs a single warn — dashboard startup never blocks on this. - `.changeset/extension-host-store-warmup.md` (patch, fix). ## Why Left on its own, the first extension tool call (`fn_task_update`, `fn_task_archive`, `fn_agent_show`, …) for a non-cwd project falls through to `createTaskStoreForBackend`, which boots a **second** PostgreSQL connection pool on demand. On busy hosts that lazy boot can time out, or the call stalls behind pool/startup contention — the classic "first `fn_task_*` call is slow or errors" experience. Pre-populating from the already-running engines removes that lazy worst-case path entirely. ## Verification - `pnpm verify:fast` — PASS (13 steps, 115s): CLI `tsup` build green, scoped typecheck/build green, boot smoke green (`fn --help` + real `serve` with `GET /api/health` 200). - Cherry-picked cleanly onto current `origin/main` (`5532019fd`); branch is up-to-date with `origin/main` at PR time. ## Files - `packages/cli/src/commands/dashboard.ts` (+30) - `.changeset/extension-host-store-warmup.md` (new) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved dashboard startup reliability by reusing existing project task connections. * Prevented extension task tools from creating duplicate connection pools. * Added non-blocking warnings when individual project initialization or discovery fails. * Dashboard startup now reports how many project task stores were successfully prepared. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: gsxdsm <gsxdsm@users.noreply.github.com>
* fix(mesh): preserve foreign-node worktree metadata * fix(engine): ignore pre-reset plan review logs * fix(core): keep plan reviews independent of worktree leaf * fix(engine): expose routed execution context * fix(cli): retry daemon after central db startup failure * fix(engine): scope executor task reads to project * fix(engine): respect node ownership during archive cleanup * fix(canary): adapt BESA routing guards to Fusion 0.75 * test(canary): make Fusion path fixtures Windows-safe * fix(BESA-167): port safe no-commit finalization * fix(BESA-167): enforce shared database node ownership * fix(BESA-167): close multiprocess node ownership races * fix(BESA-167): fence node ownership handoffs * fix(BESA-167): close node ownership race windows * fix(BESA-167): fence checkout recovery races * fix(BESA-167): fence local routes and lease renewals Fusion-Task-Id: BESA-167 * fix(BESA-257): honor node-local project path mappings * fix(engine): rebind protected foreign task branches * fix(BESA-256): add protected branch recovery changeset * fix(BESA-258): keep project memory on shared root Co-authored-by: Fusion <noreply@runfusion.ai> * fix(engine): expose shared memory in task worktrees * fix(engine): link memory before worktree refresh * fix(engine): keep heartbeat execution node-local * fix(engine): quote worktree refs for Windows shells * FN-8822: prevent worktree capacity leaks Keep live-task capacity admission visible and consistent across all lifecycle lanes. - Report canonical active holders when worktree or concurrent-task capacity is exhausted. - Persist deduplicated queue reasons for merge, triage, and workflow-continuation admission. - Cover retained worktree behavior and document the capacity model. Files changed: .changeset/fn-8822-worktree-capacity-leak.md | 7 +++ docs/architecture.md | 1 + docs/settings-reference.md | 4 +- .../src/__tests__/agent-heartbeat-worktree.test.ts | 11 ++++- .../engine/src/__tests__/project-engine.test.ts | 4 ++ ...ecutor-no-task-done-vs-worktree-reclaim.test.ts | 55 +++++++++++++++++++--- ...admission-worktree-ledger-renamed-lanes.test.ts | 4 ++ .../workflow-continuation-capacity.test.ts | 8 +++- .../src/__tests__/worktree-acquisition.test.ts | 6 +-- packages/engine/src/concurrency/concurrency.ts | 22 +++++++++ packages/engine/src/project-engine.ts | 34 ++++++++++++- packages/engine/src/runtimes/in-process-runtime.ts | 33 ++++++++++++- packages/engine/src/scheduler.ts | 53 ++++++++++----------- packages/engine/src/triage.ts | 18 +++++++ 14 files changed, 215 insertions(+), 45 deletions(-) Fusion-Task-Id: FN-8822 Fusion-Task-Lineage: 73eda70b-2b74-4b49-a0ff-29d41fd9aab8 Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai> * test(engine): normalize Windows worktree path assertions * test: make capacity guard file discovery cross-platform * fix(engine): avoid inert sync lane fallback * fix: let a node be re-attempted after its previous attempt ended terminal Tasks were parking with: Workflow principal fence write failed at node 'step-execute' — Workflow work item <id> is terminal (cancelled) and cannot be requeued as running A work item is keyed on (project_id, run_id, task_id, node_id, kind), and a node's fence run id is DERIVED rather than per-attempt (`<taskId>:<workflowId>:<nodeInstanceId>`), so every attempt at a node instance targets the same row. Fence rows are terminalized at the end of every run, so the SECOND visit to any node hit upsertWorkflowWorkItem's terminal guard and failed the fence closed. Retries, rework cycles (maxReworkCycles) and operator re-dispatch all make a second visit ordinary, so this caught any task that did not clear a node on its first pass — every step-execute row in the affected database was already terminal (succeeded, failed, or cancelled). The guard is correct; the identity was wrong — a new attempt is a new work item. replaceActiveTaskWorkflowContinuation is the sanctioned single writer for a task continuation and its contract is already "retire the predecessor, install the successor", so it now drops a TERMINAL row occupying the exact target key inside the same locked transaction before upserting. Bounded and non-destructive: at most one row per (task, node instance), the dropped row is finished work, and its transitions are already in run-audit. Terminal rows for other nodes are untouched and upsertWorkflowWorkItem still refuses to resurrect a terminal row for every other caller. Existing stuck rows heal on their next dispatch; no migration or cleanup needed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(core): cover workflow retry fences * fix: eagerly warm extension host stores from engine TaskStores (Runfusion#3340) ## Summary Warm the extension-host task stores **up front** at dashboard startup instead of letting the first `fn_task_*` call lazily boot a second PostgreSQL pool per project. ## What changed `packages/cli/src/commands/dashboard.ts`: - After the dashboard boots, iterate every registered project (from `centralCoreForEngine.listProjects()`) and call `setHostTaskStore(p.path, engine.getTaskStore())` for each non-cwd project that already has a running `ProjectEngine`. - Reuses each engine's **existing** `TaskStore` directly — no new backend connection, no schema advisory-lock contention, no extra connection-pool exhaustion. - `cwd` is skipped because its store is already injected at startup. - Per-project failures are non-fatal (warn) and a failed project listing logs a single warn — dashboard startup never blocks on this. - `.changeset/extension-host-store-warmup.md` (patch, fix). ## Why Left on its own, the first extension tool call (`fn_task_update`, `fn_task_archive`, `fn_agent_show`, …) for a non-cwd project falls through to `createTaskStoreForBackend`, which boots a **second** PostgreSQL connection pool on demand. On busy hosts that lazy boot can time out, or the call stalls behind pool/startup contention — the classic "first `fn_task_*` call is slow or errors" experience. Pre-populating from the already-running engines removes that lazy worst-case path entirely. ## Verification - `pnpm verify:fast` — PASS (13 steps, 115s): CLI `tsup` build green, scoped typecheck/build green, boot smoke green (`fn --help` + real `serve` with `GET /api/health` 200). - Cherry-picked cleanly onto current `origin/main` (`5532019fd`); branch is up-to-date with `origin/main` at PR time. ## Files - `packages/cli/src/commands/dashboard.ts` (+30) - `.changeset/extension-host-store-warmup.md` (new) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved dashboard startup reliability by reusing existing project task connections. * Prevented extension task tools from creating duplicate connection pools. * Added non-blocking warnings when individual project initialization or discovery fails. * Dashboard startup now reports how many project task stores were successfully prepared. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: gsxdsm <gsxdsm@users.noreply.github.com> * fix(engine): preserve resolved lanes across runtime boundaries * fix(engine): keep workflow failure cleanup non-blocking * fix: honor parked merge workflows without retry loops * fix: retry workflow after first terminal tool failure * fix(dashboard): preserve fresh planning status across stale fetches * fix: serialize mission slice admission * fix: resume workflows at top-level graph nodes * feat: route workflow stages through durable role agents * ci: build full cli package before agent-browser pack * fix(desktop): avoid mutable engine manager binding * ci: keep lane wiring gate portable * ci: allow agent browser packaging build time * test: model durable node reroute in scheduler cutover * test: align agent browser workflow contract * fix(dashboard): make optional storage cleanup lint-safe * fix(dashboard): preserve v1 retry lane census * fix(dashboard): resolve recommendation archive lanes * test: seed boot smoke with registered node identity * fix(engine): bound hold-release selection prefetch * fix(engine): reuse hold-release workflow selections * perf(engine): prefetch hold-release workflow definitions --------- Co-authored-by: Fusion <noreply@runfusion.ai> Co-authored-by: gsxdsm <gsxdsm@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: ischindl <ischindl@progis.sk>
* fix(mesh): preserve foreign-node worktree metadata * fix(engine): ignore pre-reset plan review logs * fix(core): keep plan reviews independent of worktree leaf * fix(engine): expose routed execution context * fix(cli): retry daemon after central db startup failure * fix(engine): scope executor task reads to project * fix(engine): respect node ownership during archive cleanup * fix(canary): adapt BESA routing guards to Fusion 0.75 * test(canary): make Fusion path fixtures Windows-safe * fix(BESA-167): port safe no-commit finalization * fix(BESA-167): enforce shared database node ownership * fix(BESA-167): close multiprocess node ownership races * fix(BESA-167): fence node ownership handoffs * fix(BESA-167): close node ownership race windows * fix(BESA-167): fence checkout recovery races * fix(BESA-167): fence local routes and lease renewals Fusion-Task-Id: BESA-167 * fix(BESA-257): honor node-local project path mappings * fix(engine): rebind protected foreign task branches * fix(BESA-256): add protected branch recovery changeset * fix(BESA-258): keep project memory on shared root Co-authored-by: Fusion <noreply@runfusion.ai> * fix(engine): expose shared memory in task worktrees * fix(engine): link memory before worktree refresh * fix(engine): keep heartbeat execution node-local * fix(engine): quote worktree refs for Windows shells * FN-8822: prevent worktree capacity leaks Keep live-task capacity admission visible and consistent across all lifecycle lanes. - Report canonical active holders when worktree or concurrent-task capacity is exhausted. - Persist deduplicated queue reasons for merge, triage, and workflow-continuation admission. - Cover retained worktree behavior and document the capacity model. Files changed: .changeset/fn-8822-worktree-capacity-leak.md | 7 +++ docs/architecture.md | 1 + docs/settings-reference.md | 4 +- .../src/__tests__/agent-heartbeat-worktree.test.ts | 11 ++++- .../engine/src/__tests__/project-engine.test.ts | 4 ++ ...ecutor-no-task-done-vs-worktree-reclaim.test.ts | 55 +++++++++++++++++++--- ...admission-worktree-ledger-renamed-lanes.test.ts | 4 ++ .../workflow-continuation-capacity.test.ts | 8 +++- .../src/__tests__/worktree-acquisition.test.ts | 6 +-- packages/engine/src/concurrency/concurrency.ts | 22 +++++++++ packages/engine/src/project-engine.ts | 34 ++++++++++++- packages/engine/src/runtimes/in-process-runtime.ts | 33 ++++++++++++- packages/engine/src/scheduler.ts | 53 ++++++++++----------- packages/engine/src/triage.ts | 18 +++++++ 14 files changed, 215 insertions(+), 45 deletions(-) Fusion-Task-Id: FN-8822 Fusion-Task-Lineage: 73eda70b-2b74-4b49-a0ff-29d41fd9aab8 Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai> * test(engine): normalize Windows worktree path assertions * test: make capacity guard file discovery cross-platform * fix(engine): avoid inert sync lane fallback * fix: let a node be re-attempted after its previous attempt ended terminal Tasks were parking with: Workflow principal fence write failed at node 'step-execute' — Workflow work item <id> is terminal (cancelled) and cannot be requeued as running A work item is keyed on (project_id, run_id, task_id, node_id, kind), and a node's fence run id is DERIVED rather than per-attempt (`<taskId>:<workflowId>:<nodeInstanceId>`), so every attempt at a node instance targets the same row. Fence rows are terminalized at the end of every run, so the SECOND visit to any node hit upsertWorkflowWorkItem's terminal guard and failed the fence closed. Retries, rework cycles (maxReworkCycles) and operator re-dispatch all make a second visit ordinary, so this caught any task that did not clear a node on its first pass — every step-execute row in the affected database was already terminal (succeeded, failed, or cancelled). The guard is correct; the identity was wrong — a new attempt is a new work item. replaceActiveTaskWorkflowContinuation is the sanctioned single writer for a task continuation and its contract is already "retire the predecessor, install the successor", so it now drops a TERMINAL row occupying the exact target key inside the same locked transaction before upserting. Bounded and non-destructive: at most one row per (task, node instance), the dropped row is finished work, and its transitions are already in run-audit. Terminal rows for other nodes are untouched and upsertWorkflowWorkItem still refuses to resurrect a terminal row for every other caller. Existing stuck rows heal on their next dispatch; no migration or cleanup needed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(core): cover workflow retry fences * fix: eagerly warm extension host stores from engine TaskStores (Runfusion#3340) ## Summary Warm the extension-host task stores **up front** at dashboard startup instead of letting the first `fn_task_*` call lazily boot a second PostgreSQL pool per project. ## What changed `packages/cli/src/commands/dashboard.ts`: - After the dashboard boots, iterate every registered project (from `centralCoreForEngine.listProjects()`) and call `setHostTaskStore(p.path, engine.getTaskStore())` for each non-cwd project that already has a running `ProjectEngine`. - Reuses each engine's **existing** `TaskStore` directly — no new backend connection, no schema advisory-lock contention, no extra connection-pool exhaustion. - `cwd` is skipped because its store is already injected at startup. - Per-project failures are non-fatal (warn) and a failed project listing logs a single warn — dashboard startup never blocks on this. - `.changeset/extension-host-store-warmup.md` (patch, fix). ## Why Left on its own, the first extension tool call (`fn_task_update`, `fn_task_archive`, `fn_agent_show`, …) for a non-cwd project falls through to `createTaskStoreForBackend`, which boots a **second** PostgreSQL connection pool on demand. On busy hosts that lazy boot can time out, or the call stalls behind pool/startup contention — the classic "first `fn_task_*` call is slow or errors" experience. Pre-populating from the already-running engines removes that lazy worst-case path entirely. ## Verification - `pnpm verify:fast` — PASS (13 steps, 115s): CLI `tsup` build green, scoped typecheck/build green, boot smoke green (`fn --help` + real `serve` with `GET /api/health` 200). - Cherry-picked cleanly onto current `origin/main` (`5532019fd`); branch is up-to-date with `origin/main` at PR time. ## Files - `packages/cli/src/commands/dashboard.ts` (+30) - `.changeset/extension-host-store-warmup.md` (new) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved dashboard startup reliability by reusing existing project task connections. * Prevented extension task tools from creating duplicate connection pools. * Added non-blocking warnings when individual project initialization or discovery fails. * Dashboard startup now reports how many project task stores were successfully prepared. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: gsxdsm <gsxdsm@users.noreply.github.com> * fix(engine): preserve resolved lanes across runtime boundaries * fix(engine): keep workflow failure cleanup non-blocking * fix: honor parked merge workflows without retry loops * fix: retry workflow after first terminal tool failure * fix(dashboard): preserve fresh planning status across stale fetches * fix: serialize mission slice admission * fix: resume workflows at top-level graph nodes * feat: route workflow stages through durable role agents * ci: build full cli package before agent-browser pack * fix(desktop): avoid mutable engine manager binding * ci: keep lane wiring gate portable * ci: allow agent browser packaging build time * test: model durable node reroute in scheduler cutover * test: align agent browser workflow contract * fix(dashboard): make optional storage cleanup lint-safe * fix(dashboard): preserve v1 retry lane census * fix(dashboard): resolve recommendation archive lanes * test: seed boot smoke with registered node identity * fix(engine): bound hold-release selection prefetch * fix(engine): reuse hold-release workflow selections * perf(engine): prefetch hold-release workflow definitions * perf(core): coalesce concurrent workflow IR reads * fix(engine): recover unregistered pinned worktree directories --------- Co-authored-by: Fusion <noreply@runfusion.ai> Co-authored-by: gsxdsm <gsxdsm@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: ischindl <ischindl@progis.sk>
Summary
Warm the extension-host task stores up front at dashboard startup instead of letting the first
fn_task_*call lazily boot a second PostgreSQL pool per project.What changed
packages/cli/src/commands/dashboard.ts:centralCoreForEngine.listProjects()) and callsetHostTaskStore(p.path, engine.getTaskStore())for each non-cwd project that already has a runningProjectEngine.TaskStoredirectly — no new backend connection, no schema advisory-lock contention, no extra connection-pool exhaustion.cwdis skipped because its store is already injected at startup..changeset/extension-host-store-warmup.md(patch, fix).Why
Left on its own, the first extension tool call (
fn_task_update,fn_task_archive,fn_agent_show, …) for a non-cwd project falls through tocreateTaskStoreForBackend, which boots a second PostgreSQL connection pool on demand. On busy hosts that lazy boot can time out, or the call stalls behind pool/startup contention — the classic "firstfn_task_*call is slow or errors" experience. Pre-populating from the already-running engines removes that lazy worst-case path entirely.Verification
pnpm verify:fast— PASS (13 steps, 115s): CLItsupbuild green, scoped typecheck/build green, boot smoke green (fn --help+ realservewithGET /api/health200).origin/main(5532019fd); branch is up-to-date withorigin/mainat PR time.Files
packages/cli/src/commands/dashboard.ts(+30).changeset/extension-host-store-warmup.md(new)Summary by CodeRabbit