feat: implement bounded parallel story fan-out with isolated worktrees

[rasanderoland]

Context

scm.max_parallel already exists, but upstream main clamps every value greater than 1 to 1 because Phase 5 parallel fan-out is not implemented. The engine remains sequential: it dispatches one story, runs its development/review/verification flow, integrates it, and only then selects the next story. Worktree isolation, per-unit branches, and merge-back already exist. This is an upstream bmad-loop Phase 5 design request, not an OpenEngine-specific fork.

Design request

Implement bounded parallel story fan-out with these invariants:

• A configured scm.max_parallel greater than 1 must hard-fail unless scm.isolation is worktree and scm.branch_per is story. Keep scm.max_parallel = 1 behavior unchanged.
• Admit work using declared file globs plus named exclusive resources, such as database, lockfile, service:name, generated artifacts, or shared verification infrastructure. File/resource conflicts serialize. Unknown or invalid footprints are exclusive, and the journal must record why. Shared verification infrastructure should be able to reserve migrations, test databases, or services so parallel source work cannot race them.
• Allow up to N workers in flight, but keep exactly one coordinator/integration lane. Workers must never write the target branch concurrently.
• Persist deterministic dispatch and integration order in run state. Completion timing must not determine run history.
• Each worker must record the target base SHA from which it was started.
• Integrate each worker against the latest target through a candidate branch/worktree. Run deterministic verification commands against the combined candidate, then advance the target only if its SHA is still the candidate's recorded target SHA; never force-advance a target that moved.
• Define and test stale-base behavior for all merge strategies: ff may advance only when the candidate is a valid fast-forward from the current target; merge must compose against the latest target and preserve/escalate conflicts; squash must create its deterministic squash candidate against the latest target. A stale target must trigger deterministic recomposition/retry or an explicit escalation, with the candidate and evidence preserved.
• Define one worker pause/escalation behavior: the coordinator stops dispatching, lets already-running siblings reach a safe terminal boundary, persists their outcomes, and pauses without silently dropping results. If maintainers prefer cancel-on-first-failure, make that an explicit design choice rather than an assumption.
• Persist durable state for every in-flight worker: task, worktree, branch, phase, session, and base SHA. Support crash-safe resume, stop, orphan cleanup, and useful TUI/status/attach behavior for these workers and candidate integrations.
• Preserve the existing semantics of max_stories, epic boundaries, selectors, plan/spec/done checkpoints, notifications, keep_failed, delete_branch, seed_adapter_defaults, and worktree_seed.
• rollback_on_failure remains an in-place-only knob and is irrelevant in parallel worktree mode.
• Defer sweep-bundle parallelism unless maintainers explicitly opt it into this issue.

Acceptance tests

The implementation should cover:

• N-in-flight cap.
• Overlap of independent stories, proven with deterministic barriers.
• Serialization of file/resource conflicts.
• Unknown footprint treated as exclusive, with the journal reason.
• Isolation and branch_per guards.
• Deterministic merge order when completion order is inverted.
• ff, merge, and squash stale-base paths.
• Post-composition verification failure.
• Target SHA movement during integration.
• Merge-conflict preservation and escalation.
• A worker pause while siblings continue to a safe terminal boundary.
• Crash/resume, stop, and orphan cleanup.
• Regression equivalence for max_parallel = 1.

A manual two-story timing run is supplementary only; deterministic barrier-based tests are the primary concurrency proof.

Implementation Notes

Relevant current-main seams include policy parsing and validation; the engine dispatch loop; per-unit worktree creation, integration, and cleanup; persisted run state and journal; adapter sessions; verification; orphan/reconciliation cleanup; sweep; and diagnostics/TUI/status/attach surfaces. Please use the current symbols and contracts in those areas without depending on exact line numbers, which are not stable.

[coderabbitai]

🔗 Related PRs

#65 - feat(stories): spec-to-loop contract layer (Phase 1) [merged]
#77 - feat(stories): spec-to-loop engine, adapter, CLI + HITL checkpoints (Phase 2 + 3) [merged]
#86 - feat(resolve): intent-gap patch-restore re-drive + inline-review-layer preflight (re-cut) [merged]
#152 - fix(worktree): detach kept worktree so branch_per=run survives a kept failure (#138) [merged]
#173 - fix: keep committed work across a dev retry; accept the baseline it advances past (#172) [open]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[Polloinfilzato]

Two pieces of field data, from running 0.8.1 with isolation = "worktree" unattended overnight for about a week (~30 stories, macOS, claude adapter). I got parallelism ahead of this issue by starting several bmad-loop run --story <key> processes concurrently, so I have some empirical exposure to the merge-back half of this design.

1. Binary file-overlap admission over-serializes.

The admission rule here — declared file globs, unknown/invalid footprints are exclusive — matches what I converged on independently, and the "unknown is exclusive" default is right. But treating any declared overlap as a conflict was, for me, unusable in practice: in one epic nearly every story touched the package's loop/__init__.py, so a binary rule would have serialized the entire epic and removed the feature's whole benefit.

The distinction that actually worked was between two kinds of shared file:

• structural / append-only — __init__.py, a registry or factory, a changelog-shaped list. Git auto-merges non-adjacent edits here, and even a genuine conflict is trivial to reconcile. These should not block admission.
• shared core logic module — genuinely exclusive.

I don't have a clean proposal for expressing that in the footprint schema (an explicit append_only: list next to the exclusive globs is the obvious shape, but it puts the burden on whoever authors the spec). Flagging it mainly because a glob rule that can't say "shared, but low-risk" will look correct in tests and over-serialize on real codebases.

2. The "one integration lane" invariant doesn't exist yet, and is reachable today.

file_lock is defined in platform_util and covered in tests/test_platform_util.py, but as far as I can tell it is never imported by any production module — _merge_local (reached from _integrate_unit) merges into the shared target checkout with no mutual exclusion. (Edited to cite symbols instead of line numbers, per the note in the issue body — I'd written them against 0.8.1 and they have already moved.)

Latent while max_parallel is clamped, but reachable now via concurrent --story runs. Worth recording the observable symptom, because it isn't corruption: git's own index.lock keeps it safe, so a collision surfaces as a spurious escalation whose message is textually indistinguishable from a real content conflict —

merge of <branch> into <target> failed (content conflict against the target)
merge of <branch> into <target> blocked: the target checkout has uncommitted changes that are not part of this branch

The only discriminator I found was behavioural: retry the merge in a quiet window (no index.lock, no MERGE_HEAD, no sibling mid-merge). Clean apply → it was a lock race. Conflict markers → genuine file overlap, i.e. the batch was mis-scoped. Staggering run starts shrank the window enough to make that rare, but it's a workaround for exactly the invariant this issue is proposing, so it should stop being necessary.

Happy to expand on either point if useful.

[pbean]

Triage 2026-08-08: #51's parallel-dispatch half folds in here — same design space (bounded fan-out, isolated worktrees, merge-back seams; verified still absent on main — the scm.max_parallel clamp plus sequential dispatch is current reality). Still needs maintainer scoping before any PR. The smallest independently landable neighbor in this design family is #270.