Skip to content

Validate hunter findings with an orchestrator before filing; autofix and merge behind an independent PR review #211

Description

@ParallelEntrepreneur

Goal

Findings from bug-hunting settlers must not go straight to GitHub. An orchestrator validates them first; validated findings become issues; the operator can opt into autofix and merge; and every resulting PR is reviewed by a fresh session running the orchestrator model — never by the colony that wrote it.

Evidence

  • Findings already exist: crates/colonizer/src/findings.rsFinding {title, body, evidence} (~27), filed as GitHub issues with Filed::Issue | Duplicate (~34), capped at MAX_PER_COLONY = 5 (~20). They are emitted as finding events into events.jsonl and counted by scripts/colony-report.mjs, but are not exposed over HTTP.
  • Model routing already exists per colony: Session.model_tier, model_routing, routed_cost_usd (web/src/types.ts ~50–55) plus the gateway/provider layer (crates/colonizer/src/providers.rs, gateway.rs, routing module). "Use the orchestrator model" is therefore a routing decision, not new plumbing.
  • POST /api/sessions launches a colony, POST /api/sessions/{id}/publish publishes, and Session.publish_stage (crates/colonizer/src/publish.rs) tracks PR state. Merges are not currently acted on (see the open issue about PR state stopping at pr_opened).
  • Autopilot plus watchdog attention reasons (stalled, waiting_for_answer, nudges_exhausted, autopilot_held) give the hold/resume primitives an orchestrator gate needs.

Requested shape

  1. Expose findings. Serve a colony's findings over HTTP (per session, plus an aggregate feed) including their Filed state, so the cockpit can show what was found and what was filed.
  2. Validation stage. Every finding goes to an orchestrator session before filing. It decides: real or not, duplicate-of, severity — and only then files the issue, reusing the findings.rs filing path including duplicate detection. Rejected findings are recorded with a reason and shown as rejected, never silently dropped.
  3. Autofix + merge (opt-in, per run and globally). When enabled, a validated finding spawns a fix colony on its own branch; its PR is reviewed by a new session on the orchestrator model with no shared context with the author. Merge only on a passing review; a failing review posts the review as a PR comment and leaves the PR open for a human.
  4. Independence is a hard invariant. The reviewer must be a different session id with a fresh context. Enforce it in code — reject a review whose session is the author's — and cover it with a test.
  5. Auditability. Each stage emits an event (finding, validated, rejected, fix_colony, review, merged) so scripts/colony-report.mjs and the cockpit timeline can reconstruct the chain and the overview can show per-stage counts.
  6. Default behaviour is validate-and-file only. Autofix, and autofix+merge, are separate escalating opt-ins.

Done when

  • Findings are readable over HTTP with their filing state.
  • No finding reaches GitHub without an orchestrator validation event; rejections carry a reason.
  • Autofix produces a PR reviewed by a distinct fresh session on the orchestrator model; self-review is rejected by construction and covered by a test.
  • Merge happens only on a passing review and only when merge is explicitly enabled.
  • The cockpit shows the chain finding → validated → fix → review → merged, and docs/protocol.md documents the new events and endpoints.

Credited to @colonizer-settlers as co-author on this issue.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions