Skip to content

Let a judge model answer a colony's questions when nobody is there - #145

Merged
ParallelEntrepreneur merged 1 commit into
mainfrom
feat/autonomous-judge
Sep 18, 2026
Merged

ParallelEntrepreneur merged 1 commit into
mainfrom
feat/autonomous-judge

Conversation

@ParallelEntrepreneur

@ParallelEntrepreneur ParallelEntrepreneur commented Sep 18, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #143.

What

A colony that asks a question stops until a person answers. That is right when someone is watching, and it is the whole cost of the thing when nobody is.

Settings → Modules → Autonomy. off by default. Set it to judge, pick a model, and a question nobody has answered goes to that model with the task and the options the agent offered. The colony carries on.

Provider      [ judge         ▾ ]
Judge model   [ fable            ]
Answer after  [ 10 ] minutes unanswered
Answers/colony[ 5 ]
Free text     [ off ]

Any model, and why the field says what it says

The judge field takes any model, because the right one depends on what you run colonies for. It also says plainly that a frontier model judges best — it is deciding for you, on less context than you have, and the difference shows.

  • A plain id (fable, opus, sonnet) goes to Anthropic with the saved Claude credential.
  • provider/model goes to a provider you added, on its own API key.

Worth knowing before merging: when the saved credential is a subscription token from claude setup-token, it was issued for Claude Code to use inside a colony. Judging from the Mothership is a use it was not issued for. That is a deliberate choice (asked and confirmed), not an oversight, and it is written into the code comment and the commit so it stays visible.

The line it is built around

The judge chooses only among the labels the agent offered, and the reply is checked against them before anything is sent. A colony reads repository content, content can carry instructions, and a judge reading the same content can be steered by it.

The judge does this What happens
Picks a label nobody offered Left for you; this colony stops being judged
Leaves a question out Same
Replies with prose instead of the JSON Same
Answers a free-text question while free_text is off Same
Keeps being needed past max_answers Left for you; the watchdog flags it

Nothing pretends a judged answer is a person's: the colony is told so in the answer itself, and the session log records the model and its reason.

Test plan

  • cargo test: 178 pass, 6 new — the prompt carries every option; an unoffered label is refused; a question left out is refused; free text refused unless switched on; several labels only where several were invited; prose around the JSON tolerated, prose instead of it not; off until it has a model
  • Clippy matches main (12 warnings, none in autonomy.rs); tsc --noEmit and vite build clean
  • End to end on a separate instance against a stub model that answers like a judge: colony asked → judge answered "tests/smoke.rs" → colony carried on and volunteered that it had been "noted as an automatic stub/judge response in autonomous mode, not an actual maintainer". Status returned to idle with no attention flag.
  • The Anthropic path is not proven. The saved subscription token authenticates — the API answers 429, not 401 — but this account is rate-limited, so no completion came back through it. The provider path is fully exercised; the plain-id path is not.

Not in this PR

  • The judge reviewing its own answers, or a second model checking the first.
  • Anything about publishing autonomously — autopilot already does that.
  • Answering by rule (always the first option, always the safest-sounding). Worth saying no to explicitly: it looks like this feature and is a worse one.

Rebuilt on main rather than merged: this branch predated the tree-wide
format (#142) and the sessions.rs split (#132), so its diff collided with
both — the agent-link code it edits now lives in events.rs, and every hunk
fought the reflow. The feature is unchanged; only where it attaches moved.

- `autonomy.rs` is taken as it was written.
- The open question carries its questions now (`Runtime::open_question`
  returns `(id, questions)`), so the judge can answer among the labels the
  agent offered — and the dispatch that fills it is the typed
  `AgentEvent::Question` arm in events.rs, which already had them.
- `Activity::judged`, the `autonomy` module kind and its settings, the
  config entry, the docs and the web bits are as the branch had them.

Verified on this tree: cargo test --workspace 246 passed (240 + this
branch's 6), clippy -D warnings clean, cargo fmt --all --check clean, and
the web suite 67 passed with tsc clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ParallelEntrepreneur

Copy link
Copy Markdown
Collaborator Author

Rebuilt on main and force-pushed. The branch predated the tree-wide format (#142) and the sessions.rs split (#132), so it no longer merged: the agent-link code this edits now lives in events.rs, and every hunk fought the reflow.

The feature is unchanged — autonomy.rs is taken as written. Only where it attaches moved: Runtime::open_question returns (id, questions) and the dispatch that fills it is the typed AgentEvent::Question arm, which already carried the questions.

Verified on the rebuilt tree: cargo test --workspace 246 passed (240 + this branch's 6), clippy -D warnings clean, cargo fmt --all --check clean, web suite 67 passed with tsc clean.

@ParallelEntrepreneur
ParallelEntrepreneur merged commit c1db524 into main Sep 18, 2026
8 checks passed
@ParallelEntrepreneur
ParallelEntrepreneur deleted the feat/autonomous-judge branch September 18, 2026 10:41
ParallelEntrepreneur added a commit that referenced this pull request Sep 23, 2026
Let a judge model answer a colony's questions when nobody is there
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Autonomous mode: a judge model answers a colony's questions, on a model you pick

1 participant