Skip to content
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,13 @@ Versions follow [SemVer](https://semver.org).

## [Unreleased]

- Add `outerloop end <run-id> [--root <root>] [--note <text>]` to request an
operator ending at the next tick, including runs waiting for an unavailable
endpoint. The tick uses the existing ending cleanup and refuses later publish.
- Upgrading: existing run records need no backfill; an absent `end-request.json`
means no request. Older kernels ignore requests and do not recognize the new
`operator` ending when writing records; keep the updated kernel for these runs.

- Author overrides accept operator-only `session_minutes` (10–240) and
`session_max_turns` (10–300). Limits bind with the author selection and survive
settings changes across wakes and review replies. Contracts can still lower
Expand Down
2 changes: 1 addition & 1 deletion docs/design/agent-protocols.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ park, terminal at the ending. Legs are turns inside it.
An ending is a completion with an outcome, not a task state. A negative
result is successful work; a rejected PR is a human's decision, not the
agent rejecting the task. Only kernel-side failure (stuck, aborted) maps to
failed or canceled. The six endings travel as data on the final message.
failed or canceled. The endings travel as data on the final message.

| Outerloop | A2A | Note |
| --- | --- | --- |
Expand Down
28 changes: 18 additions & 10 deletions docs/design/lifecycle.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ through `attempt.run_author_leg`, including the steward's review work.
`followup.py` and its job are gone. GitHub collection positions live beside
the inbox, while the run record carries the delivered sequence. Replies are
posted by the author leg; only a submit invokes the gate and one publish.
The six endings and reports remain. Old state names and collection positions
The endings and reports remain. Old state names and collection positions
migrate on read. Each tick logs `legacy follow-up records: N` from raw live
records; operators must confirm zero across every fleet before deployment.

Expand All @@ -55,14 +55,14 @@ records; operators must confirm zero across every fleet before deployment.
| State | Meaning | Leaves it |
| --- | --- | --- |
| `running` | a session is live in a job | the session ends: it slept (park), or it stopped (end) |
| `parked` | no session; the run waits for jobs it launched, for messages, or both | a wake (the same session resumes), or a human ends the PR |
| `parked` | no session; the run waits for jobs it launched, for messages, or both | a wake (the same session resumes), or a PR or operator ending |
| `ended` | terminal, with a report | never |

A PR being open is a fact about a run, recorded in `pr_url`, not a state. A
parked run with a PR is what `in-review` was. `implementing` is `running`;
`waiting` and `in-review` are `parked`; `concluding` is deleted. The six
endings stay as they are: merged, rejected, negative result, budget exhausted,
aborted, stuck. They are how a human reads the board, and every one still
`waiting` and `in-review` are `parked`; `concluding` is deleted. The endings
are merged, rejected, negative result, budget exhausted, aborted, stuck, and
operator. They are how a human reads the board, and every one still
produces a report.

### One engine: park and wake
Expand Down Expand Up @@ -213,14 +213,22 @@ nothing. A run that has spent everything can still reply and end.
### Endings

A run ends when the author ends it, when its PR is merged or closed,
when the meter runs out, or when the kernel cannot continue: a crash, a
tampered workspace, or a kernel action that made no progress
`MAX_WAKE_ATTEMPTS` times, a failed publish retry included. A merge or close
when an operator requests it with `outerloop end <run-id>`, when the meter
runs out, or when the kernel cannot continue: a crash, a tampered workspace,
or a kernel action that made no progress
`MAX_WAKE_ATTEMPTS` times, a failed publish retry included. The operator command
atomically records the request time and optional note in the run directory; it
does not end the run itself. The tick records the ending as `operator` and keeps
the note as the ending note. A merge or close
ends the run at the next tick whatever it is doing: pending jobs are
cancelled, a session in flight finishes its leg and its publish is refused.
An operator request waits for the run's lease instead: a session in flight
finishes its leg, and its publish is refused unless it had already started, a queued wake exits without a leg,
and the next tick that holds the lease ends the run. If that leg ends the run
itself (a negative result, budget exhausted, stuck), its own ending stands.
Every ending writes the report, seals the line notebook, releases the issue
claim when no PR exists, and cancels the run's live launches. No other path
ends a run. A gate verdict never ends a run by itself, and a reviewer's
claim when no PR exists, and cancels the run's live launches. These are the only
paths that end a run. A gate verdict never ends a run by itself, and a reviewer's
comment never does.

## What stays rigid
Expand Down
15 changes: 15 additions & 0 deletions docs/install.md
Original file line number Diff line number Diff line change
Expand Up @@ -699,6 +699,21 @@ message delivery, GitHub polling, self-merge sweep, board, and ending records
continue; existing runs keep spending, including their panels, author sessions,
and the authors' own `launch` submissions.

To end one run, use `outerloop end <run-id> [--root <root>] [--note <text>]`.
The root defaults to `OUTERLOOP_ROOT` in the environment or operator settings,
then `~/.outerloop`. The command atomically writes `end-request.json` in the
run directory with the requested time and note. Unknown and already ended runs
are refused; repeating a pending request preserves its original time and note.
The run ends as `operator`, with the same cleanup as a PR merge or close, at the
first active tick when no session holds it, even without a PR or an available
model endpoint. A session in flight finishes its leg and cannot start a publish (one
already under way completes, and the run ends right after the leg); a queued
wake exits without starting one. Pending jobs are cancelled. A run with an issue
waits until GitHub is reachable, so the issue is told. If a session in flight
ends the run itself first, its own ending stands.
The slot becomes free, and the next claim uses the current settings, including
`OUTERLOOP_AUTHOR_OVERRIDES`. A paused loop must resume to process the request.

**Live operator ceilings.** Create `<root>/limits.toml` to limit this fleet while
leaving target contracts under their normal review process:

Expand Down
48 changes: 47 additions & 1 deletion src/outerloop/attempt.py
Original file line number Diff line number Diff line change
Expand Up @@ -3753,7 +3753,7 @@ def publish(
) -> AttemptOutcome:
"""Publish a credited sealed tree: open a PR or fast-forward its head."""
latest = load_record(run_root, run_id)
if latest.state == ENDED:
if latest.state == ENDED or end_requested(run_root, run_id):
Comment thread
renmengye marked this conversation as resolved.
return AttemptOutcome(run_id=run_id, outcome="publish-refused", pr_url=latest.pr_url)
meter = {
k: v
Expand Down Expand Up @@ -5267,6 +5267,11 @@ def _run_id(value: str) -> str:
# a wake must never crash on an unreadable/odd record — fall back to
# the claude author (resume_author), same fail-safe as the sweep
_wake_record = None
if end_requested(Path(args.run_root), args.resume):
Comment thread
renmengye marked this conversation as resolved.
# an operator ending is pending: no new leg; the tick ends the run
# once this wake's lease is free
_release_own_lease(args.run_root, args.resume)
return 0
wake_limits = bound_limits(getattr(_wake_record, "author_limits", None))
if wake_limits is not None:
if args.job_minutes:
Expand Down Expand Up @@ -5366,6 +5371,10 @@ def _run_id(value: str) -> str:
else None,
)
wake_secrets = tuple(k for k in (bot_auth.token(), *wake_panel_secrets, wake_api_key) if k)
if end_requested(Path(args.run_root), args.resume):
# requested during setup: still no new leg
_release_own_lease(args.run_root, args.resume)
return 0
try:
resumed = resume_run(
args.run_root,
Expand Down Expand Up @@ -5575,6 +5584,39 @@ def withdraw_pr(
return ""


def end_requested(run_root: Path, run_id: str) -> bool:
from outerloop.runstate import END_REQUEST_NAME

return (run_dir_of(run_root, run_id) / END_REQUEST_NAME).is_file()


def end_on_request(
run_root: Path, record: RunRecord, github: GitHubClient | None, now: float
) -> str:
"""End a run an operator asked to end. The caller holds the run's lease, so
no session leg (and no publish) is in flight."""
from outerloop.runstate import OPERATOR

record = load_record(run_root, record.run_id)
if record.state == ENDED or not end_requested(run_root, record.run_id):
return ""
if record.issue_number and github is None:
return "" # the issue comment needs GitHub; a later tick ends it
finish_run(run_root, record, OPERATOR, requested_note(run_root, record.run_id), now, github)
return OPERATOR


def requested_note(run_root: Path, run_id: str) -> str:
from outerloop.runstate import END_REQUEST_NAME

try:
request = run_dir_of(run_root, run_id) / END_REQUEST_NAME
note = json.loads(request.read_text()).get("note", "")
return note if isinstance(note, str) else ""
except (OSError, ValueError, AttributeError):
return "" # the request still stands; a damaged note never blocks the ending


def close_if_done(run_root: Path, record: RunRecord, github: GitHubClient, now: float) -> str:
"""Finish a withdrawal intent, then route the PR ending through the run terminal."""
from outerloop.github import GitHubError
Expand Down Expand Up @@ -5733,6 +5775,10 @@ def _ending_comment(record: RunRecord, ending: str) -> str:
"""
from outerloop.steward import MAX_STEWARD_ATTEMPTS, RELEASE_MARKER

if ending == "operator":
# As for any ending without a PR, nothing else will resolve the issue.
release = "" if record.pr_url else f"{RELEASE_MARKER}\n"
return f"{release}Run `{record.run_id}` was ended by an operator."
if ending == "merged":
return (
f"Pull request {record.pr_url} was merged; run `{record.run_id}` is "
Expand Down
32 changes: 32 additions & 0 deletions src/outerloop/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -860,6 +860,32 @@ def upgrade(args: argparse.Namespace) -> int:
return 0


def end(args: argparse.Namespace) -> int:
import time

from outerloop.runstate import request_end

try:
values = env_file_values(keys=("OUTERLOOP_ROOT",))
root = Path(
args.root
or os.environ.get("OUTERLOOP_ROOT")
or values.get("OUTERLOOP_ROOT")
or DEFAULT_LOCAL_ROOT
).expanduser()
created = request_end(root, args.run_id, args.note, time.time())
except (ValueError, OSError) as exc:
print(f"outerloop end: {exc}", file=sys.stderr)
return 2
status = "requested" if created else "already requested"
print(
f"Run {args.run_id}: ending {status}. At the next tick, the run will end as "
"operator and its pending jobs will be cancelled; a session in flight will "
"finish its leg and its publish will be refused."
)
return 0


def main(argv: list[str] | None = None) -> int:
from outerloop import __version__

Expand Down Expand Up @@ -934,6 +960,10 @@ def main(argv: list[str] | None = None) -> int:
from outerloop import init

return init.main(argv[1:])
p = sub.add_parser("end", help="request an operator ending at the next tick")
p.add_argument("run_id")
p.add_argument("--root", help="state root (defaults to OUTERLOOP_ROOT or ~/.outerloop)")
p.add_argument("--note", default="", help="reason for ending the run")
p = sub.add_parser("limits", help="show live operator ceilings and fleet GPU usage")
p.add_argument("--root", help="state root (defaults to OUTERLOOP_ROOT or ~/.outerloop)")
p = sub.add_parser("status", help="show local runs and endpoint outages (read-only)")
Expand Down Expand Up @@ -981,6 +1011,8 @@ def main(argv: list[str] | None = None) -> int:
print(str(exc), file=sys.stderr)
return 1
return 0
if args.command == "end":
return end(args)
if args.command == "permissions":
return permissions(args)
if args.command == "upgrade":
Expand Down
30 changes: 28 additions & 2 deletions src/outerloop/runstate.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,16 +34,18 @@

STATES = (RUNNING, PARKED, ENDED)

# The six endings ("The life of a run" — every one produces a report).
# The endings ("The life of a run" — every one produces a report).
MERGED = "merged"
REJECTED = "rejected"
NEGATIVE_RESULT = "negative-result"
BUDGET_EXHAUSTED = "budget-exhausted"
ABORTED = "aborted"
STUCK = "stuck"
OPERATOR = "operator"

ENDINGS = (MERGED, REJECTED, NEGATIVE_RESULT, BUDGET_EXHAUSTED, ABORTED, STUCK)
ENDINGS = (MERGED, REJECTED, NEGATIVE_RESULT, BUDGET_EXHAUSTED, ABORTED, STUCK, OPERATOR)

END_REQUEST_NAME = "end-request.json"
RECORD_NAME = "state.json"
LEASE_NAME = "lease.json"

Expand Down Expand Up @@ -190,6 +192,30 @@ def run_dir(root: Path, run_id: str) -> Path:
return root / "runs" / run_id


def request_end(root: Path, run_id: str, note: str, now: float) -> bool:
"""Atomically retain the first operator request without changing the run."""
if not run_id or run_id in (".", "..") or Path(run_id).name != run_id:
raise ValueError(f"unknown run id: {run_id}")
directory = run_dir(root, run_id)
if not (directory / RECORD_NAME).is_file():
raise ValueError(f"unknown run id: {run_id}")
with (directory / ".record-lock").open("a") as lock:
fcntl.flock(lock, fcntl.LOCK_EX)
record = load_record(root, run_id)
if record.ended():
raise ValueError(f"run {run_id} already ended ({record.ending})")
path = directory / END_REQUEST_NAME
if path.exists():
return False
tmp = directory / f".{END_REQUEST_NAME}.{os.getpid()}.tmp"
try:
tmp.write_text(json.dumps({"requested_at": now, "note": note}) + "\n")
os.replace(tmp, path)
finally:
tmp.unlink(missing_ok=True)
return True


def save_record(root: Path, record: RunRecord, now: float) -> None:
"""Serialize record writers; a persisted ending cannot be replaced."""
directory = run_dir(root, record.run_id)
Expand Down
46 changes: 41 additions & 5 deletions src/outerloop/tick.py
Original file line number Diff line number Diff line change
Expand Up @@ -1141,6 +1141,25 @@ def wake(record: RunRecord, reason: str, tag: str) -> None:
migrate_inbox(root, record.run_id, now)
finally:
release_lease(root, record.run_id)
from outerloop.attempt import end_on_request, end_requested

# An operator ending waits for the lease: a live session finishes its
# leg (its publish is refused) and a queued wake exits without one.
# A running record is a session in flight (a fresh climb holds no lease):
# it finishes its leg; _sweep_running ends it if its job died.
if (
not dry_run
and record.state != RUNNING
and end_requested(root, record.run_id)
and acquire_lease(root, record.run_id, holder, "", now)
Comment thread
renmengye marked this conversation as resolved.
):
try:
ending = end_on_request(root, record, github, now)
finally:
release_lease(root, record.run_id)
if ending:
ended.append((record.run_id, ending))
continue
merged = False
blessed_before = record.auto_blessed_head
try:
Expand Down Expand Up @@ -1570,14 +1589,26 @@ def _sweep_running(
# after its run is declared dead must never survive one
with contextlib.suppress(Exception):
compute.cancel(jid)
from outerloop.attempt import finish_run

from outerloop.attempt import end_requested, finish_run, requested_note
from outerloop.runstate import OPERATOR

# A pending operator request names the ending: its session died first.
requested = end_requested(root, fresh.run_id)
if requested and fresh.issue_number and github is None:
continue # the operator ending must tell its issue; a later tick ends it
if requested:
ending, note = OPERATOR, requested_note(root, fresh.run_id)
else:
ending = ABORTED
note += " — ended by the sweep (a killed climb leaves no exception to contain)"
finish_run(
Comment thread
renmengye marked this conversation as resolved.
Comment thread
renmengye marked this conversation as resolved.
root,
fresh,
ABORTED,
f"{note} — ended by the sweep (a killed climb leaves no exception to contain)",
ending,
note,
now,
# an operator ending tells the requesting issue, like every other path
github if requested else None,
auth=getattr(github, "auth", None),
bot_login=bot_login,
)
Expand All @@ -1588,7 +1619,7 @@ def _sweep_running(
try:
report_path.write_text(
f"# Run report — {record.target} / {record.benchmark}\n"
f"Outcome: **aborted** (climb job killed)\n"
f"Outcome: **{ending}** (climb job killed)\n"
f"Note: {note}\n"
)
except OSError as exc:
Expand Down Expand Up @@ -1669,6 +1700,11 @@ def _sweep_one(
return # a concurrent tick reaped it first; it owns redelivery
reaped.append(record.run_id)

from outerloop.attempt import end_requested

if end_requested(root, record.run_id):
return # an operator ending is pending: never wake the run again

from outerloop.inbox import wake_pending

job_ids = _poll_targets(record)
Expand Down
Loading
Loading