Skip to content

Refuse out-of-scope launches and submits instead of ending the run #442

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
renmengye merged 2 commits into from
Sep 30, 2026
Merged

Refuse out-of-scope launches and submits instead of ending the run #442

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
620dfcc
Refuse out-of-scope launches and submits instead of ending the run
renmengye Sep 30, 2026
2630a39
Scope refusals repeat and never end the run; plain message with the a…
renmengye Sep 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,23 @@ Versions follow [SemVer](https://semver.org).

## [Unreleased]

- Launch, submit, and stale-submit checkpoint scope violations now refuse every
request and resume the author with the offending paths and bounded allowed
scope in the kernel inbox. Later refusals say “Refused again:”. Refusals
repeat and never end the run; session walltime and contract sleep, launch,
and GPU-hour budgets bound the loop. Refusal seals nothing, runs no jobs or
measurements, and spends no request budget. The authoritative measurement
scope check remains terminal. Abandoning a refused tree ends normally
without measurement, a line snapshot, or a push; outage and budget endings
also preserve the rejection. Scope admission precedes malformed request
and budget refusals.
- Upgrading: no action or backfill needed. Scope refusals use the existing
kernel note payload and refusal keys; old inboxes, ended runs, and in-flight
runs remain readable without rewriting delivered messages. The next request
uses the new admission behavior. The rejection flag is in-memory only;
no run-record fields change; rollback is
safe and restores terminal scope admission checks.

- Deployment author overrides select a backend/model per target or agent slot,
bind it to each run, and leave panel and CI reviewer inheritance on the fleet
author. Per-run board details identify the effective author and overrides.
Expand Down
16 changes: 15 additions & 1 deletion docs/design/lifecycle.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -228,14 +228,28 @@ comment never does.
| Kept | Why it is the kernel's |
| --- | --- |
| the paired measurement, the private seed, the floor, the suite, the cached baseline rule, the zero-change rule (an unchanged tree cannot be credited), the verdict bound to its sealed tree, base and contract | the number must be nobody's claim |
| scope on the diff before anything is sealed, launched or measured | the out-of-scope edit could be to the ruler |
| scope on the diff before anything is sealed, launched or measured; uncharged refusals repeat and never end the run | the out-of-scope edit could be to the ruler |
| containment, the lane from the contract, `--nice` on launches, always queue, cancel on end | the session cannot hold GPUs or credentials |
| launch, sleep and GPU-hour counts; refusal on exhaustion with the numbers | the meter is the only bound on spend |
| the publish: open or fast-forward the PR head to the sealed tree, the ledger row rule, disarm before a head moves, refuse when a human pushed or the contract moved; under `merge: auto`, record the blessed head and let the sweep merge a clean, quiet PR at that head only, never arm GitHub auto-merge; otherwise humans merge | credit, merge authority, and nobody's work overwritten |
| the steward's ruler measurement: full suite, sibling smoke checks, baseline reset | a ruler change must be verified as one |
| standing: which comments are messages, the bot's own markers, the task label; the issue claim and its release; one delivery per message | authorization and liveness |
| leases, the sweep, deadline floors, the stuck cap, the outage latch, the tamper guard, the report on every ending, the line seal at every terminal | liveness and audit |

Scope admission for launch, submit, and stale-submit checkpoints refuses every
out-of-scope request with a kernel inbox note listing up to ten offending paths
and the contract's allowed scope (up to ten entries and 1,000 characters).
Later refusals in the same run start with “Refused again:”. Nothing is sealed,
launched, measured, or charged for a refused request. Refusals repeat and never
end the run, including requests with malformed content or budget problems.
The session walltime and contract sleep, launch, and GPU-hour budgets bound
the loop. A session that cannot resume ends as `session-error`; endpoint loss
during refusal delivery ends as `session-outage` without sealing the rejected
tree. Abandoning a rejected tree ends without measuring, sealing, or pushing it.
Every refusal remains in the inbox for operators and later readers. The
authoritative scope re-check in `measure_and_decide`, including wake re-entry,
remains terminal. Judges still receive the run record.

## What becomes the author's, and what is deleted

| Today | After |
Expand Down
4 changes: 2 additions & 2 deletions src/outerloop/attempt.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3084,7 +3084,7 @@ def _wake_author(
result = dc_replace(result, submit_report=str(stage.get("report") or "no report was given"))
report_path = run_dir / "report.md"
report_path.write_text(result.report(config, redact_secrets=secrets))
if not snapshot_attempted:
if not snapshot_attempted and not result.tree_rejected:
_push_line_snapshot(
ws,
_line_ref_for(bench, config.agent_id),
Expand Down Expand Up @@ -3557,7 +3557,7 @@ def _finish_attempt(
)
report_path = run_dir / "report.md"
report_path.write_text(result.report(config, redact_secrets=secrets))
if not snapshot_attempted:
if not snapshot_attempted and not result.tree_rejected:
_push_line_snapshot(
ws, line_ref, run_id, result.outcome, secrets, bot_login=config.bot_login
)
Expand Down
184 changes: 135 additions & 49 deletions src/outerloop/orchestrator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -520,6 +520,8 @@ class AttemptResult:
panel_rounds: int = 0
panel_blocking_open: bool = False
panel_degraded: bool = False
# In-memory ending guard: rejected trees must never reach line snapshots.
tree_rejected: bool = False

def report(self, config: RunConfig, redact_secrets: tuple[str, ...] = ()) -> str:
lines = [
Expand Down Expand Up @@ -862,6 +864,7 @@ def measure_and_decide(
if violations:
return AttemptResult(
outcome="scope-violation",
tree_rejected=True,
note=f"out-of-scope paths: {', '.join(sorted(violations)[:10])}",
run_seed=seed,
)
Expand Down Expand Up @@ -1488,11 +1491,23 @@ def _ack(messages: list[Message]) -> None:
baseline_note = ""
measured: tuple[str, ...] = ()
refused_once = False
# Rejection protects stop/error paths; it is never a refusal limit.
tree_rejected = False
scope_refused = any(
m.source == "kernel"
and m.kind == "note"
and m.key.startswith("refusal:")
and (
str(m.payload.get("text", "")).startswith(("Refused:", "Refused again:"))
or str(m.payload.get("quoted_text", "")).startswith("out-of-scope paths")
)
for m in pending_messages(inbox_dir, 0)
)
# Reuse a negative only for the same measurement base and candidate tree.
failed_gate: tuple[str, str, AttemptResult] | None = judged
tree = tree_of or (lambda sha: sha)

def _resume(message: Message) -> AttemptResult | None:
def _resume(message: Message, *, allow_checkpoint: bool = True) -> AttemptResult | None:
"""Deliver pending messages; return an ending only if the resume fails."""
nonlocal session
if on_meter is not None:
Comment thread
renmengye marked this conversation as resolved.
Expand All @@ -1517,7 +1532,17 @@ def _resume(message: Message) -> AttemptResult | None:
wake_result = run_role(
spec, harness, prompt, workspace, resume_session_id=session.session_id
)
except EndpointUnavailable:
except EndpointUnavailable as exc:
if not allow_checkpoint or scope_validator(list(changed_paths()), contract):
# A rejected tree cannot be sealed even to preserve an outage park.
return AttemptResult(
outcome="session-outage",
tree_rejected=True,
baseline=baseline,
session=session,
note=f"scope refusal delivery unavailable: {exc}",
run_seed=run_seed,
)
# A prior leg already ran: keep its native context and budget meters.
raise RunParked(
phase="author-sleep",
Expand All @@ -1534,6 +1559,18 @@ def _resume(message: Message) -> AttemptResult | None:
judged=failed_gate,
capacity_wait=True,
) from None
except Exception as exc:
if not tree_rejected:
raise
# Preserve rejection even when the harness fails without a session result.
return AttemptResult(
outcome="session-error",
tree_rejected=True,
baseline=baseline,
session=session,
note=f"scope refusal delivery failed: {type(exc).__name__}: {exc}",
run_seed=run_seed,
)
session = wake_result.session
if wake_result.ok:
_ack(messages)
Expand All @@ -1550,6 +1587,7 @@ def _resume(message: Message) -> AttemptResult | None:
baseline=baseline,
candidate=candidate,
session=session,
tree_rejected=tree_rejected,
note=wake_result.error or session.error_detail or session.stop_reason,
run_seed=run_seed,
panel_transcript="\n\n".join(panel_sections),
Expand All @@ -1575,18 +1613,95 @@ def _not_run_note(request: SyscallRequest | None) -> str:
evals_charge = 0.0 # GPU-hours this pass took for gate evals
presealed = "" # a seal taken early to compare against the judged tree
while launcher is not None or on_replies is not None:
read_error = ""
try:
request = _consume_request()
except SyscallError as exc:
# loud, never silent: the author meant something by the file
# Scope admission also covers unreadable requests.
read_error = f"unhonorable syscall request: {exc}"
request = SyscallRequest(launches=(), problem=read_error)
if request is None:
break
# Refresh ancestry before deriving the paths admitted below.
no_backend = (
"sleep is not available here: this run has no compute backend for "
"launches; end your leg instead"
if launcher is None and not (on_stop and request.submit)
else ""
)
preflight = (
submit_preflight()
if request.sleep
and not request.problem
and request.submit
and not no_backend
and submit_preflight is not None
else SubmitPreflight("ready")
)
stale_submit = preflight.status in ("stale", "outdated-pin")
# A valid end abandons the tree; it does not request admission.
# Otherwise scope takes precedence over syscall/budget problems.
if not (request.end and not request.problem):
# Admission precedes every seal, dispatch and budget charge, including
# a stale submit's jobless checkpoint. Keep the measurement backstop.
measured = tuple(changed_paths())
violations = scope_validator(list(measured), contract)
if violations:
where = (
"checkpoint" if stale_submit else "submit" if request.submit else "launch"
)
# Preserve scope entries as written, bounded by count and length.
allowed = ", ".join(contract.scope.allowed[:10])
if len(contract.scope.allowed) > 10:
allowed += ", …"
if len(allowed) > 1000:
allowed = allowed[:997] + "…"
prefix = "Refused again:" if scope_refused else "Refused:"
problem = (
f"{prefix} this request at {where} changes paths the contract "
"does not allow authors to change: "
f"{', '.join(sorted(violations)[:10])}. "
"Nothing ran and nothing was charged. "
f"Authors may change: {allowed}."
)
scope_refused = True
tree_rejected = True
presealed = ""
message = Message(
0,
"note",
"kernel",
inbox_thread,
time.time(),
f"refusal:{session.session_id}:{inbox_seq}",
{"text": problem},
origin=inbox_dir.name,
)
if not _can_resume():
append(inbox_dir, message)
_write_messages()
return AttemptResult(
outcome="session-error",
tree_rejected=True,
baseline=baseline,
session=session,
note="Session cannot resume to receive the scope refusal.",
run_seed=run_seed,
panel_transcript="\n\n".join(panel_sections),
panel_rounds=panel_reads,
)
failed = _resume(message, allow_checkpoint=False)
if failed is not None:
return failed
continue
if read_error:
return AttemptResult(
outcome="session-error",
baseline=baseline,
session=session,
note=f"unhonorable syscall request: {exc}",
note=read_error,
tree_rejected=tree_rejected,
)
if request is None:
break
if request.withdraw and not request.problem:
problem = (
on_withdraw(request.withdraw)
Expand Down Expand Up @@ -1627,28 +1742,17 @@ def _not_run_note(request: SyscallRequest | None) -> str:
if request.report and on_replies is not None:
on_replies(({"to": "thread", "text": request.report, "reply_to": None},))
session = dc_replace(session, final_text=request.report)
return on_stop(session)
return dc_replace(on_stop(session), tree_rejected=tree_rejected)
return AttemptResult(
outcome="no-improvement",
tree_rejected=tree_rejected,
session=session,
note=(failed_gate[2].note or failed_gate[2].outcome)
if failed_gate
else "ended without a submit",
)
if not request.sleep:
break
no_backend = (
"sleep is not available here: this run has no compute backend for "
"launches; end your leg instead"
if launcher is None and not (on_stop and request.submit)
else ""
)
preflight = (
submit_preflight()
if request.submit and not no_backend and submit_preflight is not None
else SubmitPreflight("ready")
)
stale_submit = preflight.status in ("stale", "outdated-pin")
if stale_submit:
# Budget only the effective checkpoint, never the rejected compute.
request = SyscallRequest(launches=())
Expand Down Expand Up @@ -1694,18 +1798,6 @@ def _not_run_note(request: SyscallRequest | None) -> str:
)
if not problem:
Comment thread
renmengye marked this conversation as resolved.
if stale_submit:
violations = scope_validator(list(changed_paths()), contract)
if violations:
return AttemptResult(
outcome="scope-violation",
baseline=baseline,
session=session,
note=(
"out-of-scope paths at checkpoint: "
+ ", ".join(sorted(violations)[:10])
),
run_seed=run_seed,
)
sha = snapshot()
sleeps_used += 1
if preflight.status == "stale":
Expand Down Expand Up @@ -1778,6 +1870,7 @@ def _not_run_note(request: SyscallRequest | None) -> str:
# the walltime THIS submit declares (else the contract's):
# a resubmit without a declaration reverts to the default,
# never inheriting a prior park's.
tree_rejected = False
submitted = request
sleeps_used += 1
evals_charge = evals_gpu_hours(
Expand All @@ -1793,22 +1886,6 @@ def _not_run_note(request: SyscallRequest | None) -> str:
break
# a launch park: its launches are dispatched right below, so
# they are charged now
# Scope BEFORE the snapshot, same invariant as the candidate
# path below: an out-of-scope tree is never snapshotted OR
# executed — the out-of-scope edit could be to the ruler
# itself, and a launch runs code from this tree in an external
# job. Same ending as the candidate path.
violations = scope_validator(list(changed_paths()), contract)
if violations:
return AttemptResult(
outcome="scope-violation",
baseline=baseline,
session=session,
note=(
f"out-of-scope paths at launch: {', '.join(sorted(violations)[:10])}"
),
run_seed=run_seed,
)
sha = snapshot()
assert launcher is not None
try:
Expand Down Expand Up @@ -1868,12 +1945,20 @@ def _not_run_note(request: SyscallRequest | None) -> str:
on_meter(launches_used, sleeps_used, gpu_hours_used)
if submitted is None:
if on_stop is not None:
return on_stop(session)
return dc_replace(on_stop(session), tree_rejected=tree_rejected)
if tree_rejected:
return AttemptResult(
outcome="no-improvement",
session=session,
tree_rejected=True,
note="ended without a submit",
)
if launcher is not None and getattr(harness, "supports_resume", True):
return AttemptResult(
outcome="no-improvement", session=session, note="ended without a submit"
)
measured = tuple(changed_paths())
if submitted is None:
measured = tuple(changed_paths())
# Scope BEFORE the snapshot: an out-of-scope tree is never snapshotted
# OR measured — the out-of-scope edit could be to the ruler itself. This
# early exit keeps the snapshot off a rejected tree; measure_and_decide
Expand All @@ -1883,6 +1968,7 @@ def _not_run_note(request: SyscallRequest | None) -> str:
if violations:
return AttemptResult(
outcome="scope-violation",
tree_rejected=True,
baseline=baseline,
session=session,
note=f"out-of-scope paths: {', '.join(sorted(violations)[:10])}",
Expand Down
Loading
Loading