Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,21 @@ Versions follow [SemVer](https://semver.org).

## [Unreleased]

- Deployment author overrides select a backend/model per target or agent slot,
bind it to each run, and leave panel and CI reviewer inheritance on the fleet
author. Per-run board details identify the effective author and overrides.
- Endpoint profiles accept an absolute `URL_FILE` instead of `URL`, reading bare
URLs or JSON addresses on every session. Missing files, failed bounded health
checks, and interrupted checks defer sessions without spending wake retries.
Init provisions fleet and override backends before validating prerequisites.
- Upgrading: no action or backfill needed; records without `author_overridden`
retain their existing author and panel behavior, including ended runs. New
overridden records reuse the saved author route and add this optional flag.
Rollback reads the records but loses fleet-only panel inheritance for overrides;
finish overridden runs and fresh endpoint capacity parks before rolling back.
Fresh endpoint deferrals reuse author-sleep capacity parks without a session ID;
older kernels cannot resume those parks.

- Launch ledger submissions require dispatched launch job IDs, preventing stale
checkpoints from attributing discarded launches to a commit. Gate capacity
waits still record any dispatched sibling launches. Existing ledger rows and
Expand Down
23 changes: 23 additions & 0 deletions docs/endpoints.md
Original file line number Diff line number Diff line change
Expand Up @@ -163,3 +163,26 @@ fallback. That requires the pinned clients under network observation.

Legacy records with no model use native model defaults only. A missing native
Codex model fails preflight; it never adopts an endpoint. No backfill is needed.

## Server addresses that change

Instead of `OUTERLOOP_ENDPOINT_ONPREM_URL`, set
`OUTERLOOP_ENDPOINT_ONPREM_URL_FILE=/absolute/path/server-address`.
Exactly one of `URL` and `URL_FILE` must be configured for a profile. The file
contains either a bare URL (a trailing newline is fine) or a JSON object:

```json
{"url": "http://localhost:8000/v1"}
```

Both forms use the same HTTP(S) base URL validation as `URL`: no credentials,
query string or fragment. Publish updates by atomic rename. The kernel reads
this file outside the session sandbox each time a session starts, including
resumes and wakes; an already constructed harness does not cache its value.

A missing or unreadable address defers intake and parked wakes without spending
wake retries. At session time the file is read once, then a single models-list
request checks the server with a three-second timeout and the profile key in an
Authorization header. Missing files, dead servers, and interrupted checks park
fresh runs and defer wakes without consuming a wake retry. There is no polling. Malformed contents are configuration errors. Profile names, served model,
API capabilities and credential paths retain their existing semantics.
25 changes: 25 additions & 0 deletions docs/install.md
Original file line number Diff line number Diff line change
Expand Up @@ -541,6 +541,31 @@ default, and a lens that names no model runs the author's model when it
shares the author's backend, and must name an explicit model on any other
backend); the author backend is
`OUTERLOOP_AUTHOR_BACKEND`/`OUTERLOOP_AUTHOR_MODEL`.

`OUTERLOOP_AUTHOR_OVERRIDES` optionally selects an author for individual targets
and agent slots, without changing judges or other targets:

```sh
OUTERLOOP_AUTHOR_OVERRIDES='{"owner/repo":{"backend":"claude","model":"served-model[endpoint=onprem]","slots":["agent-05"]}}'
```

Each entry requires `backend` (`claude`, `codex`, or `hermes`) and `model`.
Omit `slots` to cover every author slot on that target; otherwise use the existing
`agent-01`, `agent-02`, … identities allocated by the contract's authors-abreast
width. This is deployment configuration, not a contract setting. The setting is
parsed and validated at startup. Endpoint overrides select their own profile in
`model`; they do not inherit `OUTERLOOP_AUTHOR_ENDPOINT`. Native overrides use
the selected backend's author credential. Normal author/judge credential
separation still applies to the effective override credential.

Queued climbs bind the selection when submitted (before an intake claim is
launched); direct climbs bind at startup. Backend, model and key path are saved
in the run record. Changing overrides cannot switch an existing run, even at a
resume or wake. Panel inheritance and CI reviewers still use the fleet author,
exactly as for a run without an override. Per-run board details show the author
backend/model and mark overrides. Remove the setting (or use `{}`) to stop
selecting overrides for new work.

`OUTERLOOP_CLAUDE_MODEL` names the model for every Claude role (author,
panel judges, steward) when no explicit or inherited model covers that role:
there is no built-in default, and `start` refuses when a role needs it, naming
Expand Down
5 changes: 3 additions & 2 deletions scripts/tick_deploy.sh
Original file line number Diff line number Diff line change
Expand Up @@ -139,13 +139,14 @@ if [ -n "$ENV_TRUSTED" ]; then
env_line
env_value
export "$_k=$_v"
done < <(sed -nE 's/^(OUTERLOOP_ENDPOINT_[A-Z][A-Z0-9_]*_(URL|KEY_FILE|MODEL|API))=.*/\1/p' "$ENV_FILE" | sort -u)
done < <(sed -nE 's/^(OUTERLOOP_ENDPOINT_[A-Z][A-Z0-9_]*_(URL|URL_FILE|KEY_FILE|MODEL|API))=.*/\1/p' "$ENV_FILE" | sort -u)
for _k in OUTERLOOP_CLAUDE_VERSION OUTERLOOP_CODEX_VERSION \
OUTERLOOP_CLAUDE_SHA256 OUTERLOOP_CODEX_SHA256 \
OUTERLOOP_HERMES_REF OUTERLOOP_HERMES_SHA \
OUTERLOOP_CACHE_ROOT REVIEW_BACKEND \
OUTERLOOP_AUTHOR_ENDPOINT REVIEW_ENDPOINT REVIEW_MODEL \
OUTERLOOP_AUTHOR_BACKEND OUTERLOOP_AUTHOR_MODEL OUTERLOOP_CLAUDE_MODEL \
OUTERLOOP_AUTHOR_BACKEND OUTERLOOP_AUTHOR_MODEL OUTERLOOP_AUTHOR_OVERRIDES \
OUTERLOOP_CLAUDE_MODEL \
OUTERLOOP_CLAUDE_BIN OUTERLOOP_CODEX_BIN OUTERLOOP_CODEX_KEY_FILE \
OUTERLOOP_HERMES_KEY_FILE OUTERLOOP_HERMES_RESUME_MAX_CHARS \
OUTERLOOP_CLAUDE_KEY_FILE OUTERLOOP_STEWARD_KEY_FILE \
Expand Down
106 changes: 99 additions & 7 deletions src/outerloop/attempt.py
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,13 @@
should_dispatch,
snapshot_tree,
)
from outerloop.endpoints import author_model_setting, model_key, resolve_endpoint, split_endpoint
from outerloop.endpoints import (
EndpointUnavailable,
author_model_setting,
model_key,
resolve_endpoint,
split_endpoint,
)
from outerloop.evalcache import seed_dir
from outerloop.github import (
GitError,
Expand Down Expand Up @@ -273,6 +279,27 @@ def fleet_author_model(backend: str) -> str:
return model


def defer_endpoint_wake(root: Path, record: RunRecord) -> bool:
"""Keep the existing park and refund this delivery when its server is down."""
_, profile = resolve_endpoint(record.author_model, record.author_backend or "claude")
try:
if profile:
_ = profile.url
except EndpointUnavailable as exc:
_defer_endpoint(root, record, exc)
return True
return False


def _defer_endpoint(root: Path, record: RunRecord, exc: EndpointUnavailable) -> None:
save_record(
root,
dc_replace(record, state=PARKED, wake_attempts=max(0, record.wake_attempts - 1)),
time.time(),
)
log.warning("run %s: %s", record.run_id, exc)


def resume_author(
record: object,
fleet_model: str,
Expand Down Expand Up @@ -1328,6 +1355,8 @@ def post_leg_replies(messages: tuple[dict, ...]) -> None:

kwargs.setdefault("scope_validator", steward_out_of_scope)
kwargs.setdefault("ruler", RULER)
if not record.resume_session_id:
kwargs.setdefault("task_hypothesis", str(record.stage.get("hypothesis") or ""))
result = attempt_once(
config,
contract_text,
Expand Down Expand Up @@ -1502,14 +1531,15 @@ def _end(result: AttemptResult, drop_refs: list[str]) -> AttemptOutcome:
drop_snapshot(ws, Snapshot(commit="", tree="", ref=ref))
return outcome

# The wake NEEDS the author harness (it resumes the session). Fail as a
# A capacity park before the first session starts with a fresh brief.
# Other wakes NEED the author harness and saved session. Fail as a
# named ending, not a crash: the run cannot proceed and re-waking will not
# help without the harness, so leaving it PARKED would just hit the stuck
# cap slowly.
if (
harness is None
or spec is None
or not record.resume_session_id
or (not record.resume_session_id and not record.stage.get("capacity_wait"))
or not getattr(harness, "supports_resume", True)
):
return _end(
Expand Down Expand Up @@ -1740,6 +1770,10 @@ def changed_paths() -> list[str]:
if sleep_ref:
drop_snapshot(ws, Snapshot(commit="", tree="", ref=sleep_ref))
return AttemptOutcome(run_id=run_id, outcome="parked")
except EndpointUnavailable as exc:
latest = load_record(run_root, run_id)
_defer_endpoint(run_root, latest, exc)
return AttemptOutcome(run_id=run_id, outcome="parked", pr_url=latest.pr_url)
except ResumeContextBlocked as exc:
latest = load_record(run_root, run_id)
save_record(
Expand Down Expand Up @@ -3238,7 +3272,11 @@ def _panel_lenses_from_args(
author_backend = getattr(args, "author_backend", "") or "claude"
if author_model is None:
author_model = getattr(args, "model", "") or ""
parsed = resolve_lenses(args.panel, author_backend, author_model)
panel_backend, panel_model = author_backend, author_model
if getattr(args, "author_overridden", False):
panel_backend = os.environ.get("OUTERLOOP_AUTHOR_BACKEND") or "claude"
panel_model = fleet_author_model(panel_backend)
parsed = resolve_lenses(args.panel, panel_backend, panel_model)
author_credential = effective_author_credential(
author_backend, author_model, getattr(args, "key_file", "")
)
Expand Down Expand Up @@ -4264,6 +4302,7 @@ def live_attempt(
author_backend: str = "claude",
author_model: str = "",
author_key_file: str = "",
author_overridden: bool = False,
task_hypothesis: str = "",
spec: RoleSpec | None = None,
panel_lenses: tuple[PanelLens, ...] = (),
Expand Down Expand Up @@ -4293,8 +4332,10 @@ def live_attempt(
issue_number=issue_number,
author_backend=author_backend,
author_model=author_model,
author_overridden=author_overridden,
author_key_file=author_key_file,
run_job_id=_os.environ.get("SLURM_JOB_ID", ""),
stage={"hypothesis": redact(task_hypothesis, secrets)} if task_hypothesis else {},
)
try:
save_record(run_root, record, now)
Expand Down Expand Up @@ -4632,6 +4673,32 @@ def acknowledge(seq: int) -> None:
),
tree_of=lambda sha: ws.git("rev-parse", f"{sha}^{{tree}}").strip(),
)
except EndpointUnavailable as exc:
# Reuse a jobless capacity park; its first wake starts the author.
sha = snapshot()
kept_ref = snapshots[-1].ref
p = RunParked(
phase="author-sleep",
afterany="",
base_sha=pre_session_sha,
seed=0,
suite_seed=0,
candidate_sha=sha,
capacity_wait=True,
)
_park_run(
run_root,
record,
p,
kept_ref,
eval_minutes,
time.time(),
secrets,
base_branch=base_branch,
)
parked = p
log.warning("run %s: %s", run_id, exc)
return AttemptOutcome(run_id=run_id, outcome="parked")
except RunParked as p:
# The climb dispatched its measures and hibernated. Persist the
# re-entry stage as a PARKED record (not an error), keep the
Expand Down Expand Up @@ -5001,7 +5068,19 @@ def _run_id(value: str) -> str:
parser.add_argument(
"--hypothesis-b64", default="", help="base64 task hypothesis (issue text, fenced)"
)
parser.add_argument("--author-bound", action="store_true", help=argparse.SUPPRESS)
parser.add_argument("--author-overridden", action="store_true", help=argparse.SUPPRESS)
args = parser.parse_args()
from outerloop.author_overrides import select_override

try:
if not args.resume and not args.author_bound:
selected = select_override(args.target, args.agent_id)
if selected:
args.author_backend, args.model = selected.backend, selected.resolved_model()
args.author_overridden = True
except ValueError as exc:
parser.error(str(exc))
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(message)s")
if not args.model and not args.resume:
# resolved after parsing, never at parser build: a deployment without
Expand Down Expand Up @@ -5079,6 +5158,17 @@ def _run_id(value: str) -> str:
# reap (the resume_run finally below only runs once we reach it)
_release_own_lease(args.run_root, args.resume)
parser.error(f"parked run {args.resume}: {_err}")
try:
deferred = isinstance(_wake_record, RunRecord) and defer_endpoint_wake(
args.run_root, _wake_record
)
except ValueError as exc:
_release_own_lease(args.run_root, args.resume)
parser.error(f"parked run {args.resume}: {exc}")
if deferred:
_release_own_lease(args.run_root, args.resume)
return 0
args.author_overridden = bool(getattr(_wake_record, "author_overridden", False))
args.key_file = wake_key_file
# the wake runs the SAME verification panel as a fresh climb, so a
# dispatched improvement is not published unverified.
Expand Down Expand Up @@ -5182,10 +5272,11 @@ def _run_id(value: str) -> str:
if not (args.target and args.benchmark):
parser.error("--target and --benchmark are required for a fresh climb")

# a fresh climb authors on the FLEET's configured backend; validate it (codex
# writes+executes, so --image + a non-claude model) before any spend.
# Validate the selected author before spending; an override's endpoint
# selector is independent of the fleet endpoint.
try:
args.model = author_model_setting(args.author_backend, args.model)
if not args.author_overridden and not args.author_bound:
args.model = author_model_setting(args.author_backend, args.model)
except ValueError as exc:
parser.error(str(exc))
_err = author_config_error(args.author_backend, args.model, args.image)
Expand Down Expand Up @@ -5294,6 +5385,7 @@ def _run_id(value: str) -> str:
issue_number=args.issue,
author_backend=args.author_backend,
author_model=args.model,
author_overridden=args.author_overridden,
author_key_file=args.key_file,
task_hypothesis=(
__import__("base64").b64decode(args.hypothesis_b64).decode()
Expand Down
Loading
Loading