Live operator limits: tighten-only GPU ceiling and attempt width

[renmengye]

An operator who shares a cluster account with a fleet had two ways to slow it down: edit a target's contract (a reviewed PR in the target repo) or create HOLD_LAUNCHES, which blocks new runs but lets in-flight runs keep launching. Neither frees GPUs quickly, and a contract change is the wrong tool for a temporary need.

Change

• <state root>/limits.toml: [defaults] and [targets."owner/repo"], with max_gpus (a GPU ceiling for everything the fleet submits) and max_active_attempts. Re-read at every admission; no restart.
• Tighten-only: the effective value is the minimum of the contract and the operator's value. No operator value can loosen a contract budget. A malformed file blocks new GPU admissions and attempts and is reported in the log and by outerloop limits.
• Usage from the scheduler: one squeue --me --json snapshot per check. A job counts toward a target only when its name carries a run id that exists under <root>/runs/, so the operator's own jobs never count and requeued jobs count as the scheduler reports them. Handles the wrapped numeric fields of recent Slurm JSON as well as the older plain form.
• Enforced at every GPU-bearing submission path: author experiments and sweeps, evaluations and verification, fresh/steward/self-initiated attempts, and wake/session jobs. An author's over-ceiling launch is refused with a message and no launch charge; evaluations wait for capacity; a combined submit refuses only the over-cap launches. Control-plane tick jobs are CPU-only (asserted by a test).
• Two simultaneous admissions can overshoot the ceiling by at most one batch (documented).
• No limits file: admission is unchanged and no scheduler query is made.
• outerloop limits: effective limits and current fleet GPU usage per target (read-only).

The contract schema is unchanged.

Verified

• Full gate: 2218 passed, 2 skipped; ruff, format, mypy clean; mutation checks fail all 8 submission-path tests when admission is bypassed.
• Read-only against a live Slurm 25.05 cluster: the usage query parsed real output and attributed only the fleet's sweep jobs to their target, ignoring the operator's own GPU jobs.

🤖 Generated with Claude Code

[github-actions]

Round 1 — reviewed head 7aa87294 — reviewer summarizer:hermes/gpt-5.6-terra over coverage+credentials+deployment+general+lifecycle+prose.

terra
Advisory findings from outerloop — the code owner decides. Reply to disagree; the outerloop:no-review label opts this PR out.

Verdict: 4 blocking, 0 advisory.

3 findings attached to the lines below.

Intake submissions do not reserve an attempt slot. [coverage] After submit succeeds, this path returns without writing a pending marker, while its width check counts only existing run records, so another tick can submit a different intake issue before the first queued climb creates its record and exceed max_active_attempts. (src/outerloop/tick.py:3066; high confidence)

Blocking findings: intake admission can exceed max_active_attempts because queued submissions do not reserve capacity; launch, evaluation, and wake scheduler job names can exceed Slurm limits for valid long identifiers. Rejected findings: none; the four reported findings address distinct code locations and submission paths.

[github-actions]

Round 1 — reviewed head 02f794fb — reviewer hermes/gpt-5.6-terra.

terra
Advisory findings from outerloop — the code owner decides. Reply to disagree; the outerloop:no-review label opts this PR out.

Verdict: no defects found.

[github-actions]

Round 2 — reviewed head a097791f — reviewer hermes/gpt-5.6-terra.

terra
Advisory findings from outerloop — the code owner decides. Reply to disagree; the outerloop:no-review label opts this PR out.

Verdict: no defects found.

[renmengye]

Compatibility statement (added for the 0.3.0rc1 release audit, per RELEASING.md).

• Surfaces: the capacity-wait stage flag; scheduler job-name attribution and deduplication; intake pending markers.
• Oldest state read: v0.2.1 pending markers and records without the flag; job names from older submitters are attributed with documented tolerance.
• First tick after upgrade: in-flight jobs keep their names; new submissions use the new names; existing pending markers are honored.
• Rollback: older kernels read the markers but may not attribute new-format job names; drain pending intake first.
• Fixtures: a prior-release run and job layout across the changed surfaces is added on the release branch for 0.3.0rc1, alongside F15.