Skip to content

A capacity refusal parks the run instead of ending it - #453

Merged
renmengye merged 3 commits into
mainfrom
fix/capacity-refusal-parks
Oct 1, 2026
Merged

renmengye merged 3 commits into
mainfrom
fix/capacity-refusal-parks

Conversation

@renmengye

Copy link
Copy Markdown
Member

A run ended because its launch was refused for capacity. Live case: a parked run woke and its author asked for a 2-GPU sweep, which the operator GPU limit refused (the target's 4 GPUs were held by other slots). The kernel resumed the session once with the refusal. The author re-staged the sweep at 1 GPU and slept; that was refused too, so the request, including the sleep, was dropped ("measuring as-is"). The leg ended "ended without a submit" and the run ended as negative-result. The author did nothing wrong, and the project's rule (#442) is that admission refusals never end a run.

  • When a launch is refused with a CapacityError and the kernel will not resume the session again (after the in-session refusal, or when it cannot resume), the run now parks in the existing capacity wait (RunParked(..., capacity_wait=True)). It keeps its session, sealed candidate and meters, and nothing is charged.
  • The next wake resumes the same session with the refusal as a kernel note ("The run was parked waiting for capacity. Capacity may now be free; you can retry the launch.").
  • Capacity-wait wakes keep their existing deadline and wake-counter handling, so repeated waits never reach the stuck threshold.
  • Other refusals (budget, malformed, scope) and the stale-submit outcome are unchanged; the stale-submit check still comes first.
  • Docs: lifecycle; CHANGELOG under [Unreleased].

Compatibility: no new persisted state. capacity_wait already exists in the stage and the inbox note uses the existing message format, so no backfill or fixture is needed; a rolled-back kernel handles a capacity-wait park as it does today.

Review: built by Codex; I reviewed it and moved the capacity park after the stale-submit check so that case keeps its outcome. Tests: two capacity refusals in one leg park with the same session and meters, and the next wake delivers the note; repeated budget refusals keep the existing ending; capacity-wait wakes don't count toward stuck. Gate: pytest, ruff check, ruff format --check, mypy.

A launch refused for capacity after the in-session retry now parks the
run in the capacity wait, keeping its session, candidate and meters; the
next wake resumes the session with the refusal and a note that capacity
may now be free. Other refusals and stale submits are unchanged.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round 1 — reviewed head 9c3fbd9b — reviewer summarizer:hermes/gpt-5.6-terra over coverage+credentials+deployment+general+lifecycle+prose.

terra
Advisory findings from outerloop — the code owner decides. Reply to disagree; the outerloop:no-review label opts this PR out.

Verdict: 1 blocking, 0 advisory.

Capacity waits from non-resumable authors end on their first wake. [coverage+lifecycle+deployment] The capacity-refusal path parks when _can_resume() is false, but the scheduled wake rejects the same harness when supports_resume is false and ends the run as session-error instead of retrying capacity. (src/outerloop/attempt.py:1583; high confidence)

Merged one blocking finding: coverage, lifecycle, and deployment agree that a capacity refusal from a non-resumable author is parked but becomes session-error at its first wake. Rejected findings: none; the three reports were duplicates of the same issue, merged with combined lens attribution.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round 1 — reviewed head e1ab3dfb — reviewer hermes/gpt-5.6-terra.

terra
Advisory findings from outerloop — the code owner decides. Reply to disagree; the outerloop:no-review label opts this PR out.

Verdict: 1 blocking, 0 advisory.

1 finding attached to the lines below.

Capacity refusals can still end runs for authors without session resume support.

Comment thread src/outerloop/orchestrator.py Outdated

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round 2 — reviewed head 8567a2e5 — reviewer hermes/gpt-5.6-terra.

terra
Advisory findings from outerloop — the code owner decides. Reply to disagree; the outerloop:no-review label opts this PR out.

Verdict: 1 blocking, 0 advisory.

1 finding attached to the lines below.

Non-resumable authors cannot use the new capacity-wait flow.

Comment thread src/outerloop/attempt.py

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round 3 (re-run on the same head) — reviewed head 8567a2e5 — reviewer hermes/gpt-5.6-terra.

terra
Advisory findings from outerloop — the code owner decides. Reply to disagree; the outerloop:no-review label opts this PR out.

Verdict: no defects found.

@renmengye
renmengye merged commit d4ad752 into main Oct 1, 2026
9 checks passed
@renmengye
renmengye deleted the fix/capacity-refusal-parks branch October 1, 2026 16:07
@renmengye

Copy link
Copy Markdown
Member Author

Compatibility statement (added for the 0.3.0rc1 release audit, per RELEASING.md).

  • Surfaces: capacity-wait stage fields written on a launch capacity refusal; the durable refusal message and its deduplication key; how a wake interprets a capacity park.
  • Oldest state read: v0.2.1 run records and inboxes. A record without the capacity-wait flag keeps the previous behavior; existing message formats and keys remain accepted.
  • First tick after upgrade: in-flight runs are unchanged; the next capacity refusal parks the run (every author, resumable or not) instead of ending it. Ended runs and open PRs are untouched.
  • Rollback: an older kernel reads the records but cannot resume a capacity park it does not know; finish or end parked runs before rolling back.
  • Fixtures: legacy-park first pass, idempotent second pass and interrupted resume are added on the release branch for 0.3.0rc1.

@renmengye renmengye mentioned this pull request Oct 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant