Skip to content

Per-target GPU lanes from the deployment's settings - #433

Merged
renmengye merged 6 commits into
mainfrom
feat/gpu-lanes
Sep 30, 2026
Merged

renmengye merged 6 commits into
mainfrom
feat/gpu-lanes

Conversation

@renmengye

@renmengye renmengye commented Sep 26, 2026 •

Copy link
Copy Markdown
Member

Today every GPU job the kernel submits goes to one fleet-wide lane (OUTERLOOP_GPU_PARTITION / OUTERLOOP_GPU_ACCOUNT) with an untyped --gpus-per-node. A deployment that has a dedicated partition for one target, needing its own account, a typed GPU request or extra scheduler flags, has no way to route that target's jobs there while other targets keep the fleet lane.

What changes

  • One optional deployment setting, OUTERLOOP_GPU_LANES: JSON mapping a target to a lane. It is validated once at startup; a mistake is one clear error naming the setting.
    OUTERLOOP_GPU_LANES='{"owner/repo":{"partition":"gpu-large","account":"my-account","gpu_type":"a100","extra":["--comment=reserved"]}}'
    
  • Every GPU job for a laned target uses it: gate evals, author launches and sweep arrays, and wake jobs (through the target on the record). CPU jobs, budgets, max_concurrent_gpus, nice and the queue view are unchanged.
  • A lane with a GPU type requests --gres=gpu:<type>:N; without one, jobs keep --gpus-per-node=N.
  • Extra flags must be --name=value and may not repeat any flag the kernel sets (account, partition, gres, gpus*, cpus*, mem*, time, qos, nice, array, dependency, begin, job-name, output, error, wrap, parsable, chdir): sbatch lets the later flag win, so a repeat would silently override the kernel's value.
  • Placement stays in the deployment, not the target contract: partition names are facts about a cluster, and the same contract has to work on another one.

Tests

31 new cases: parsing, valid and invalid; forbidden flags; placement per target, including resume; eval and author-array submissions for a laned target; typed gres argv; one startup error for bad JSON; CPU jobs ignore the lane; tick preflight accepts a lane without a fleet GPU partition; byte-identical argv for targets with no lane. Full suite 2180 passed.

Compatibility

No persisted state changes. The lane config is kept out of the persisted wake recipe; wake jobs read it from the environment. The setting is optional. Upgrading: no action needed.

🤖 Generated with Claude Code

OUTERLOOP_GPU_LANES maps a target to a partition, account, GPU type and
extra sbatch flags, so one repo's GPU jobs (evals, author launches,
sweeps) can go to a dedicated node while others keep the fleet lane. A
typed lane requests --gres=gpu:<type>:N. Extra flags may not repeat any
flag the kernel sets, since sbatch lets the later one win.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round 1 — reviewed head fd217dd8 — reviewer summarizer:hermes/gpt-5.6-terra over coverage+credentials+deployment+general+lifecycle+prose.

terra
Advisory findings from outerloop — the code owner decides. Reply to disagree; the outerloop:no-review label opts this PR out.

Verdict: nothing blocking — 1 advisory note.

Advisory (non-blocking):

  • Suggestion: The GPU request documentation omits typed lanes. [prose] GPU jobs request GPUs per node: typed lanes use --gres=gpu:<type>:N; other lanes use --gpus-per-node=N. (docs/install.md:497; high confidence)

One advisory documentation finding: the install guide omits the typed GPU request syntax. Rejected findings: none.

renmengye and others added 3 commits September 26, 2026 15:57
…es the code

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@renmengye

Copy link
Copy Markdown
Member Author

Advisory batched in 8ee5e07: the install guide now says typed lanes request GPUs with --gres=gpu:<type>:N and other lanes with --gpus-per-node=N.

@renmengye renmengye added the outerloop:review re-request the advisory review label Sep 26, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round 1 — reviewed head 8ee5e076 — reviewer hermes/gpt-5.6-terra.

terra
Advisory findings from outerloop — the code owner decides. Reply to disagree; the outerloop:no-review label opts this PR out.

Verdict: no defects found.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@renmengye renmengye added the autoresearch:review Request a fresh advisory review of this PR's current state label Sep 30, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round 2 — reviewed head dc6e3f30 — reviewer hermes/gpt-5.6-terra.

terra
Advisory findings from outerloop — the code owner decides. Reply to disagree; the outerloop:no-review label opts this PR out.

Verdict: 1 blocking, 0 advisory.

1 finding attached to the lines below.

One blocking defect: lane extras can increase nodes and bypass GPU ceilings.

Comment thread src/outerloop/gpu_lanes.py Outdated
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@renmengye renmengye added autoresearch:review Request a fresh advisory review of this PR's current state and removed autoresearch:review Request a fresh advisory review of this PR's current state labels Sep 30, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round 3 — reviewed head 86b028f5 — reviewer hermes/gpt-5.6-terra.

terra
Advisory findings from outerloop — the code owner decides. Reply to disagree; the outerloop:no-review label opts this PR out.

Verdict: no defects found.

@renmengye
renmengye merged commit e727c51 into main Sep 30, 2026
5 checks passed
@renmengye
renmengye deleted the feat/gpu-lanes branch September 30, 2026 14:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

autoresearch:review Request a fresh advisory review of this PR's current state outerloop:review re-request the advisory review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant