A runner's remaining subscription usage is not part of eligibility — an exhausted machine still takes dispatches, and a reset time is a schedulable fact nothing can express

[mkreyman]

A runner's remaining subscription usage is not part of eligibility, so a machine that cannot start a session for two days is still handed dispatches — and a fleet with no capacity now but capacity at 10:00 tomorrow looks identical to a fleet with none at all.

Raised by Mark, 2026-09-15.

What is wrong today

Runners.accepts?/5 decides eligibility on: connected, not draining, repository declared, kind declared, and in_flight < max_sessions. Every session runs on the machine's claude.ai subscription — the runner unsets ANTHROPIC_API_KEY and refuses to start unless its launcher strips it — so the scarce resource is the subscription WINDOW, and nothing in that list can see it.

Two consequences, and the second is the one worth building for:

1. A runner out of quota is still "available". Socket up, not draining, slots free — so available_runner/2 picks it and the dispatch goes out. The session then fails immediately or stalls, the ledger row burns a slot until Capacity.heal/3 releases it, and the story goes back to the queue to be handed to the same machine a minute later.

2. "No capacity now" and "no capacity until 10:00 tomorrow" are the same answer. DispatchDriver and TriageDispatcher both answer :no_runner and retry every minute. With a reset time they could report when capacity returns, which is a fact an operator can act on and a scheduler could eventually use. Today a fleet that will be fine in the morning is indistinguishable from one that is broken.

The open question, which decides the shape

Does the CLI emit a reset time when a session exhausts the limit? claude --help lists no usage subcommand, so there is no documented surface to poll; the failure itself is the honest source. Asked of the loopctl-runner maintaining session, since only that side can observe it:

• what the runner sees on exhaustion — exit code, stderr, a structured error on the stream-json output — and whether it carries a reset timestamp;
• whether that is distinguishable from a transient 429, which wants the opposite handling (a five-hour window excludes the machine for hours; a rate limit is a retry);
• whether anything under ~/.claude holds it readably, asked only to know if a poll is possible — building on an undocumented file is not the preferred route.

Proposed shape, if a reset time is available

• The runner records the reset it was GIVEN, never one it computed, and reports exhausted_until on its next status.
• RunnerStatus gains that optional field. Additive, and silence MUST mean available — the same asymmetry as Kinds.implied_by_silence/0. A control plane that excluded quiet runners would idle the whole fleet on a field nobody had shipped yet.
• Runners.accepts?/5 excludes a runner whose exhausted_until is in the future, so the rule lives in the ONE reader every dispatcher already consults rather than being re-implemented in each.
• The flag clears when a session succeeds, so a wrong or stale reset self-corrects rather than stranding a machine.
• :no_runner gains the earliest reset across the eligible-but-exhausted set, so the driver's log and the operator can see when the queue will move.

What this is NOT

Not a scheduler. "Defer until T" is the whole of it; a calendar, a priority queue across machines, or holding work for a machine that might reset sooner are all out of scope and should be argued separately if ever wanted.

If the CLI emits no reset time, the fallback is weaker but still worth having: derive exhaustion from repeated immediate failures on one machine and exclude it for a fixed backoff, with no claim about when it returns.

Acceptance

• A runner reporting exhausted_until in the future is not selected by available_runner/2 or by the driver, proved by a test.
• A runner that reports NOTHING is selected exactly as today, proved by a test — this is the clause that must not regress.
• :no_runner carries the earliest known reset when one is known, and nothing when none is.
• The contract's new field is additive with a version bump, and a runner at the previous version keeps working unchanged.

[mkreyman]

The KB already answers half the open question, and it changes the design's viability.

b38ef90e (incident, mac-mini 2026-09-07/08) records the exhaustion error verbatim:

API Error: 400 You have reached your specified API usage limits.
You will regain access on 2026-10-01 at 00:00 UTC.

So the CLI DOES surface a regain-access timestamp in the error text, in a parseable form. That was the fact this issue said would decide the shape, and it lands on the side that makes exhausted_until buildable.

Two caveats that matter for the discriminator this issue asks for:

1. That example is a METERED KEY's spend cap, not the subscription window. The article's own tell is the date four weeks out — "a subscription does not talk that way". The runner unsets ANTHROPIC_API_KEY and refuses a launcher that does not strip it, so a loopctl runner should never see this particular form; what it will see is the subscription's own window message, whose exact text is still the thing to observe rather than assume.
2. That gives the classifier a starting rule rather than a guess: a regain-access time DAYS or WEEKS out is an API spend cap and means the machine is misconfigured (an auth-source problem, per that article's rule 3), while a window measured in HOURS is the subscription resetting normally. Those want opposite handling — the first is an alert, the second is the deferral this issue is about.

Still owed by the runner side: the exact subscription-exhaustion text and whether it is distinguishable from a transient 429 on the stream-json output.

[mkreyman]

MEASURED, from the loopctl-runner session's own recording — this rewrites the design, and the open question is nearly closed.

The CLI emits a structured line on the stream-json stream, written to stdout.ndjson in every run directory as a matter of course:

{"type": "rate_limit_event", "rate_limit_info": {"status": "allowed_warning", "resetsAt": 1789488000,
 "rateLimitType": "seven_day", "utilization": 0.94, "isUsingOverage": false, "surpassedThreshold": 0.75,
 "unifiedWindows": {"five_hour": {"utilization": 0.04, "resetsAt": 1789410000},
                    "seven_day": {"utilization": 0.94, "resetsAt": 1789488000}}}}

Four things follow, and three of them make this issue's first draft wrong:

1. No text parsing, and no failure needed. The reset is resetsAt, unix epoch SECONDS, absolute, with no timezone to interpret — on a structured line, not in stderr. The issue's premise that the failure is the only honest source is FALSE.

2. It arrives as a WARNING while the session is still running. status: "allowed_warning" at utilization: 0.94 against surpassedThreshold: 0.75. So a machine can be known to be nearly out BEFORE anything fails, and control never has to learn it from a dead run. This is strictly better than what was proposed.

3. The two windows are named, so the discriminator needs no heuristic. unifiedWindows carries five_hour and seven_day separately, each with its own utilisation and reset, and rateLimitType names which one caused the event. My earlier comment proposed guessing the window from how far out the reset was; that is superseded — the window says its own name. (The hours-vs-weeks rule still holds for its OTHER purpose: a regain time weeks out on a metered key message means ANTHROPIC_API_KEY leaked into the environment, which is an alert about misconfiguration, not a scheduling fact.)

4. exhausted_until is the wrong field. The runner session's push-back, and it is right: a boolean-ish exclusion throws away the utilisation, which is what lets a dispatcher pick the LEAST LOADED runner rather than merely avoid a dead one. Carry the window name, its utilisation and its reset.

Corrected shape

• RunnerStatus gains an optional rate_limits object: per-window {utilization, resets_at} for each window the CLI reports, plus the governing rate_limit_type. Additive; silence still means available, unchanged and still the clause that must not regress.
• Runners.accepts?/5 excludes a runner whose governing window is exhausted — see the open question below for what "exhausted" is spelled as.
• available_runner/2 currently orders by in_flight ascending. With utilisation on the wire it can order by quota headroom too, so a fleet spreads load across windows instead of driving one machine to its limit while another idles.
• :no_runner carries the earliest resets_at across the excluded set, which is the original point of this issue and unchanged.

What is still open, and how to close it honestly

The status value when a session is actually BLOCKED is unknown. Every line in the sample says allowed_warning, which is self-evidently the pre-exhaustion state, so at least one other value exists and nobody has seen it. isUsingOverage: false implies an overage state as well. The runner session declined to invent the spelling, correctly, and declined to burn a window deliberately to find out — also correctly.

So the exclusion rule should key on utilization >= 1.0 FIRST, which is observable without knowing any status string, and treat the status field as corroborating rather than deciding. When a real block happens on any of the three runners the line will be in that run's stdout.ndjson, and the rule can be tightened against a recording instead of a guess.

[mkreyman]

RETRACTION — the duration rule I wrote two comments ago is FALSE, and it would have fired on a healthy machine.

I proposed: a regain time DAYS or WEEKS out is an API spend cap and means the machine is misconfigured; a window measured in HOURS is the subscription resetting normally. That is wrong, and it is refuted by the same recording, on a subscription login with no API key anywhere near it:

"unifiedWindows": {"five_hour": {"utilization": 0.04, "resetsAt": 1789410000},
                   "seven_day":  {"utilization": 0.94, "resetsAt": 1789488000}}

with the event's own rateLimitType reading seven_day.

A SUBSCRIPTION HAS A SEVEN-DAY WINDOW and resets it days out, routinely. A classifier built on my rule would have raised a leaked-key alert against a healthy machine sitting at 94% of its normal weekly quota — and it would have raised it at exactly the moment the fleet needed that machine excluded for CAPACITY, so the one alert produced would have been about the wrong thing. That is worse than no rule.

The discriminator is the SURFACE, not the duration (offered as a hypothesis to break, not as a finding):

• the subscription window arrives as a STRUCTURED rate_limit_event line on stream-json, with named windows, ahead of any failure;
• the metered spend cap in b38ef90e arrives as API Error: 400 PROSE about regaining access, and it ends the session.

If that holds, the runner never parses prose for the subscription case at all — and any API Error: 400 ... usage limits it does see means an API key reached a session that the runner's own containment (it unsets ANTHROPIC_API_KEY and refuses a launcher that does not strip it) says cannot happen. That makes it an auth-source alert exactly as b38ef90e's rule 3 states, and a strong one, because it is evidence the containment FAILED rather than merely that a key exists.

Two consequences for the design above:

1. The classifier is: structured event -> scheduling fact; prose 400 -> containment alert. No duration arithmetic anywhere.
2. A runner at 0.94 is not excluded and should not be preferred either. Exclusion and preference are separate questions, which is the whole argument for carrying utilisation rather than a boolean.

Credit where it belongs: this came from the loopctl-runner session measuring its own recording and telling me my rule was false. I asked it to let its observation win over my hypothesis and it did.