[decision] Demote AI budget circuit breakers to observe-only (BUDGET_ENFORCE, default off) — keep structural caps + meter; re-enable at paid launch

[serge-ivo]

Decision

Demote the AI budget circuit breakers to observe-only at this stage. Keep the structural caps and the usage meter; stop hard-blocking on dollar/token ceilings. Re-enable enforcement at paid launch.

Why

• BYOK means we can't get a real dollar figure. For BYOK users the AI spend is on their provider key, and cost_micros is list-price arithmetic (ai-pricing.ts), not what the provider actually charged (caching/discounts/batch move it). A hard stop on an estimate of someone else's bill guards the wrong thing with a number we can't trust. (Root of the [bug] The $50 circuit breaker is denominated in dollars for work nobody is billed for — subscription engine usage blocks delegations #343/[coordination] Autonomous work budget — bound spend across every unattended entry point #184 denomination pain.)
• The breaker has only ever done net harm. The 250M-token daily ceiling silently stopped 4 legit Coder agents overnight (they read as "idle" — same failure family as [bug] Agent reports a healthy run as "stalled" — step N/M reads as progress, liveness is only ever stated negatively, and the honesty rule covers false success but not false failure #459). There is no recorded case of it catching a real runaway. A guardrail that has hurt real work and never caught a real fault is negative value.
• The catastrophic failure mode is already covered structurally. Unbounded looping/recursion — the actual "print money" risk — is bounded by the structural caps (tool loop max 3 rounds; Loop MAX_ITERATIONS_CAP=50; delegation depth-4 / 50-fanout; apply single-flight). Those are deterministic, need no calibration, and don't false-positive. They stay.
• The values were never calibrated ($50, 250M, $5/tree, ×25, 50 delegations are round-number guesses; the acceptance criterion at delegation-budget.ts:37 — "target ~0 interrupted runs" — was never filled in). Rather than calibrate a control we don't need yet, turn it off.

What changed (this issue)

• New soft-launch switch BUDGET_ENFORCE (default off), mirroring isPaywallEnforced (billing.ts:185). The two daily account circuit breakers (chargedMicros and tokens) in reserveDelegation (delegation-budget-store.ts) only block when it is on; otherwise usage is metered but never stopped.
• Kept, untouched: the structural per-tree caps (delegations / depth in the atomic UPDATE) and the whole ai_usage meter + /v1/usage.
• Visible reminders that it's off (so we get nudged to turn it on):
  • Loud comment at the gate referencing this issue.
  • Commented-out BUDGET_ENFORCE in workers/api/wrangler.toml next to PAYWALL_ENFORCE.
  • A note on the console Usage page: "Budget limits are disabled (observe-only) — usage is metered but never blocked."
• Enforcement tests set BUDGET_ENFORCE:"1" so the breaker logic stays covered.

Re-enable trigger

Turn BUDGET_ENFORCE on when the subscription-pool economics go live (creators paid from platform Workers-AI/quota spend) — that is when unattributed token use becomes our cost and a real, calibrated breaker earns its place. By then the ai_usage distribution exists to calibrate it from. Until then: meter + structural caps + (later) an alert on payer='platform' spend, which is the only path that is real platform money.

Parked under this decision (deferred, not before external users)

• [frontend] Budget & limits settings surface — show 24h consumption vs each ceiling + distance-to-limit, and edit the limits (console owner + admin) #475 (Budget & limits settings UI) — when resumed, ship it disabled with a visible "enforcement off" note.
• [mcp] get/set AI budget limits via MCP — account ceilings, per-tree defaults, Loop iteration cap #476 (get/set budget limits via MCP).
• [enhancement] Per-run knobs configurable too — per-tree budget ($5/50/depth-4) and Loop max-iterations, via the same UI + MCP surfaces #477 (per-run knobs: per-tree budget + Loop max-iterations).
• The calibration work (measure defaults against ai_usage p99) — only needed once enforcement returns.

Reframes epic #478; the D1 config resolver from #474 stays in place (we'll want it when enforcement returns).

Acceptance criteria

• BUDGET_ENFORCE env switch exists, default off; account circuit breakers are no-ops when off.
• Structural caps and the ai_usage meter are unchanged.
• Console Usage page shows the "limits disabled (observe-only)" note.
• wrangler.toml carries the commented-out flag; gate comment references this issue.
• Tests: breaker-enforcement tests pass with BUDGET_ENFORCE:"1"; a new test asserts no block when off.

[serge-ivo]

This is already shipped and deployed — the open question is narrower than the issue

Every acceptance criterion in this issue is met on main and live in production. Verified:

Criterion	Evidence
BUDGET_ENFORCE exists, default off; breakers no-op when off	d951b5f. isBudgetEnforced at workers/api/src/lib/delegation-budget-store.ts:279-281; gated at :310 and :324
Structural caps + meter unchanged	The atomic UPDATE at delegation-budget-store.ts:335-347 still carries ?4 <= max_depth AND delegations_used < delegations_limit AND (cost_micros_limit - reserved - spent) >= ?3, ungated
Console Usage note	store/console/src/pages/Usage.tsx:237 and :652. Both strings are in the deployed bundle — curl https://proagentstore.online/console/ greps 1 hit each for Budget limits are off (observe-only). and Observe-only — not currently enforced.
wrangler.toml commented flag + gate comment cites this issue	workers/api/wrangler.toml:30-32; comment block at delegation-budget-store.ts:270-277
Tests both ways	delegation-budget-store.test.ts:295 (BUDGET_ENFORCE:"1"), :354 ("observe-only by default (#485)")

Live confirmation, GET /v1/budget/limits on production, 2026-08-11:

"effective": { "chargedMicrosCeiling": 50000000, "tokenCeiling": 250000000, ... all tiers "default" },
"consumption": { "chargedMicros": 7303714, "tokens": 221731036 },
"enforced": false

The "Parked" section of this issue is stale. It was written at 08:10 saying #475/#476/#477 were deferred; all three were closed delivered by 08:56 the same morning (2d8f9d5, ca476cd, c363fa9), plus #486 (5805332) and #484 (7f2eba5). So the plan diverged from the decision within the hour, in the direction of more budget machinery, not less. Nothing is parked.

So: close this as delivered. What is genuinely undecided is one thing the issue folded into a single switch — which breaker comes back on, and when. That is below.

---

The measured exposure

All from the live ledger, usage_summary range=30d + GET /v1/admin/spending, 2026-08-11.

What is actually money — $18.69 over 30 days.

Payer	30d	Share of notional
unknown (engine, credential never established)	$2,678.03	99.3%
byok-api (owner's Anthropic key — real money)	$18.68	0.7%
platform (our Workers AI — real platform cost)	$0.009	0.0003%

totals.chargedCostMicros = 18,685,197 µ$ = $18.69 for the month, mean $0.62/day. byKind.engine is 2,634,475,688 µ$ of the 2,696,716,383 µ$ notional total = 97.7% (the brief said ~98.7%; the measured figure is 97.7%). That $2,678 is list-price arithmetic on subscription engine turns, not a bill anyone received, and I am not presenting it as one.

Platform-paid AI is not a real exposure today. Two attributions, both trivial: payer='platform' = $0.009 / 220 calls (30d); provider='platform' in the admin view = $0.029 / 626 calls, flagged estimated:true with background ingest embeds not yet metered. So PLATFORM_AI_ENABLED = "true" (workers/api/wrangler.toml:22) currently costs on the order of three cents a month. topSpenders returns exactly one row (serge.the.dev@gmail.com) — there is no second account for it to subsidise.

Also worth recording, because the wrangler comment does not say it: the platform-AI switch no longer needs a deploy to flip. readPlatformAiSetting (lib/platform-settings.ts:82-110) resolves a D1 override above the env var (PUT /v1/admin/settings/platform-ai, migration 0081); production reports platformAi: {enabled:true, source:"env", override:null}. The stranger-spend threat has a one-click kill that is independent of the budget breakers.

The two breakers have opposite risk profiles, and this issue turned both off with one switch.

Breaker	24h consumption	Ceiling	Headroom
Charged dollars	$7.30	$50	6.8×
Tokens	221,731,036	250,000,000	1.13× — 88.7% consumed

That token figure is an ordinary working day with no runaway in progress. Enforcement on today would stop work within hours — which is exactly the reported incident behind #478 (four Coder agents idle overnight), reproduced as a live number rather than a memory.

Why the token ceiling misfires: it is 96% a cache-read ceiling. Over 30d the account's tokens are 470,718,474 cache-read out of 491,153,095 total = 95.8%. accountUsageSince (lib/usage.ts:644-645) sums input + output + cache_read + cache_write undifferentiated, so the cheapest token on the price sheet (cache read, a tenth of input) counts at parity with the most expensive. A ceiling denominated that way is not measuring spend or risk; it is measuring how much context the Coder re-read.

What enforcement ever covered — much less than it sounds

accountUsageSince has exactly one enforcement caller: reserve() at delegation-budget-store.ts:305. (connectors/supervision.ts:334,638 and routes/budget.ts:221,284 only read it to report.) reserve() in turn has exactly two callers: workflows/agent-loop.ts:88 and workflows/coding-session.ts:453 — and the coding one is if (!budgetId) return decideCodingAction(...) at :452, so it applies only to a Pilot started with a pool.

Never covered, enforced or not: plain /chat, /loop-decide (routes/instances-chat.ts:134-165 — no reserve, no budget), /explain, /overseer, pipeline ai_generate unless the pipeline declares delegate steps (lib/pipeline-run-start.ts:83), voice STT/TTS, translation. That is where the voice ($12.22 of 30d cost, 1,213 calls) and most chat spend lives.

And the per-tree dollar pool is blind to the dominant consumption by construction: instanceSpendMicros filters on CHARGED_SQL = "payer IN ('byok-api','platform')" (lib/usage-payer.ts:58), so subscription engine turns settle zero against the $5 pool. What actually bounds a runaway Coder tree is the delegation count — reserve() increments delegations_used per decide, limit 50 — not the money.

Two live gaps in the same class are filed separately as #502 (do not fix them here).

The collision with #184 — name it rather than let it surface later

#184 ("Autonomous work budget", closed 2026-08-04) promised, in its own words, a "hard stop rather than a soft warning" for BYOK spend, justified explicitly as user-safety (the platform is not exposed; the user is), and "admission check at every autonomous entry point (cron, webhook, delivery, delegation) and on retry/replay", at "the BYOK/Workers-AI call path that already writes ai_usage".

At HEAD neither holds: the check sits in reserve(), not at the ledger chokepoint, and it is now a soft warning. This issue is a deliberate reversal of a shipped promise. #343 gave a good reason to reverse part of it (a dollar ceiling that counted subscription tokens as dollars). It gave no reason to reverse the charged-dollar half, which is the half #184 was actually about.

Options

A — leave it exactly as shipped (both observe-only). Cost: nothing today. Risk: an unattended BYOK loop bills the owner's Anthropic card with only the per-tree $5 pool and the 50-delegation cap between it and the bill, and on the uncovered paths above, not even that. Defensible against the platform-cost threat (3¢/month, one user). Not defensible against the owner's-own-runaway threat, which is the one that is live.

B — re-enable both. Rejected on the measurement: 88.7% of the token ceiling consumed on an ordinary day. It would fire this week, on legitimate work, and the failure looks like "the agent went idle" (#459 family). Do not do this.

C — split them: enforce charged dollars, leave tokens observe-only. ← recommended.

The two isBudgetEnforced(env) calls are already separate lines (:310, :324), and the ceiling resolver already reads a per-account D1 row above the constant. So C needs no code:

1. PUT /v1/budget/limits {"tokenCeiling": 100000000000} (= MAX_TOKEN_CEILING, routes/budget.ts:33) — or set_budget_limits over MCP. The token breaker becomes unreachable in practice while still being metered and displayed.
2. Uncomment BUDGET_ENFORCE = "1" in workers/api/wrangler.toml:32 and deploy.

Result: the only quantity that is real money gets a hard stop with 6.8× headroom over the heaviest day measured; the quantity that is 96% cache reads stops interrupting work. #184's user-safety promise is kept in the denomination #343 established as the correct one.

Caveat to accept knowingly: step 1 sets a per-account row, and PUT is a full replace, not a patch — see #501. Send all six fields, or send only tokenCeiling while the other five are still inheriting default (which is their state today, so a single-field PUT is safe right now).

If the owner prefers the intent to live in code rather than in a data row, the durable form is two flags (BUDGET_ENFORCE_COST / BUDGET_ENFORCE_TOKENS) — roughly five lines at :279-281, :310, :324. That is a nice-to-have, not a prerequisite; the config path above works today and can be reverted with one API call.

D — recalibrate the token ceiling instead of disabling it. Rejected: there is no defensible number while the metric counts cache reads at parity with output. Fixing the metric (weight cache reads, or exclude payer='unknown' from the token sum) is a real option, but it is a design change with its own calibration problem, and it should not block C. Worth its own ticket if the token dimension is ever wanted back.

Recommendation

Close this issue as delivered, and take C as a config change: tokenCeiling to the max on the owner's row, BUDGET_ENFORCE = "1" uncommented. It is two actions, both reversible in under a minute, and it converts "no enforcement anywhere" into "the one denomination that is money is hard-stopped, 6.8× above the heaviest day yet observed".

What would change my mind

• A second user signs in. topSpenders has one row today. With strangers, PLATFORM_AI_ENABLED=true stops being 3¢ and the right control is the D1 kill switch plus a per-user platform-paid cap — the delegation breaker does not guard that path at all. Re-read this brief at that moment; its numbers are single-tenant numbers.
• A measured 24h charged figure above ~$25. Peak observed is $7.30. If normal work reaches half the ceiling, the $50 number needs raising before it is enforced, not after it fires.
• Loop-path spend stops being a minority. If reserve()-covered runs ever dominate the ledger, enforcement there becomes a real control rather than a partial one, and both halves deserve reconsidering together.
• Evidence of a real runaway that the token ceiling would have caught. I looked: list_errors over the last 40 entries shows no account_ceiling event, and the one runaway-shaped record is browser-task max_steps: stopped after 60 actions (2026-08-10) — caught by a structural cap, not a budget one. That is one data point for "structural caps are the ones that work", not proof.

Verified vs inferred

Verified by reading code at HEAD and by live calls: every file:line above, the deployed console strings, the production /v1/budget/limits payload, the 30d ledger aggregates, the caller sets for accountUsageSince / reserve() / openBudget (grepped, not assumed). Inferred: that the four-Coder-agents incident was this token ceiling — I take that from #478's background and did not reproduce it, though the 88.7% reading makes it entirely consistent. Not measured: what fraction of the $18.69 flowed through reserve(); the ledger has no run-id dimension to answer it.

[serge-ivo]

Correction: the incident is now verified, not inferred

My comment above listed as inferred: "that the four-Coder-agents incident was this token ceiling — I
take that from #478's background and did not reproduce it." I have since read the Coder Lead
transcript (MCP instance_messages on 5fab318d…) and it is direct evidence. Verbatim:

10 Aug 02:53 agent — "All 6 agents are idle — every run hit the 250M token circuit breaker and
stopped. FAS — stopped at iter 12/50 … FWS — iter 14/50 … FGS — iter 13/50 … Heartfull — iter 18/50"

10 Aug 07:53 — the owner has the Lead re-delegate all four.
10 Aug 07:54 agent — "All 4 agents are idle and failed — the token limit hit again immediately
(268M tokens, 250M cap). Every run stopped at iteration 0, meaning no work was done."

So it was six agents, not four, and it happened twice — the second time destroying four fresh
delegations at iteration 0. Both the "silently stopped legit overnight work" claim in the issue body
and my 88.7%-of-ceiling reading are corroborated by the run itself. 268M against a 250M cap is 107%;
the reading I took today was 88.7% of the same ceiling on a quieter day.

This strengthens the recommendation rather than changing it. Option C (enforce charged dollars,
leave tokens observe-only) is precisely the split that would have let this work continue: the same
period's charged spend never approached $50.

Two further things the transcript settles, both already filed elsewhere but worth recording here
because this is the decision page:

The "ask your platform operator" wording is not hypothetical. At 02:54 the owner asked "Which
token limit is it? … What can I change that limit?" and the agent answered:

"Raise the limit — contact ProAgentStore support to increase your account's token allowance.
There is no setting in your console that controls this."

That is delegation-budget-store.ts:321/331 reaching a real user and sending him to a support desk
that does not exist, for a control he owns. It was true when the breaker shipped and false since
#475. Filed as a remainder on #478.

#484's fix is confirmed by the same transcript. At 07:51 the Lead reported "token limit is clear,
every agent shows canWork: true", the owner re-delegated on that basis at 07:52, and at 07:54 all
four had already failed at iteration 0. #484 ("subordinate_status / canWork is blind to the account
budget") was closed at 08:51 the same morning — i.e. this transcript is the incident that produced
it, and the fix landed an hour after the pain. Nothing further needed; recorded so the lineage is not
re-derived later.