Skip to content

fix(engine): key chokepoint ledger paused off kill_switch eventType - #8965

Closed
galuis116 wants to merge 1 commit into
JSONbored:mainfrom
galuis116:fix/8864-budget-cap-kill-switch-ledger-paused
Closed

fix(engine): key chokepoint ledger paused off kill_switch eventType#8965
galuis116 wants to merge 1 commit into
JSONbored:mainfrom
galuis116:fix/8864-budget-cap-kill-switch-ledger-paused

Conversation

@galuis116

Copy link
Copy Markdown
Contributor

Summary

  • Key denyResult's ledger decision ternary off eventType === kill_switch instead of stage === kill_switch, so budget-cap termination (stage budget_cap, eventType kill_switch) records paused like the top-level kill-switch path.
  • Extend the existing termination-ceiling package test to assert ledgerEvent.decision === paused, and add a vitest suite that covers the kill-switch / soft-deny / termination branches for codecov patch coverage.

Closes #8864

Scope

  • The PR title follows type(scope): short summary Conventional Commit format, for example fix(api): restore profile access checks.
  • This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes.
  • This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
  • I linked a currently open issue this PR resolves (e.g. Closes #123) — a linked open issue is required for every contributor PR.

Validation

  • git diff --check
  • npm run actionlint
  • npm run typecheck
  • npm run test:coverage locally; codecov/patch requires ≥99% coverage of the lines AND branches you changed (aim for 100% on your diff so CI variance does not fail near the threshold). Global coverage is a non-blocking trend with a loose 90% backstop, not the gate.
  • npm run test:workers
  • npm run build:mcp
  • npm run test:mcp-pack
  • npm run ui:openapi:check
  • npm run ui:lint
  • npm run ui:typecheck
  • npm run ui:build
  • npm audit --audit-level=moderate
  • New or changed behavior has unit/integration tests for new branches, fallback paths, and sanitizer boundaries

If any required check was skipped, explain why:

  • Engine-only change: ran packages/loopover-engine build + chokepoint.test.js (27 pass) and vitest run test/unit/engine-governor-chokepoint-kill-switch-decision.test.ts (3 pass). Unrelated UI/MCP/workers/actionlint checks not exercised.

Safety

  • No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
  • Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
  • Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests.
  • API/OpenAPI/MCP behavior is updated and tested where needed.
  • UI changes use live API data or real empty/error/loading states, not production mock/demo fallbacks.
  • Visible UI changes include a UI Evidence section below with JPG/JPEG or PNG screenshots arranged as organized, captioned, clickable thumbnails. SVG screenshots are not used as review evidence. Review-only screenshots or recordings are not committed to the repository.
  • Public docs/changelogs are updated where needed; changelogs are only edited for release-prep PRs.

UI Evidence

N/A — engine-only governor ledger decision fix; no visible UI.

Notes

Budget-cap termination fires stage budget_cap with eventType kill_switch;
denyResult previously keyed decision off stage, so the ledger recorded
deny instead of paused. Key the ternary off eventType instead.

Closes #8864
@galuis116
galuis116 requested a review from JSONbored as a code owner July 26, 2026 14:27
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 26, 2026
@loopover-orb

loopover-orb Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Caution

🛑 LoopOver review result - fixes required

Review updated: 2026-07-26 15:07:46 UTC

3 files · 1 AI reviewer · no blockers · CI failing · unstable

🛑 Suggested Action - Fix Blockers

Review summary
This swaps the ledger-decision ternary in denyResult from keying off input.stage to input.eventType, fixing the real bug where budget-cap termination (stage `budget_cap`, eventType `kill_switch`) was ledgered as `deny` instead of `paused`. I traced the call site: evaluateGovernorCaps' termination verdict is passed as `eventType: budgetCap.verdict`, which is `kill_switch` for the hard wall-clock stop, so the old `stage === 'kill_switch'` check genuinely missed that case — this is the correct source-level fix, not a symptom patch. The new vitest suite exercises the exact three relevant eventType/stage combinations (budget-cap kill_switch, top-level kill_switch, budget-cap soft deny) against the real src file (needed since the existing node:test suite runs against dist/index.js, which codecov doesn't measure), and the existing chokepoint.test.ts assertion is extended consistently.

Nits — 3 non-blocking
  • packages/loopover-engine/src/governor/chokepoint.ts:139 — the nested ternary (`eventType === 'kill_switch' ? ... : eventType === 'throttled' ? ... : 'deny'`) is getting harder to read as more eventTypes accumulate; consider a small lookup map (`{kill_switch: 'paused', throttled: 'throttle'}[eventType] ?? 'deny'`) next time this needs a fourth branch.
  • test/unit/engine-governor-chokepoint-kill-switch-decision.test.ts and packages/loopover-engine/test/chokepoint.test.ts now assert near-identical budget-cap-termination/kill-switch-stage scenarios in two different test runners — acceptable given the dist-vs-src coverage-measurement split, but worth a one-line comment in the vitest file noting why the overlap exists so a future reader doesn't dedupe it away.
  • packages/loopover-engine/src/governor/chokepoint.ts:139 — a short lookup table for the decision mapping would make it easier to extend without deepening the ternary.

CI checks failing

  • validate
  • validate-tests

Decision drivers

  • ✅ Code review — No blockers (1 reviewer)
  • ✅ Gate result — Passing (No configured blocker found.)
Context & advisory signals — never blocks the verdict
Signal Result Evidence
Linked issue ✅ Linked #8864
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (1 linked issue).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 1856 registered-repo PR(s), 1210 merged, 54 issue(s).
Contributor context ✅ Confirmed Gittensor contributor galuis116; Gittensor profile; 1856 PR(s), 54 issue(s).
Improvement ✅ Minor risk: clean · value: minor
Linked issue satisfaction

Addressed
The diff changes denyResult's decision ternary to key off input.eventType === "kill_switch" instead of stage, exactly as required, and adds both a new test file and an extended assertion in test/chokepoint.test.ts verifying budget-cap termination now ledgers decision === "paused".

Review context
  • Author: galuis116
  • Role context: outside_contributor
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: JavaScript, Python, TypeScript, Dart, HTML, MDX, Rust, C++
  • Official Gittensor activity: 1856 PR(s), 54 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Start here: Triage stale or unlinked PRs.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.
🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

  • @loopover ask <question> answers contribution-quality Q&A with source citations and freshness.
  • @loopover chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
  • A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

  • Re-run LoopOver review

@loopover-orb

loopover-orb Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

LoopOver is closing this pull request on the maintainer's behalf (CI is failing (validate, validate-tests)). This is an automated maintenance action — to pursue this change, please open a new pull request with the issues resolved. Closed PRs may be analyzed later to improve review accuracy, but they are not automatically reopened or re-reviewed.

@loopover-orb loopover-orb Bot closed this Jul 26, 2026
JSONbored added a commit that referenced this pull request Jul 27, 2026
…k, and release AI-review locks before a hard kill can (#9174)

#8997 — a deploy restart can leave a PR wearing a decisive panel with no
matching disposition. SIGTERM kills whatever pass is in flight at an arbitrary
point, and the worst possible cut is between the public-surface publish (panel,
gate check-run, CI aggregate — all committed) and maybeRunAgentMaintenance ever
claiming its per-PR actuation lock for that head. Confirmed live on #8965: the
panel published at 14:55:24Z from the dying container, and the matching close
only landed at 15:07:53Z — ~12 minutes later, purely because an unrelated later
sweep tick happened to re-run the whole pass from scratch. The existing
outage-repair check (lastPublishedSurfaceSha !== headSha) cannot see this: the
surface IS current, so it already looks healthy.

maybeRunAgentMaintenance now records a marker (agent.maintenance.disposition_
considered) the moment it claims the actuation lock — the moment a real
disposition attempt begins, independent of what it decides. A new standalone
scan, reconcileSurfaceWithoutDisposition, rides the same sweep tick as the
existing pr-outcome/pending-closure repair scans (#9026/#9031) and re-enqueues a
regate for any open PR whose current-head surface carries no such marker.
Deliberately NOT folded into the existing surfaceRepairPriorityPullNumbers: that
function's return value drives the regular sweep's staleness-ordered fan-out and
is asserted against by a large, already-green test surface (dispatch shape,
ordering, backlog-restriction semantics) — an early attempt at injecting this
check there flagged every "surface current, nothing else recorded" fixture in
that file, which is a real, deliberate policy widening this fix makes, but one
that belongs in its own independently-testable scan rather than retrofitted onto
a heavily-relied-upon function blind.

#8998 — an orphaned ai-review-lock (30-min TTL) starves every subsequent
re-review, including an explicit maintainer re-run, with the "already in
progress" placeholder for real time nobody is reviewing anything. Two of the
three layers this needed were already shipped earlier this session: the
boot-time flush (#9021, ORPHANED_LOCK_KEY_PATTERNS already includes
ai-review-lock:*) and the explicit force-lock-steal for a maintainer's forced
re-run (#9008, `steal: webhook.forceAiReview === true` already threaded into
claimAiReviewLock). What was still missing: queue.stop() (#9007) lets an
in-flight job finish naturally, releasing its own lock via its own finally
block — but only if the orchestrator's SIGKILL grace period is long enough for
that drain to complete, and a common 10-30s grace period is shorter than an
AI-review LLM call legitimately runs. A new held-lock-registry tracks every real
ai-review-lock claim this process currently holds; the shutdown handler now
releases all of them FIRST, before anything else in the shutdown sequence, so
the lock is gone the instant SIGTERM/SIGINT arrives regardless of whether the
rest of shutdown gets time to finish. Complementary to the boot-time flush, not
a replacement for it: a true `kill -9` delivers no signal at all, so that
backstop still matters.

#8999 — a lock-contended hold applied the sticky manual-review label, and the
label then froze every LATER pass from running a FRESH AI review at all (only
replay) — meaning the one thing that could produce the real verdict needed to
auto-clear the label (already implemented, #9009) could never run, because the
freeze it depends on being lifted is gated by the very label it's trying to
clear. #9009 only handled the case where a fresh review DID run and showed
contention had resolved; it could not reach that state on a repo requiring
blocking AI review, because the freeze blocks the fresh attempt outright. Fixed
by exempting the freeze specifically when the label's own provenance — per the
same transient marker #9009 already reads/writes — is the lock-contention hold
and nothing else. This does not reopen the gaming surface the freeze exists to
close (a contributor iterating pushes to buy a fresh verdict): a lock-contention
hold is an infra artifact of two ORB passes racing, not a contributor action, and
if some OTHER reason also justifies the hold, that reason re-applies the label on
this same pass's own disposition regardless of the exemption.

#9013's largest remaining piece (a single per-PR mutex spanning the
public-surface publish AND the disposition plan/execute, currently two
separately-claimed critical sections) is NOT included here — an initial attempt
showed it needs a moderate refactor of both giant call sites (the sweep/CI-
completion path and the webhook path) with real risk of subtly changing
span/catch/decisionOutcome semantics the existing test suite exercises
extensively, and that deserves its own dedicated, carefully-tested pass rather
than a rushed addition alongside three already-substantial fixes.

Targeted tests only (no full local gate run this pass, per instruction):
typecheck clean, and every touched/new test file passes locally — 39 new tests
across surface-disposition-reconciler.test.ts, held-lock-registry.test.ts, and
the extended job-dispatch.test.ts fan-out coverage — plus a broader targeted
sweep of 967 tests across the queue/lifecycle/transient-lock/precision-breaker
suites most likely to interact with the actuation-lock, freeze, and sweep-
priority code paths this change touches. CI runs the full gate on push.

Closes #8997
Closes #8998
Closes #8999
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(engine): budget-cap kill-switch ceiling misclassifies as "deny" in the governor ledger

1 participant