Repository navigation
feat(keeper): exclusive per-round lease across queue, store, and watch loop - #435
Merged
karagozemin merged 1 commit intoSep 30, 2026
Merged
Conversation
…h loop Two keeper processes sharing one store could reveal or settle the same round at the same time. The store now persists a lease (owner, round id, network, contract id, expiry) beside the queue: the watch loop claims before each tick and skips rounds another owner holds, releases only on terminal success or a definitive contract failure, and expires leases on an injectable clock. Adds `queue claim/release`, KEEPER_OWNER/KEEPER_LEASE_MS wiring, and offline FakeClock tests for the queue, store, and watch loop. Closes Sub-Rosa-Issue#384
|
@classikdev Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits. You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #384
Problem
The keeper queue, the on-disk store, and the watch loop schedule rounds
independently, so two keeper processes pointed at the same
KEEPER_STORE_PATHcan both see a due round and both reveal/settle it. Issue #384 asks for an
exclusive lease per round that sits across all three layers.
What changed
services/keeper/src/store.ts— the lease itselfRoundLeaserecordsowner,roundId,network,contractId, andexpiresAtMs, persisted asStoreData.leasesbeside the round queue, keyedby
contractId|network|roundId(absent scope =*, which matches anything).claimRound()re-reads the store, then refuses while a live(
expiresAtMs > now) lease for the same round is held by a different ownerwith an overlapping scope. Re-claiming with the same owner renews.
releaseLease()only removes leases owned by the caller, so a watcher cannever hand back somebody else's round.
getLease()/listLeases()expose stored leases (expired or not).addRound,removeRound,updateRound,claimRound,releaseLease) re-reads from disk before writing so a lease claimed byanother process is merged forward instead of clobbered by a stale copy;
saves are write-temp-then-
rename, so no process ever reads a half-writtenstore while deciding a claim.
new KeeperStore(path, logger, clock), defaultsystemClock); leases expire on that clock, so tests useFakeClock.DEFAULT_LEASE_MS = 120_000,generateLeaseOwner(),parseLeaseMs().services/keeper/src/watch-loop.ts— where the lease is heldrunWatchLooptakesowner/leaseMsand claims each round beforethe tick; a refused claim logs who holds it and skips that round for this
cycle instead of submitting alongside the winner.
(
Settled/Voided) or when the failure is definitive for the contract(
isDefinitiveContractFailure: non-retryable, not a transport marker, andcarrying a contract/Wasm marker), so the round goes back to the queue for
whoever picks it up next. A transient failure (timeout, refused/reset
connection) keeps the lease, so the same owner retries on the next tick.
services/keeper/src/queue.ts— CLI surfaceclaim <roundId>/release <roundId>(owner fromKEEPER_OWNER, defaultqueue-cli-<pid>), lease display inlist, exit code 1 when a claim isrefused or the owner does not hold the lease.
watch.ts/serve.ts/index.ts—KEEPER_OWNERandKEEPER_LEASE_MSwiring, plus the lease API exported from the package entry.Docs —
services/keeper/README.mdgains an "Exclusive round leases"section (behaviour, store shape, env table) and the two new CLI rows;
docs/THREAT_MODEL.md"Double settle" row now mentions the per-round lease(the
Idempotent settleanchor is preserved).Acceptance criteria
watch-lease.test.tsruns tworunWatchLoopinstances against onestore with
FakeClock; only the lease owner submits, the loser skipsand its
submittedstays empty. The queue harness(
queue-replay.test.ts) asserts the same at the scheduling layer.scope comparison in
scopesOverlap, tested instore.test.tsandwatch-lease.test.ts.the injected clock; tested after
clock.advance()in store, watch, andqueue tests.
tests run offline against temp store files with
FakeClock.Validation
pnpm keeper:typecheck— cleanpnpm keeper:test— 93 pass / 0 fail (store lease block, newwatch-lease.test.ts, queue CLI claim/release, queue replay lease case)pnpm coverage:test— gate passed, aggregate 80.47%, keeper 82.58%pnpm logging:check,pnpm logging:test,pnpm errors:normalize:check,pnpm errors:normalize:test,pnpm docs:check,pnpm docs:check-links,pnpm threat-model:check— all passpnpm time:guardstill reports the 9 pre-existing violations inpackages/sdk/src/status-client.tsandapps/web/.../useDashboardData.test.ts(unchanged on this branch; not run by CI)