Skip to content

feat(keeper): exclusive per-round lease across queue, store, and watch loop - #435

Merged
karagozemin merged 1 commit into
Sub-Rosa-Issue:mainfrom
classikdev:feat/keeper-round-lease
Sep 30, 2026
Merged

karagozemin merged 1 commit into
Sub-Rosa-Issue:mainfrom
classikdev:feat/keeper-round-lease

Conversation

@classikdev

Copy link
Copy Markdown
Contributor

Closes #384

Problem

The keeper queue, the on-disk store, and the watch loop schedule rounds
independently, so two keeper processes pointed at the same KEEPER_STORE_PATH
can both see a due round and both reveal/settle it. Issue #384 asks for an
exclusive lease per round that sits across all three layers.

What changed

services/keeper/src/store.ts — the lease itself

  • New RoundLease records owner, roundId, network, contractId, and
    expiresAtMs, persisted as StoreData.leases beside the round queue, keyed
    by contractId|network|roundId (absent scope = *, which matches anything).
  • claimRound() re-reads the store, then refuses while a live
    (expiresAtMs > now) lease for the same round is held by a different owner
    with an overlapping scope. Re-claiming with the same owner renews.
  • releaseLease() only removes leases owned by the caller, so a watcher can
    never hand back somebody else's round.
  • getLease() / listLeases() expose stored leases (expired or not).
  • Every mutator (addRound, removeRound, updateRound, claimRound,
    releaseLease) re-reads from disk before writing so a lease claimed by
    another process is merged forward instead of clobbered by a stale copy;
    saves are write-temp-then-rename, so no process ever reads a half-written
    store while deciding a claim.
  • Clock is injected (new KeeperStore(path, logger, clock), default
    systemClock); leases expire on that clock, so tests use FakeClock.
  • DEFAULT_LEASE_MS = 120_000, generateLeaseOwner(), parseLeaseMs().

services/keeper/src/watch-loop.ts — where the lease is held

  • runWatchLoop takes owner / leaseMs and claims each round before
    the tick; a refused claim logs who holds it and skips that round for this
    cycle instead of submitting alongside the winner.
  • The lease is released when the tick reaches a terminal status
    (Settled / Voided) or when the failure is definitive for the contract
    (isDefinitiveContractFailure: non-retryable, not a transport marker, and
    carrying a contract/Wasm marker), so the round goes back to the queue for
    whoever picks it up next. A transient failure (timeout, refused/reset
    connection) keeps the lease, so the same owner retries on the next tick.

services/keeper/src/queue.ts — CLI surface

  • claim <roundId> / release <roundId> (owner from KEEPER_OWNER, default
    queue-cli-<pid>), lease display in list, exit code 1 when a claim is
    refused or the owner does not hold the lease.

watch.ts / serve.ts / index.ts — KEEPER_OWNER and
KEEPER_LEASE_MS wiring, plus the lease API exported from the package entry.

Docs — services/keeper/README.md gains an "Exclusive round leases"
section (behaviour, store shape, env table) and the two new CLI rows;
docs/THREAT_MODEL.md "Double settle" row now mentions the per-round lease
(the Idempotent settle anchor is preserved).

Acceptance criteria

  • Two workers claiming the same round produce one submission —
    watch-lease.test.ts runs two runWatchLoop instances against one
    store with FakeClock; only the lease owner submits, the loser skips
    and its submitted stays empty. The queue harness
    (queue-replay.test.ts) asserts the same at the scheduling layer.
  • A lease for a different contract id does not block this contract —
    scope comparison in scopesOverlap, tested in store.test.ts and
    watch-lease.test.ts.
  • An expired lease can be claimed again once — expiry is computed from
    the injected clock; tested after clock.advance() in store, watch, and
    queue tests.
  • Tests use a fake clock and do not start a network server — all new
    tests run offline against temp store files with FakeClock.

Validation

  • pnpm keeper:typecheck — clean
  • pnpm keeper:test — 93 pass / 0 fail (store lease block, new
    watch-lease.test.ts, queue CLI claim/release, queue replay lease case)
  • pnpm coverage:test — gate passed, aggregate 80.47%, keeper 82.58%
  • pnpm logging:check, pnpm logging:test, pnpm errors:normalize:check,
    pnpm errors:normalize:test, pnpm docs:check, pnpm docs:check-links,
    pnpm threat-model:check — all pass
  • pnpm time:guard still reports the 9 pre-existing violations in
    packages/sdk/src/status-client.ts and apps/web/.../useDashboardData.test.ts
    (unchanged on this branch; not run by CI)

…h loop

Two keeper processes sharing one store could reveal or settle the same round
at the same time. The store now persists a lease (owner, round id, network,
contract id, expiry) beside the queue: the watch loop claims before each tick
and skips rounds another owner holds, releases only on terminal success or a
definitive contract failure, and expires leases on an injectable clock. Adds
`queue claim/release`, KEEPER_OWNER/KEEPER_LEASE_MS wiring, and offline
FakeClock tests for the queue, store, and watch loop.

Closes Sub-Rosa-Issue#384
@drips-wave

drips-wave Bot commented Sep 29, 2026

Copy link
Copy Markdown

@classikdev Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stop two keeper watchers from settling the same queued round

2 participants