Skip to content

add credit leases, reservations, and preflight checks - #109

Open
bpapillon wants to merge 20 commits into
mainfrom
bpapillon/sch-7509-schematic-java-credit-leases-reservations-and-preflight
Open

bpapillon wants to merge 20 commits into
mainfrom
bpapillon/sch-7509-schematic-java-credit-leases-reservations-and-preflight

Conversation

@bpapillon

@bpapillon bpapillon commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Brings the Java SDK to parity with Node, Go, and Python on credit leases and reservations.

check with a usage now reserves credits: in client mode against a lease held locally (in memory, or in Redis so a fleet shares one lease), in server mode with one check-and-reserve call. trackWithReservation settles the reservation and release refunds an unused one. The plain check takes usage, eventUsage, and creditCost preflight options, threaded to both the API body and the local engine. Identify can prewarm a lease. Extend and check-and-reserve each send a fresh idempotency key per call and keep the transport's default retries. A per-check timeout bounds the server-mode call, the lease acquire and extend, and a joiner's wait on a shared extend.

The Redis key layout and Lua scripts are Node's, byte for byte, so a mixed-language fleet shares leases. conformance/ is copied verbatim from schematic-node and the runner passes every vector on both backends. Java has no in-process Redis fake, so the Redis backend runs on embedded Redis; the Lua expiry guard reads the Redis server clock, so the runner moves time by shifting stored deadlines back rather than by faking the clock.

Holds, settle debits and the settle's track quantity all round a fractional usage up to whole event units, as every SDK now does. RulesEngine gains a default four-argument checkFlag that ignores its options, so existing implementations and mocks keep working. Everything new is Java 8 source compatible.

Behaviour change: a DataStreamClient that fails to start no longer throws from the constructor. The client logs it and falls back to the API, and auto lease mode resolves to server gating per check.

build.gradle is generated; the embedded-redis test dependency lives in fern-config custom-dependencies (schematic-fern-config #492, merged) so a regen keeps it.

@bpapillon
bpapillon requested a review from a team as a code owner September 17, 2026 15:20
Zero prewarm timeout reads the cache instead of skipping, a lease
won in a lost acquire race is released during shutdown, the Redis key
prefix inherits from DataStream independently of the client, the
per-check timeout reaches the REST fallback, and a debit that names no
lease is a store failure rather than pinned to the acquired lease.
The API's flag check takes a preflight now, so a check that falls back
to it answers the same hypothetical the local engine would. A
preflighted check skips the flag cache, which is keyed without it.
Close passes what remains of its one deadline to the drain, the release
loop, and the manager, so a shutdown cannot exceed it. Identify with a
prewarm after close logs instead of throwing, prewarm accepts a null
list, a dropped lease retires its lock, and a zero-usage check stays
cacheable.
The plain check resolved a failed API call through the client-wide
flag default and never saw the per-check one. Also passes no credit
source without DataStream instead of one that throws, completes a
lease flight in finally so an Error cannot strand joiners, and debits
a fractional settle by the units the event bills.
…units

A check tests the water mark on its own thread and spawns a top-up
only when one is due, and a spawned top-up skips a slot with an
extend already in flight. A reservation is sized from the rounded-up
usage so it matches what the settle bills. Also re-reads the row after
winning a flight, re-checks stop under its lock, clamps timeouts, and
bounds the prewarm fetch.
A lease that lands after stop is left to the drain or server expiry,
as in node; releasing it refunded a lease sibling pods share. Hold
sizing and the settle debit go back to the spec's usage times rate.
The lease path reads only the cache when the stream is down or in
replicator mode, and a joiner behind a flight that sent no extend may
issue its own.
Auto mode resolved to client gating whenever a DataStream client existed,
including when the WASM engine failed to load. Every credit check then
failed its probe and fell through to a plain, ungated flag check: credits
handed out with no hold and no debit. Auto now requires a loaded engine
and falls to server gating without one.

releaseAllLocalLeases issued its releases serially and only consulted the
budget between them, so one hung release spent the whole budget and every
lease behind it was abandoned. They now go out together with the budget
bounding the set, matching schematic-node's releaseAllLocalLeases
(src/credits/lease-manager.ts:571-593, settleWithin over all releases).

Also: guard a check-and-reserve response with no data rather than
dereferencing it, floor millisAsInt so a negative duration cannot read as
an instant timeout, and decode the reservation eval context field by
field so a sibling SDK's wider request body does not silently cost the
entity keys a recovered hold bills with.

The conformance runner rounded the expected track quantity up before
comparing, which would pass a vector that schematic-node
(tests/conformance/runner.test.ts:690) and the Go and Python runners
fail. It now asserts the number the vector states. The README said the
suite was copied from conformance/ on main in schematic-node; it tracks
that repo's conformance directory, not a particular branch.
plainCheck passed the DataStream client's answer straight through, and
that answer is the flag's own default whenever the engine declined
(RULES_ENGINE_UNAVAILABLE, RULES_ENGINE_ERROR). A caller's defaultValue
was therefore honoured offline and on the API fallback but ignored on the
branch that answers most checks. Matches schematic-node 8610595
(src/wrapper.ts, `value: resp.value ?? getDefault()`), found in the Ruby
port review and fixed there as aca9075.

Also warn at startup when an explicit client mode has no rules engine.
Auto resolves that away by gating server-side, but an explicit mode is
the caller's to keep, so say what it costs: every check falls through to
a plain flag check with no credit gating.
PreflightOptions claimed the engine evaluates the raw fractional quantity
and only the wire body rounds. WasmRulesEngine.putQuantity rounds at the
engine boundary too, and has since the envelope started carrying the
preflight: its quantity fields are integers and a decimal point fails to
deserialize there. Say where the rounding happens, and note that the
flag_check event is reported before the caller's default is resolved.
Auto resolves to server gating without a loaded rules engine, but the
constructor still built the stores, the manager and its sweeper whenever
a DataStream existed, leaving a sweeper polling an index nothing writes
to and a "server mode, so these options are ignored" warning that never
fired for that case. Both now use the same readiness the check does.

Also: the reserved-credits javadoc still described the pre-rounding
figure, the README quoted a one-hour server TTL cap where the clamp
leaves a minute for clock skew, and getPrewarm handed back the list the
constructor copies defensively.
@bpapillon bpapillon self-assigned this Sep 21, 2026
@cbrady

cbrady commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

here's some claude-slop:

  1. Extend single-flight is check-then-act, so concurrent callers each send their own extend. In CreditLeaseManager.java:
  Flight inFlight = extendFlights.get(key);
  if (inFlight != null && joinsLeft > 0) { ... }
  return startExtend(...);   // startExtend: extendFlights.put(key, flight);
     - When several threads see the lease drop below its low-water mark at once, they can all get null from extendFlights.get(key). Each then calls put and sends a wire extend with its own fresh idempotency key.
     - The re-check in startExtend (stillNeedsExtending) only catches an extend that has already landed, not one still on the wire.
     - With N threads, the server grants N top-ups, and the lease holds N×leaseSize of the company's balance, which can starve other processes sharing that balance.
     - Node doesn't have this because its get-then-set runs without an await in between.
     - Fix: register with putIfAbsent when there is no flight. Keep the deliberate overwrite only for the path where a caller has used up its joins, via replace(key, old, new).
  2. Server mode doesn't round fractional usage up. ServerCreditCheck.java sends .quantity(request.getUsage()), but Node sends quantity: Math.ceil(options.usage) (added in node #194).
     - The settle does round up ((long) Math.ceil(actualQuantity)). So with usage(0.5) at a rate of 10, the hold is 5 credits but the settle bills 10.
     - Every fractional server-mode check holds less than it will bill, and client mode already rounds up, so the two modes disagree.
     - ServerCreditCheckTest.sendsAFractionalUsageAsTheQuantityToHold asserts 0.5, which locks in the old behaviour. Its comment ("the server holds the same amount every other SDK would") is now wrong.
  3. The per-check timeout restarts at every step instead of bounding the whole check.
     - In the maybeExtend loop, every join calls inFlight.await(timeout) with the full budget. After up to MAX_EXTEND_JOINS = 2 joins, the caller sends its own extend with another full timeout.
     - CreditCheck also passes the full request.getTimeout() to both acquireIfNeeded and maybeExtend.
     - A 200ms check can take about 800ms. Node computes one joinDeadline up front and waits only for whatever time is left.
     - Fix: turn the timeout into a nanoTime deadline once, at the start of the check, and pass the time remaining to each step.

Both lease stores read an empty pin as no pin and credit whichever lease
holds the slot, which could inflate a successor. The reservation stores
now decline the refund themselves, matching the other SDKs on a shared
Redis; the slice returns when the lease expires.
drain() re-snapshots pending work until none is left. A future that had
completed but not yet left its set made that loop spin without waiting.
Each lease step was handed the full per-check timeout, and each extend
join restarted it, so a check that joined an acquire and then an extend
could spend its timeout several times over.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants