feat: Add retry state for RETRY-spec backoff - #522
Conversation
|
|
||
| # The delay bounds of the extended regime, in seconds. A component enters the | ||
| # extended regime after an unexpected failure. | ||
| EXTENDED_INITIAL_DELAY = 5 * 60 |
There was a problem hiding this comment.
Is it the norm in python that all time values / APIs are float seconds?
There was a problem hiding this comment.
Yes. time.monotonic(), time.sleep(), threading.Event.wait() and asyncio.sleep() all take or return float seconds, so a float is what every call site here needs. Config already exposes poll_interval and initial_reconnect_delay as float seconds too, so this matches what callers already pass in.
|
|
||
| # An upper bound on the backoff exponent, so a long outage cannot overflow the | ||
| # delay computation. Any real ceiling is reached long before this. | ||
| _MAX_BACKOFF_EXPONENT = 30 |
There was a problem hiding this comment.
This approach for calculation can remove the need for bounds protection logic.
There was a problem hiding this comment.
We are limited by the float seconds data type that is used in the SDK so those bound protections won't work.
| self._extended_initial_delay = extended_initial_delay | ||
| self._extended_ceiling = extended_ceiling | ||
| self._reset_policy = reset_policy | ||
| self._operating_cadence = operating_cadence |
There was a problem hiding this comment.
It seems like this operating_cadence is to help with the normal second poll after the first success? Can't this be calculated based on normal_initial_delay? It seems like redundant state? Maybe I'm not understanding the usage of operating_cadence and initial_delay set to different non-zero values.
There was a problem hiding this comment.
You can think of operating cadence as the time between normal healthy operations and the normal_initial_delay as the starting point for delays between failed operations. For polling, these are the same, but for streaming the default cadence is 0s, but the failed starts at 1s or whatever the user configures so that a backoff can actually be calculated.
| """ | ||
| self._reset_policy.note_healthy() | ||
| self._reset_if_due() | ||
| self._next_delay = self._operating_cadence |
There was a problem hiding this comment.
This can be 0? I think this indicates a logical flaw in how the values are flowing through / overriding each other.
There was a problem hiding this comment.
It can be 0, we can meet if it would be helpful to talk through this but 0 is intentional (for streaming).
| log.warning( | ||
| "%s must be a positive, finite number of seconds; using the default of %ss" | ||
| % (name, default) | ||
| ) |
There was a problem hiding this comment.
Are logger instances ever passed around in the code base? This static call stands out to me coming from our other code bases.
There was a problem hiding this comment.
No, its is the standard in this repo to pull from the import. And it is different then most other languages.
…loor Streaming's operating cadence is zero rather than absent, so a healthy stream schedules no wait and a retry delay has no floor. Polling is the only data source that reads next_delay as a DelaySource, and its cadence floor keeps that from reaching zero. Make the seven observational properties private. No data source reads them; they are test instrumentation, and as public API they invite callers to attach logic to unsynchronized state.
The RETRY spec binds a normal ceiling and an extended ceiling. Use those words for the configured bounds so the parameters read as the spec does, and keep the _delay suffix the initial-delay names already use. The mutable maxDelay keeps its name: Requirement 1.3.2 calls it that.
…e delay The exponent driver was _n and a second counter held the name attempts, which is what Requirement 1.4.1 calls the exponent driver. A reviewer reading self.attempts against the spec was reading the wrong field. The second counter is gone. It had no reader outside tests, not even a logger, and every test that used it recorded only normal failures -- where the two counters are equal by construction.
Summary
Adds the retry state machine that the FDv1 data sources will use for RETRY-spec conformance. Pure addition — nothing imports it yet, so it can be reviewed against the spec without reading any data-source code.
RetryStateanswers one question: how long to wait before the next attempt. It serves all four data sources; streaming and polling differ only in their construction parameters and their reset policy.The algorithm
Every failure is classified
NORMALorUNEXPECTED.400,408,429and all5xxare normal; every other4xx, including401and403, is unexpected. Only an HTTP status can be unexpected — every network and TLS failure is normal.A normal failure advances the delay on the current curve. An unexpected failure raises both bounds and keeps them raised until the reset condition is met, so a normal failure that follows cannot lower them.
Each delay is then reduced by a jitter of up to half itself, and never falls below the caller's operating cadence —
poll_intervalfor polling, absent for streaming.Two things that are easy to get wrong
A backoff wait applies to a retry, not to every operation.
record_successrestores the operating cadence even while the retry state is still raised, so a recovered service is polled at its normal rate immediately. Another SDK shipped without this: after an outage its first successful poll still waited twenty minutes.Clearing the retry state is separate from restoring the cadence.
record_successdoes the latter;_reset_if_duedoes the former, and only when the reset policy is satisfied. Streaming's policy is 60 seconds of continuous healthy operation, polling's is two consecutive successes._reset_if_dueis called fromrecord_failurebecause nothing runs while a stream is healthy — the moment the 60-second threshold is crossed is otherwise unobservable, so the next failure is the only place it can be noticed.Why TLS failures are normal
The narrow classification was built first and reverted. All SDK traffic is HTTPS, so a genuine certificate problem cannot be distinguished from an ordinary transient fault across platforms — a peer sending FIN mid-handshake surfaces differently from one sending RST, and the classification would depend on which an intermediary happened to send. A connection flapping faster than the reset threshold then ratchets to the hour ceiling with no way out.
Testing
make test: 1599 passed.make lint: clean across 226 files.71 tests covering both delay ladders exactly, jitter bounds, ceiling stickiness, both reset policies, the cadence floor, and the classification table. No test sleeps: the clock and the jitter are patched at the module level, so the 60-second reset and a 20-cycle flapping scenario both run instantly.
Note
Overview
Introduces a standalone
RetryStatemodule (ldclient/impl/retry.py) that will drive FDv1 data-source reconnect/poll timing. Nothing in production imports it yet.Failures are classified as normal vs unexpected (via
classify_http_status: most 4xx except 400/408/429 are unexpected; 5xx and transport errors stay normal). Backoff doubles to a ceiling, subtracts up to half jitter, and never waits less than the operating cadence (zero for streaming,poll_intervalfor polling). An unexpected failure switches to an extended delay ladder (5–60 minutes) that stays sticky until reset.Streaming uses
for_streamingwith exponential normal delays (default 1s → 30s cap) and resets after 60s of healthy operation. Polling usesfor_pollingwith flat normal retries at the poll interval and resets after two consecutive successes.record_successrestores the healthy cadence immediately even while extended state is still raised; clearing extended state is separate and gated by the reset policy.Adds
test_retry.py(delay tables, jitter bounds, reset policies, invalid config fallbacks) plustest_utilhelpers to patch retry’stime/randomwithout sleeping.Reviewed by Cursor Bugbot for commit 4d300c4. Bugbot is set up for automated code reviews on this repo. Configure here.