fix(dispatcher): circuit breaker + bounded retry against sustained LLM overload (#1058)

[bedaHovorka]

Summary

• Fixes Timeouts during system overload #1058: under sustained LLM slowness, KoogAgentPlanAdapter paid a full 30s withTimeout stall on every tick with no backoff, and each timeout printed a duplicate full stack trace from Koog's own internal logger.
• Adds LlmCircuitBreaker (CLOSED → OPEN → HALF_OPEN), keyed on simulation time rather than wall clock: after circuitBreakerFailureThreshold consecutive cycle failures it opens and skips the LLM entirely for circuitBreakerCooldownSeconds, then probes recovery once. Both knobs follow the existing -D > dispatcher-defaults.properties > code-constant precedence.
• Adds one bounded retry for a plain (non-timeout) failure only — a timeout is never retried, since that would double the stall an overloaded system can least afford.
• Quiets Koog's own duplicate "Execution exception reported by server!" ERROR log via the Logback API in desktop-ui/Main.kt, deliberately not by editing :core's shared logback.xml (kept :core untouched entirely for this fix). Tradeoff: this also silences any other Koog-internal error from that package, not just timeouts.
• plan() was restructured so the outer exception handler only wraps getOrCreateAgent(), not the whole cycle — needed to avoid double-reporting a tick when a retried/failed cycle's own fallback throws.

Test plan

• :dispatcher-agent:test — 1785 tests pass, including 17 new tests (LlmCircuitBreakerTest state-machine coverage, KoogAgentPlanAdapterTest retry/breaker scenarios, DispatcherRunConfigTest/DispatcherDefaultsResourceTest for the two new config knobs)
• :dispatcher-agent:integrationTest — passes, including live-Ollama-tagged tests
• :desktop-ui:test — 807/808 pass (1 pre-existing skip)
• ktlintCheck, detekt — clean
• Built shadowJar and ran a real headless example end to end to confirm startup and the new logging code path work

🤖 Generated with Claude Code

[bedaHovorka]

code cleanup findigs

[Copilot]

🟡 Changes recommended

Retries can duplicate partially emitted actions, half-open probing is not concurrency-safe, and breaker settings are missing from run identity.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds bounded retries and a simulation-time circuit breaker to reduce repeated LLM stalls during sustained overload.

Changes:

• Adds configurable CLOSED/OPEN/HALF_OPEN breaker behavior.
• Retries non-timeout inference failures once and expands tests.
• Suppresses duplicate Koog exception logging.

File summaries

File	Description
dispatcher-agent/.../MeasuringPlanAdapterTest.kt	Updates measurement expectations for retries.
dispatcher-agent/.../LlmCircuitBreakerTest.kt	Tests breaker state transitions.
dispatcher-agent/.../KoogAgentPlanAdapterTest.kt	Tests retries, timeouts, and breaker behavior.
dispatcher-agent/.../DispatcherRunConfigTest.kt	Tests breaker configuration parsing.
dispatcher-agent/.../DispatcherDefaultsResourceTest.kt	Verifies shipped breaker defaults.
dispatcher-agent/.../dispatcher-defaults.properties	Defines breaker defaults.
dispatcher-agent/.../LlmCircuitBreaker.kt	Implements the circuit breaker.
dispatcher-agent/.../KoogAgentPlanAdapter.kt	Integrates retries and fallback handling.
dispatcher-agent/.../DispatcherRunConfig.kt	Adds breaker configuration fields.
dispatcher-agent/.../DispatcherDefaultsResource.kt	Recognizes the new properties.
desktop-ui/.../Main.kt	Suppresses duplicate Koog logs.
desktop-ui/.../ExampleRegistry.kt	Wires configured breakers into LLM planners.

Review details

Suppressed comments (1)

dispatcher-agent/src/main/kotlin/cz/vutbr/fit/interlockSim/dispatcher/planner/LlmCircuitBreaker.kt:90

• HALF_OPEN allows every caller, not exactly one probe: after one concurrent caller transitions OPEN to HALF_OPEN, all others entering shouldAttempt take this branch and hit the overloaded LLM too. Track a single in-flight probe and reject later callers until it records success/failure; also release that state if the probe coroutine is cancelled.

	fun shouldAttempt(simTime: Double): Boolean =
		when (state) {
			State.CLOSED -> true
			State.HALF_OPEN -> true
			State.OPEN -> {
				if (simTime - openedAtSimTime >= cooldownSeconds) {
					state = State.HALF_OPEN
					true

• Files reviewed: 12/12 changed files
• Comments generated: 4
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[bedaHovorka]

The review summary also named one suppressed finding with no inline thread: half-open probing is not concurrency-safe. That one is fixed too, in 3f5ca32:

• shouldAttempt now grants exactly one HALF_OPEN probe at a time — a probeInFlight flag under the class monitor; a second caller within the same window is skipped (totalSkips incremented).
• recordSuccess/recordFailure release the claim on every verdict path.
• New abandonProbe() releases the claim without a verdict, and plan() wraps the attempt section in try/finally { circuitBreaker.abandonProbe() } — a probe coroutine cancelled mid-inference can no longer wedge the breaker in HALF_OPEN for the rest of the run. It is a no-op on every normal path.

Covered by a cancelled HALF_OPEN probe does not wedge the breaker, HALF_OPEN grants exactly one probe at a time, a failed probe re-arms the single-probe guard for the next cooldown window, and an abandoned probe can be attempted again.

Full gate green locally (build, detekt, ktlintCheck, test, integrationTest). Run-JSON breaker stats are filed as follow-up #1074.

[bedaHovorka]

ok

[Copilot]

🟡 Changes recommended

Review findings remain in breaker initialization/cycle handling and run-report configuration identity.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (2)

dispatcher-agent/src/main/kotlin/cz/vutbr/fit/interlockSim/dispatcher/planner/KoogAgentPlanAdapter.kt:363

• The breaker is consulted only after getOrCreateAgent(). If agent creation/assembly fails, this returns to the fallback before shouldAttempt() or recordFailure(), leaving agent null so every subsequent tick repeats the initialization and warm-up work without ever opening the breaker. Check the breaker before initialization or route creation failures through the same breaker failure path so sustained initialization failures are bounded too.

		val a =
			try {
				getOrCreateAgent()
			} catch (e: CancellationException) {
				throw e
			} catch (e: Exception) {

dispatcher-agent/src/main/kotlin/cz/vutbr/fit/interlockSim/dispatcher/planner/RunParameters.kt:49

• These defaults misrepresent pre-#1058 snapshots: those runs had no circuit breaker, so a 3-failure threshold and 60-second cooldown did not actually apply. Because RunReportAggregator groups on the full RunParameters, decoding an old JSON with these values can merge it with a current default-breaker run and report them as the same experiment. Preserve an explicit legacy/unknown value (or version the snapshot) rather than claiming these settings were used.

 *   Defaults to [LlmCircuitBreaker.DEFAULT_FAILURE_THRESHOLD], the threshold every run used before
 *   this field existed, so decoding a run JSON that predates the field records the value that
 *   actually applied rather than an unknown one.

• Files reviewed: 18/18 changed files
• Comments generated: 2
• Review effort level: Lite

[bedaHovorka]

ok

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
96.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud