Skip to content

fix(tui): recover poisoned TLS client generations - #694

Closed
yxlyx wants to merge 2 commits into
mainfrom
fix/691-tls-client-recovery
Closed

fix(tui): recover poisoned TLS client generations#694
yxlyx wants to merge 2 commits into
mainfrom
fix/691-tls-client-recovery

Conversation

@yxlyx

@yxlyx yxlyx commented Aug 30, 2026

Copy link
Copy Markdown
Collaborator

Closes #691.

What changed

  • Added a launch-level recovery manager for model HTTP traffic. Both non-streaming http.postWatched calls and streaming Agent.postStreamWithClient calls lease the active client generation.
  • When std.http.Client.request returns TlsInitializationFailed before a Request exists, retire that generation and publish a CA-prewarmed replacement. Concurrent callers keep the retired generation alive until their final lease releases; stale reports do not rotate twice.
  • Added a lifecycle gate around global recovery lookup/publication and teardown. Shutdown closes admission, drains active/retired leases and CA-ready waiters, then destroys owned clients; calls targeting the retired launch client are rejected while unrelated std.http.Client instances remain usable.
  • Centralized the launch CA-ready barrier at the two request constructors, including direct postWatched users such as server compaction, and added distinct breadcrumbs for launch CA failure, replacement CA failure, TLS rotation, and final retry exhaustion.
  • Corrected retry accounting so a six-attempt plan performs six attempts rather than seven, keeps one total-attempt budget when error classes change, and preserves 429/5xx attribution at exhaustion.
  • Added deterministic regressions for real malformed TLS handshakes through both production catches, repeated and simultaneous stale-generation failures, replacement allocation/CA failures, shutdown races, lease reclamation, retry exhaustion followed by successful later root/child calls, TUI root-agent construction, foreground runSub, and background execSubagent/agentOutput.

Why

A transient network transition can poison the launch-owned std.http.Client at request construction, before the existing post-construction connection cleanup has a Request to mark closing. Retrying the same client therefore leaves every later model-backed TUI trajectory failing until the process is replaced.

Generation leasing fixes that lifecycle boundary rather than masking individual failures. One poisoned client can be retired immediately while in-flight compaction, root, or child calls safely finish on the object they already leased, and later retries resolve the original launch pointer to the replacement.

The design intentionally applies to model request constructors rather than every best-effort launch HTTP consumer. The original launch client is never deinitialized by rotation because telemetry/fleet callers may still hold its pointer. The cost is one mutex-protected lookup per model request plus temporary retention of retired generations while calls remain in flight.

Reinitializing the shared client in place was rejected because concurrent users can still dereference it. Creating a fresh client for every request was rejected because it discards connection pooling and repeatedly reloads CA state. Relying on debug assertions or external job-join ordering was rejected because ReleaseFast teardown still needs an in-process lifetime guarantee.

Verification

  • Three independent five-agent adversarial review rounds covered streaming roots/synthetic wakes, foreground/background children and workflows, compaction/server-compaction, title/recap and CA observability, retry accounting, plus concurrency/memory/shutdown edges.
  • scripts/eval-tier1.sh — green with the test-count ratchet updated to the verified suite size.
  • Unit suite: 1,783 passed, 1 skipped.
  • Focused TLS suite: 54/54 passed.
  • Fullscreen TUI suite: 459/459 passed.
  • Real-binary TUI guards: 17/17 passed.
  • zig build -Doptimize=ReleaseFast — green.
  • git diff --check — clean.

A request-construction TLS failure occurs before the existing connection poison can run, so retries and later TUI trajectories kept reusing a launch-scoped client that only process replacement could heal. Route model HTTP calls through leased generations, rotate on construction failure without deinitializing in-flight users, and preserve the original client for unrelated launch consumers.

Gate every managed constructor on CA readiness, distinguish CA warm and request-construction failures in traces, and cover concurrent stale reports plus later ordinary and child POST recovery with a loopback regression.

Co-Authored-By: Codegraff <blackfloofie@codegraff.com>
Close model-client admission before teardown, drain active generation leases and CA-ready waiters, and serialize global lookup with destruction. This prevents ReleaseFast use-after-free races while allowing unrelated HTTP clients to remain usable.\n\nCorrect retry accounting so the advertised limit is the actual total, preserve throttle/server failure attribution, and add deterministic real-TLS, concurrent, allocation, TUI-root, foreground-child, and background-child recovery coverage. Remove the unrequested ADR and raise the test-count ratchet to the verified suite size.\n\nCo-Authored-By: Codegraff <blackfloofie@codegraff.com>
@justrach

Copy link
Copy Markdown
Owner

Folded into release/v0.0.282 (#696), including the TLS-lifecycle follow-up. Closing as superseded. ADR is 0048.

@justrach justrach closed this Aug 31, 2026
@cursor
cursor Bot deleted the fix/691-tls-client-recovery branch August 31, 2026 01:30
cursor Bot pushed a commit that referenced this pull request Aug 31, 2026
…ycle

yxlyx: drain generation leases on teardown, fix retry accounting, add
real-TLS / child coverage. Keep ADR 0048 (the first #694 commit's
record, remapped off the TUI-claims 0042 slot).

# Conflicts:
#	docs/adr/0048-model-http-client-recovery-uses-generations.md
#	docs/adr/README.md
#	scripts/eval/tier1-manifest.json
cursor Bot pushed a commit that referenced this pull request Aug 31, 2026
The follow-up dropped the TLS-generation record; 0042 is TUI claims
and 0048 stays the leased-client decision. Floor stays 1817.
cursor Bot pushed a commit that referenced this pull request Aug 31, 2026
Complementary to #694 model-HTTP generations: MCP HTTP/WSS keep the
warmed client and accept gzip catalogs. ADR remapped off 0043.

# Conflicts:
#	docs/adr/README.md
#	src/http_warm.zig
#	src/test_hooks.zig
cursor Bot pushed a commit that referenced this pull request Aug 31, 2026
Record the ADR remaps (0049 / 0050) and the extra #694 lifecycle
hardening. Still no tag.
cursor Bot pushed a commit that referenced this pull request Aug 31, 2026
The remote advanced from da6511b with #695 while this tip already had
#697/#679/#694 follow-up and the 1848 ratchet.
justrach added a commit that referenced this pull request Aug 31, 2026
…recovery

The malformed-handshake test asserts error.TlsRequestConstructionFailed
from both production constructor catches, and green on Linux/macOS. On
Windows CI the handshake read dies with NTSTATUS LOCAL_DISCONNECT,
which std surfaces as error.Unexpected — raw, bypassing the recovery
rotation the test (and #694's contract) demands.

constructionTlsFailure() is the shared predicate: TlsInitializationFailed
always, plus error.Unexpected when the request is https (a plain-HTTP
connect error must not be mislabeled a TLS failure). Both catches —
http.post and the streaming path — consult it.

Co-Authored-By: Codegraff <blackfloofie@codegraff.com>
yermakoffivan pushed a commit to yermakoffivan/codegraff that referenced this pull request Aug 31, 2026
… generations

yxlyx: lease/rotate the shared std.http.Client on TlsInitializationFailed
at request construction. ADR number will be remapped off 0042.

# Conflicts:
#	docs/adr/README.md
yermakoffivan pushed a commit to yermakoffivan/codegraff that referenced this pull request Aug 31, 2026
…justrach#694

Record the three product merges on the new release branch, remap the
TLS-generation ADR to 0048 so 0042 stays TUI claims, and leave justrach#277 /
justrach#200 parked.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tui/http: TlsInitializationFailed persists across turns until process restart

2 participants