adaptive: forced switches leave keep-alive connections behind — async conns miss the 1.2 s flap window in every run, and a one-worker io_uring strands sync conns until their requests fail

[FumingPower3925]

Measured on celeris 468ce53 for celeris#641: go test -race -count=1 -timeout=300s -json ./adaptive/... in a Docker Desktop linux/arm64 container (golang:1.27 = go1.27.1, --cpus 4, seccomp=unconfined). Three runs at memlock 8 MiB, the GitHub-hosted default, where io_uring is capped to one worker. Three more at 128 MiB, where it gets four. Every number below comes from the tests' own log lines.

TestBidirectionalFlapAsync fails in all 6 runs, at both memlock limits

The test runs 64 keep-alive clients with AsyncHandlers: true. Each flap calls ForceSwitch, sleeps a flat 1200 ms, then asserts at least 32 conns on the new active engine and at most 16 left on the standby.

run	flap 1 → io_uring, active / standby	flap 3 → io_uring, active / standby	request errors	async promotions, epoll / io_uring
8 MiB r1	18 / 46	19 / 45	0	82 / 37
8 MiB r2	18 / 46	18 / 46	0	82 / 36
8 MiB r3	31 / 33	21 / 43	0	95 / 52
128 MiB r1	14 / 50	22 / 42	0	78 / 36
128 MiB r2	22 / 42	21 / 43	0	86 / 43
128 MiB r3	18 / 46	22 / 42	0	82 / 40

Flap 2 (io_uring → epoll) reaches 64 / 0 in every run, so the loss is one direction only. epoll → io_uring moves about a third of the async connections within 1.2 s, whatever io_uring's worker count, and no request fails.

By design, a promoted async conn moves only when its dispatch goroutine is parked, idle and flushed (engine/epoll/transplant.go:82-133). A client that sends its next request at once keeps that goroutine busy. Two readings fit the data, and they need different fixes:

• The window is too short. The transplant converges, just slower than 1.2 s under back-to-back load. That is the same class fix(iouring): stop the celeris#364 revert deciding on a dangling parser-buffer view, and repair the three test oracles that hid it (celeris#631, celeris#628) #637 fixed for the ramp tests by polling instead of sleeping.
• Busy async conns never reach the park boundary. While load continues they stay on epoll after a promotion. That is the hollow promotion Adaptive engine never promotes to io_uring at high concurrency (1024c) — default engine leaves ~14.5% throughput unused #396 fixed for sync conns, still open for async ones.

To tell them apart, poll the per-engine counts after ForceSwitch (as awaitPhase does) and record how long it takes to reach 60 on the active engine. Record both engines' transplant accounting from EngineMetrics at the end.

TestBidirectionalFlap (sync) and TestReverseTransplant fail only with a one-worker io_uring

run	flap 3 → io_uring, active / standby	missing at flap 3 (64 − active − standby)	errors at flap 3	errors by test end
8 MiB r1	31 / 12	21	21	58
8 MiB r2	27 / 34	3	3	49
8 MiB r3	34 / 27	3	3	49
128 MiB r1–r3	passes	0	0	0

TestReverseTransplant at 8 MiB converged at 64/64 with 0 errors in runs 1 and 2. In run 3 it converged at 47/64 with 17 errors. At 128 MiB it reached 64/64 with 0 errors in all three runs.

The load clients (driveKeepAlive) stop at their first write or read error, with a 2 s read deadline, and never redial. Each error is one connection gone for good.

• At flap 3, the number of missing connections equals the error count exactly, in all three 8 MiB runs.
• Between that measurement and the end of the test, another 37–46 clients fail. Those connections were still counted as active on one engine. Their requests were not answered within 2 s.

Every run in both arms hit oscillation detected, locking switches right before flap 3. The 128 MiB runs still pass, so the lock alone does not explain the loss.

Not established: what the errors are. The test counts them and discards the values, so they could be resets, EOFs or deadline expiries. Also unknown: why a one-worker io_uring strands transplanted connections.

First steps:

1. Record the error strings; the ramp rig's errSamples already does this.
2. Record both engines' transplant counters.
3. Reproduce at 8 MiB.

The CI shape is exactly where this shows, and it is where ./adaptive/... would run under #641.

[FumingPower3925]

TestReverseTransplant, repeated in one process

I ran -count=10 in a single test process, with memlock at 8 MiB (io_uring capped to one worker). Base is celeris 468ce53, and the arms ran in the order main, #639 fix, main, fix.

arm	failures	converged in the failing iterations (of 64)
main 1	6/10	51, 58, 60, 60, 61, 61
fix 1	7/10	43, 53, 54, 54, 54, 55, 56
main 2	6/10	45, 45, 47, 49, 57, 61
fix 2	4/10	49, 55, 61, 61

Totals: 12/20 on main, 11/20 on the fix. So #639's change does not move it.

In separate package runs this test failed 1 time in 3. Repeated inside one process it fails more often than it passes. Whatever strands the connections gets worse when an earlier engine has already run in the same process. That fits alongside #658, where descriptors from earlier iterations are still open when the next iteration starts.

[FumingPower3925]

With celeris#662 fixed, what remains in TestRampH1Sync/Async is all yours

An attempt to lift the -skip quarantine on TestRampH1Sync and TestRampH1Async (they were
quarantined naming #662) was made on PR #674 and reverted, because CI refused it twice. The
useful part is that #662's contribution is now subtracted, so what is left isolates this issue.

CI run 35082406457, GitHub-hosted, memlock unlimited, with the #662 fix applied:

attempt 1: --- FAIL: TestRampH1Sync (65.02s)   --- FAIL: TestRampH1Async (56.96s)   [91 PASS / 2 FAIL / 0 SKIP]
attempt 2: --- FAIL: TestRampH1Sync (57.07s)   --- FAIL: TestRampH1Async (56.86s)   [91 PASS / 2 FAIL / 0 SKIP]

Attempt 2 is a re-run of identical bytes, and in both runs those two were the only failures —
so this is not the #670/#673 flake family, which are different tests.

What #662's fix did clear

• Phase 0 went from io_uring 1363 of 2048, 685 errors (the figure in epoll: PauseAccept silently drops connections already waiting in the accept queue — their requests get EOF (8/8, deterministic), and a promotion on a GitHub runner loses ~1/3 of 2048 connections #662's own body) to
  2048/2048, err=0, in both the sync and async tests.
• Job-wide error census, honouring each sample's (Nx) multiplier: READ-resets = 0 — the
  epoll: PauseAccept silently drops connections already waiting in the accept queue — their requests get EOF (8/8, deterministic), and a promotion on a GitHub runner loses ~1/3 of 2048 connections #662 signature, eliminated. EOF = 0. The async test recorded err=0 across 938,818 requests.
• Remaining and not epoll: PauseAccept silently drops connections already waiting in the accept queue — their requests get EOF (8/8, deterministic), and a promotion on a GitHub runner loses ~1/3 of 2048 connections #662's: 52 dial-time resets (a handshake still in progress when the listen
  socket closes — acceptQueuedOnPause documents this as inherent and uncovered) and 199 read
  i/o timeouts.

What is left, and it is this issue

• async phase 4: io_uring=1316 of 2048 after 20 s, with err=0. Nothing is being dropped —
  no resets, no EOFs, no errors of any kind. The transplant simply does not finish inside the
  window. That is the "forced switches leave keep-alive connections behind" behaviour this issue
  describes, now visible with every competing cause removed.
• sync phases 3 and 5: connections stranded on the DOWN-switch — epoll=0 io_uring=0 where
  roughly epoll=1 is expected.

Practical note for whoever fixes this

Do not trust a local pass. These two tests pass locally at 128 MiB memlock (39.28 s / 39.48 s
measured on the #674 branch) and they pass locally on main too — exactly as this issue's own
body warned about a faster local container. Only the GitHub runner discriminates. ci.yml now
carries a comment recording that measurement, and the quarantine entry names #657 rather than
#662, so the next person to try the lift starts from the right place.

[FumingPower3925]

Correction: withdraw the error-census claim in my previous comment

My comment above cited a "job-wide error census" with READ-resets = 0 and called that the
#662 signature eliminated. That claim is not supported and I am withdrawing it. I relayed the
number without checking the instrument.

The census comes from recordErr, and at main 985a386 it reads:

// adaptive/ramp_transplant_test.go:66-72
func recordErr(s string) {
	errSampleMu.Lock()
	if len(errSamples) < 200 {
		errSamples[s]++
	}
	errSampleMu.Unlock()
}

The guard is on map size, not on key novelty — so once 200 distinct keys are held, even a
repeat of a key already in the map stops being counted. And the keys are built as
"read: " + rerr.Error() (:194, with :174 and :187 the same shape), whose text carries the
client's ephemeral port, making keys effectively unique per connection. The map therefore
saturates after roughly 200 connections and silently discards everything afterwards.

So "READ-resets = 0" means "no read-reset appeared among the first ~200 distinct error keys
recorded", not "no read-resets occurred". In a run of that size, most errors were never
classified. The zero is consistent with the resets being gone, but it does not establish it.

What survives, and what does not

Still good, because it does not depend on the census:

• The two tests fail reproducibly: run 35082406457 attempts 1 and 2, identical bytes, --- FAIL: TestRampH1Sync and --- FAIL: TestRampH1Async, and in both runs those two were the only
  failures out of 93. That is not the adaptive: TestRampAutoMixedH1H2 flakes on the phase-0 migration threshold (949 of the 1024 required) behind a flat 2500 ms sleep #670/adaptive: TestRouteAdaptive_NoReopenerWhenEngineCreationFails flakes on a shared runner (2 ms polls plus a runtime.Stack goroutine census) #673 flake family.
• async phase 4: io_uring=1316 of 2048 after 20 s — read straight from the phase line, not
  from the census. The phase lines also report err=0 for that phase, which is the engine's own
  per-phase counter rather than the saturating map.
• Sync phases 3 and 5 stranding connections on the down-switch.

Withdrawn pending a non-saturating instrument: the claim that read-resets are eliminated, and the
specific figures 52 dial-resets / 199 read timeouts / 0 EOF, all of which came from the same
truncated map.

What would fix the instrument

Count every error unconditionally and cap only the distinct samples retained for printing —
i.e. keep a total counter and a bounded exemplar set, rather than letting the exemplar bound
silence the counter. That is worth doing before this issue's fix is judged, since the same map is
how anyone would measure it.

Apologies for the noise; better to correct it here than to leave a number standing that the next
person would build on.

[FumingPower3925]

Diagnosed: two separate defects, and the second silently loses requests on main today

Instrumented, pre-registered reproductions on main 985a386 and on the #674 branch, with every
hand-off attempt counted by reason and every lost request joined to the operation that lost it.
Predictions were hashed before the first run; 444 evidence files are hashed and verified.

Face 1 — after a promotion, busy async connections almost never move

engine/epoll examines a connection for transplant only right after handling that
connection's own epoll event (loop.go:587-589, the only call site). For a keep-alive client
that event is its next request — by which time drainRead has appended the request and signalled
the dispatch goroutine, so the gate at epoll/transplant.go:88-90 sees it busy and returns.

• The deciding refusal, parked but not idle, accounts for 100% of the stuck pool at every
  non-empty verdict. The chance of a move per examination is 1.5e-4 to 4.7e-4.
• A connection that is idle at the switch is never examined at all: nothing sweeps, nothing
  retries, and StartTransplant wakes nothing. "The 1.2 s flap window" in this issue's title is
  test code, not engine code — the engine has no deadline.
• A prototype that lets the dispatch goroutine request its own hand-off at its park boundary, plus
  a sweep at StartTransplant, drains the epoll residue to 0 within ~1 s.

Face 2 — a hand-off can steal ANOTHER connection's request (fd lifetime)

When io_uring moves a sync connection to epoll (a revert), tryTransplant runs at the udSend CQE,
duplicates the fd, closes the original, and hands the duplicate over. But the connection's next
RECV was linked behind that SEND and can still be issued or completed afterwards. It resolves the
fd number — which the next transplant's dup has by then handed to a different connection. The
RECV takes that connection's request; its CQE lands on an empty slot in staleConnCQE, and the
bytes are discarded with no counter.

• 1,707 of 1,707 client errors joined, one to one, to such stale RECV CQEs, in 48 of 48
  runs.
• 1,181 of 1,181 thefts after the first harvest took the request of the connection that by
  then held the recycled fd number — not the moved connection's own.
• The hand-off ledger (TransplantDetached == TransplantAdopted) balanced in 166 of 166 runs,
  which is exactly why the celeris#624 counters could never see this.
• A second route: after a revert the worker can park with the cancels it just queued still
  unsubmitted (2 pending SQEs per final-iteration transplant, 32/32 runs). Those RECVs come back at
  the next promotion — the flap-3 failure.
• The unattributed keep-alive stall under switches on main is this bug: 50 of 50 failed
  keep-alives joined by port to a stolen RECV. Main stalls in 5 of 30 rounds; the fix(engine): keep serving on a paused listener for 1.5 s with TCP_DEFER_ACCEPT cleared, so a switch or PauseAccept no longer resets clients that had not yet sent a request (celeris#662, #675) #674 branch in 7
  of 10 (Fisher p = 0.0033) because its lingering worker keeps entering the ring.

So "one-worker io_uring strands sync conns until their requests fail" is real, but the mechanism
is an fd-lifetime bug with cross-connection request loss, not a placement problem. Moving
connections sooner or sweeping more cannot fix it: no connection may be handed off while any read
can still resolve its fd. A prototype that holds the next RECV at the hand-off boundary and reaps
an already-armed one first lost 0 in 6 of 6 flap runs, against 52-57 per run without it.

How this interacts with #674 (celeris#662)

The #662 fix keeps the outgoing listener accepting for 1.5 s so no handshake-complete client is
reset. That is correct, but it amplifies face 1: the outgoing engine keeps taking new connections,
and busy async ones never leave. TestRampH1Async at 4 io_uring workers: main fails 0/6, the #662
branch 6/6, the branch with the linger forced to 0 fails 0/6 (p = 0.0022). Routing connections the
outgoing engine accepts straight to the incoming engine at accept time takes the residue back to 0.

Next

Fix designs are being judged now against these measured mechanisms. Per the maintainer, #657 is
fixed completely before #674 ships — the aim for this release is stability.