epoll: a timeout reap parks the whole loop thread on cs.detachMu for as long as an async handler runs (the epoll twin of the closed celeris#593)

[FumingPower3925]

Split out of celeris#654 / PR #665, which found it while reading the same path. Not measured on epoll yet — read at df1269c.

Mechanism

With AsyncHandlers: true, runAsyncHandler holds cs.detachMu across the entire ProcessH1 call, i.e. for the whole user handler. Two loop-thread sites then block on that mutex:

1. checkTimeouts (engine/epoll/loop.go:2575) → closeConn (:2651) → cs.detachMu.Lock() (:2673). Reached whenever a deadline expires on a conn with a handler in flight — and nothing refreshes cs.lastActivity while a handler runs, so with the default ReadTimeout of 60 s any handler that outlives its read deadline parks the loop thread until it returns.
2. The dirty-flush pass (:605) takes cs.detachMu for every dirty conn on every iteration, so a slow handler holding the lock on a conn with pending bytes stalls the loop there too.

While the loop thread is parked, every other connection on that loop waits: no epoll_wait, no accepts, no flushes.

Relation to the closed io_uring twin

celeris#593 is the same family on io_uring and is CLOSED, fixed in bb4d774 ("fix(iouring): TryLock the timeout sweep's h1State snapshot so a slow async handler no longer pins the worker", PR #604): the sweep now goes through snapshotH1Deadlines, which uses TryLock and skips the conn for one pass rather than blocking the LockOSThread'd worker.

Two differences worth stating, because they are why this was not just the same bug:

• io_uring: checkTimeouts locks cs.detachMu for every live conn while runAsyncHandler holds it across ProcessH1, so a slow async handler pins the whole worker #593's io_uring sweep locked every live conn on every sweep, so any slow handler stalled the worker. The epoll sweep reads h1State without the lock — that contrast is in io_uring: checkTimeouts locks cs.detachMu for every live conn while runAsyncHandler holds it across ProcessH1, so a slow async handler pins the whole worker #593's own body — so on epoll only a conn that is actually being reaped blocks the loop. Narrower trigger, same consequence once it fires.
• The remedy fix(iouring): TryLock the timeout sweep's h1State snapshot so a slow async handler no longer pins the worker #604 shipped (TryLock + skip for one pass) maps directly onto both epoll sites, which is why the framing in fix(epoll): re-check conn ownership after closeConn waits on detachMu, so a hijacked fd is not torn down twice (celeris#654) #665 that fixing this "changes timeout semantics, so it must not ride along" is too strong: there is a reviewed precedent for the shape of the fix.

It does still change when a timeout closes a conn — from "at the deadline" to "at or after the deadline, once the handler yields" — and on epoll it also has to decide whether to defer the close to the dispatch goroutine's detachQueue hand-off (when cs.asyncRun && !cs.asyncParked) rather than skip a pass. That is why it is its own issue rather than a rider on #665.

How to measure it

The celeris#589 rig (negctrl_async, its ANOMALY589 lines) reported 0 stalled samples on epoll in 40/40 runs — because that rig's 300 ms handler never outlives a timeout, so site 1 is never entered. To exercise it, set ReadTimeout below the handler duration (e.g. ReadTimeout: 100ms with a 500 ms async handler), keep a second fast keep-alive conn on the same loop via SO_REUSEPORT co-location, and sample the fast conn's latency: it should show a stall of roughly (handler duration − time already elapsed) once per reap, and none after a fix.

[FumingPower3925]

PR #698 fixes this. The PR also closes #668, which touches the same code.

Mechanism, confirmed on 9f4d89b. Four loop-thread sites took cs.detachMu with a blocking Lock while runAsyncHandler held it across ProcessH1. The two this issue names are:

• the timeout reap, through closeConn;
• the dirty pass.

Reading found two more:

• drainRead's EOF and read-error branches, reached when a client disconnects mid-handler;
• the EPOLLOUT resume (handleWritable).

closeConn is also reached from EPOLLRDHUP and EPOLLHUP.

Fix. Each site now uses TryLock. When the lock is held and the conn's dispatch goroutine is running a handler (asyncRun && !asyncParked && !asyncDetachUnlocked, read under asyncInMu), the site does not wait:

• closeConn leaves the close to the goroutine. The goroutine exits at its next check and hands the conn back through the detach queue, and the loop closes it with the lock free.
• The dirty pass and the EPOLLOUT resume give the conn up. The goroutine hands it back at its next park, after its own flush.

Every other holder of the lock is bounded (a guarded writeFn), and those are still waited out as before. After a Detach the goroutine counts as such a holder.

Semantics. A timeout close now happens when the handler yields, after its response is flushed. That is what the old blocking close produced too, minus the stall.

Evidence. Everything was regenerated by saved scripts; the evidence dir is evidence/celeris-669-668/lane-20260926/.

Failing-first, on 9f4d89b's code.

• CI x86, run 36242060999: 9 FAIL, 51 data-race reports.
• The issue's rig: an 800 ms async handler, plus a fast keep-alive conn pinned to the same loop by its worker id. The budget for a fast request is 300 ms.
  • trigger=peerclose: a fast request took 751 ms.
  • trigger=reap (ReadTimeout 100 ms): the fast conn was reaped as collateral. The parked loop resumed its sweep and found the fast conn idle.
• arm64 locally: the same result, 768 ms and a reaped fast conn.

Fixed head, e038a40.

• CI x86, run 36245610547, every job log read:
  • The epoll package is ok, with 0 FAIL and 0 data races.
  • The Adaptive job's results match main's name for name: 103 PASS lines over 101 test names, 0 FAIL, 0 SKIP, 0 data races on both. (Corrected in review round 2: an unanchored count had also taken three lines of the workflow's own echoed script as PASS and two as SKIP, giving 106 and 2. Recount: round2/tools/recount_r1_adaptive.sh.)
  • The first attempt's one failure was io_uring's TestDriverHTTPZeroOverhead, which is io_uring: UnregisterConn then Close leaks the driver socket — the fd-keyed ASYNC_CANCEL misses once the caller has closed the fd, so onClose never fires and the peer never sees EOF #691 and on main. The rerun is green.
• arm64 under -race, whole ./engine/epoll package: 120 PASS, 0 FAIL, 6 SKIP. Main gives 95 / 0 / 6. The 101 tests the two runs share have 0 outcome differences, and the 6 skips are the same ones.
• arm64, the lane's tests x10: 340 PASS, 0 FAIL, 0 SKIP.
• Fast-request max over the 20 end-to-end runs: 0.6–10.7 ms, on a host shared with another container. Before the fix it was 751 and 768 ms.

Mutants. 18 of 18 are killed at e038a40. There is one mutant per guard, and each reintroduces one defect through go test -overlay. The EPOLLRDHUP guard's mutant survived at an earlier head, when the end-to-end run was its only witness. The direct test added in e038a40 now kills it.

Not fixed here. The io_uring worker has the same sites. At 9f4d89b:

• worker.go:1393-1394: the dirty pass;
• 2546-2547 and 2595-2596: the recv EOF and error branches;
• 3515: closeConn.

These come from reading and have not been measured. By reading, the recv EOF and error branches are the worst of them: a client FIN during a slow handler would park the worker. That file belongs to #674, so they are not fixed here. They are now tracked as #704.