middleware/websocket: a pause decided on a stale depth snapshot wedges the connection permanently when the handler drains to empty first (distinct from #667)

[FumingPower3925]

Summary

chanReader.requestPause decides to pause from a depth snapshot. If the handler drains the
channel to EMPTY before that pause is applied to the engine, the reader ends up paused with
nothing buffered — and Read only re-evaluates the resume after a successful dequeue, which
can never happen again because the engine is paused and will deliver nothing more. The
connection wedges permanently.

This is a stale-pause / lost-wakeup, and it is distinct from #667. #667 is a reordering
defect: two intents decided in one order and applied in the other. This one fires even when the
order is correct — the pause is simply decided against state that no longer holds by the time it
lands. Fixing #667 does not fix this, and this is present with or without that fix.

Found while A/B-ing the #667 fix (PR #671). Not fixed there, and deliberately not smuggled into
it.

Evidence

A SIGQUIT goroutine dump taken while the benchmark was stalled shows the consumer parked in
Read's blocking select at middleware/websocket/engineread.go:313 while the producer spun with
desired == true — i.e. the engine believes recv is paused, the reader is blocked waiting for
data that the pause guarantees will not arrive, and nothing will re-evaluate.

The rate is not incidental. In the #671 A/B apparatus the escape-hatch counter forced/op
distinguishes a wedged run from a healthy one, and at cap16 the wedge degenerated 9 of 20
samples at 8 MiB memlock and 13 of 20 at 128 MiB on the unfixed arm (edge rate collapsing
~40x, forced/op 0.26-1.0 against ~1e-4 healthy). The benchmark only completes at all because it
carries a documented escape hatch; production has none.

Why it needs its own fix

The resume decision has to be reachable from a state where the buffer is empty and the engine is
paused. Candidate directions, to be judged by measurement rather than by argument:

• Re-evaluate the watermark after applying the pause, under the same lock, and resume
  immediately if the depth has since fallen below lowWater.
• Make Read re-evaluate the resume condition on entry, not only after a successful dequeue, so
  a reader that arrives to an empty buffer while paused can still release the engine.

Either way the failing-first test should assert the wedge directly — park the pause until the
handler has drained to empty, then require that the connection still delivers — and must not
decide anything by timing (see the #667 tests in PR #671 for the shape: a happens-before edge
forced by a channel, with anti-vacuity guards on the callback counts).

Interaction with PR #671

In the like-for-like healthy comparison, forced/op is roughly 3x higher on the #671 fix arm
in both batches. That is the review's open question on that PR: whether taking the callbacks
inside pausedMu makes this wedge more likely by widening the window in which the depth can
change between the decision and its application. It should be settled before or alongside this
fix, not after.

[FumingPower3925]

Measured: this wedge is real, it fires at the production default, and a one-line re-check takes it to zero

Filed from reading; now measured. A direct per-connection experiment (40 rounds/arm, 16
connections/round, 640 connections/arm, round-level analysis) with the benchmark's escape hatch
removed, so one wedge ends one connection the way it would in production. Detection is
structural — engine paused AND buffer empty AND reader progress unchanged across 512
runtime.Gosched() yields — never a wall-clock timeout, and every connection passed a
per-connection anti-vacuity assertion (edges>0 && pauses>0), so no zero anywhere is a clean-zero
artifact.

On main's current behaviour, this is not a rare corner:

capacity	wedges/kiloEdge	connections wedged	median ops to first wedge
cap16	0.0427	489/640 (76.4%)	49,210
cap256 (production default)	0.1573	556/640 (86.9%)	285,066

The candidate fix works. Arm C — one re-check of the watermark under the same lock that
applies the pause, lifting it if it has already gone stale — wedged 0 of 640 at cap16 and 0 of
640 at cap256, at matched edge rates (+1.47% / −0.02% against base), p≈1.9e-16. It also held at
0/320 under -race. Variant sha256 123165ca55de462be383ce45fe0368e1b4767ff661dc3ad3df3f953a18805b06.

Two caveats I want recorded rather than glossed:

1. The detector cannot separate a middleware/websocket: chanReader applies pause/resume to the engine outside the lock that decides them, so a resume can be lost and the connection stops delivering (found by reading, not reproduced) #667 reorder stall from a middleware/websocket: a pause decided on a stale depth snapshot wedges the connection permanently when the handler drains to empty first (distinct from #667) #672 stale-pause stall, and the
   base arm carries middleware/websocket: chanReader applies pause/resume to the engine outside the lock that decides them, so a resume can be lost and the connection stops delivering (found by reading, not reproduced) #667 by construction, so part of base's 0.0427/0.1573 belongs to middleware/websocket: chanReader applies pause/resume to the engine outside the lock that decides them, so a resume can be lost and the connection stops delivering (found by reading, not reproduced) #667. That is
   conservative for the conclusion here (it inflates base, i.e. understates the fix's benefit) but
   it means these numbers are "wedges from either cause", not "middleware/websocket: a pause decided on a stale depth snapshot wedges the connection permanently when the handler drains to empty first (distinct from #667) #672 alone".
2. Arm C has not been priced. It is a correctness measurement, not a performance one. It needs
   its own A/B before it ships — to the standard the fix(websocket): apply pause/resume under the lock that decides them, and lift a stale pause (celeris#667, celeris#672) #671 round established: unfiltered primary,
   slot-matched A/A floors, counterbalanced order, one round = one observation.

Why this is now urgent rather than merely open

PR #671 (the #667 reordering fix) increases this wedge rate ×1.63 at cap256 (0.1573 → 0.2560,
p=8.4e-12, 556/640 → 615/640 connections) and ×13.0 at cap16, corroborated by end-to-end round
failures rising 1.7% → 16.7% (Fisher p=6.0e-05). So #667 cannot land on its own without making
this worse at the default capacity — the two need to ship together, with arm C or an equivalent.

Nothing here has run on amd64 or on the cluster; all of it is Docker/linux/arm64. Cross-arch
confirmation is still owed.

[FumingPower3925]

It showed up in CI: the Unit job of #678, run 35427490195 / job 105855844933, on e9ccc87. That commit is main 5b2e83b plus a test-only change in internal/wakefd, which nothing in middleware/websocket touches.

--- FAIL: TestBackpressureInboundSequenceIntegrity/io_uring (51.48s)
    inbound_sequence_linux_test.go:252: io_uring: handlers still running 20s after the clients finished; shutting down now would truncate them
    inbound_sequence_linux_test.go:407: io_uring: conns=16 framesSent=32000 framesIn=30734 seqGaps=0 parseErr=0 overflowErr=0 protocolErrors=0 clientCloseFail=0 closedOK=15 clientRST=0 serverRST=0 closeTimeout=1 dialFail=0 hsFail=0
    inbound_sequence_linux_test.go:412: io_uring: conn 9: server closed 30s after the client's Close
    inbound_sequence_linux_test.go:433: io_uring: RECVARM bp=256 resumeWhileCancelPending=0 resumeWhileRecvInFlight=0 armDeclined=0 doubleArmed=0 cqeUnaccounted=0 parseErr=0
    inbound_sequence_linux_test.go:469: io_uring conn 9: frame count mismatch: in=734, sent=2000

Shape: GitHub runner, kernel 6.17.0-1022-azure, memlock 8 MiB, so io_uring ran on one worker (workers=1).

One connection stopped delivering at 734 of 2000 frames and its handler never finished. Sequence gaps, parse errors and protocol errors are all 0. Every engine-side recv-arm witness is 0: no resume while a cancel was pending, no double arm, no unaccounted CQE. So the engine did not lose a recv. It stopped being asked for one. That fits a pause stranded on the websocket side, which is this issue or #667. From one log I can't tell which of the two it was.

I'm re-running the job once. This is the WS failure a main CI run can hit, which is one more reason #667 and #672 have to land together (PR #671, after the #657 PRs).