feat(iouring): SEND_ZC exposure counters so the race tier and the bench A/B can prove the branch ran

[FumingPower3925]

Fixes #591. Measured, not argued: the rig that found the defect is the rig that judges the fix, and the negative control was run.

Mechanism

Five validation counters (-tags=validation; no-op Counter stubs in production) added to validation/{assertions,counters,disabled}.go and bumped at the four real sites in engine/iouring/worker.go: IouringSendZCSubmits and IouringSendZCSubmitsDetached inside prepSendSQE's useSendZC arm (detached split keyed on cs.h1State.Detached, nil-guarded for driver conns); IouringSendZCNotifs in handleSend's cqeIsNotif branch; IouringInlineGuardBlockedZC in the initProtocol guarded closure's new else if cs.zcNotifPending arm (read under the held detachMu); IouringZCCompletionWithPendingWrite when a NOTIF lands with len(cs.writeBuf)>0 (also under detachMu). Four always-on engine.EngineMetrics fields — ZCSendsSubmitted, ZCNotifs, InlineBytes, RingBytes — are fed by a new nil-safe zcStats struct (submits/notifs/inlineBytes/ringBytes atomics) hung off Engine.metrics.zc and wired to each Worker in createWorkers, published in iouring Engine.Metrics() and summed in adaptive/engine.go Metrics() alongside DetachedConnections/DetachWindowCloses. No new per-request hot-path cost: the submit and notif adds sit inside branches only a ZC send reaches (every sub-4096 and every linked send falls through to plain SEND untouched); the inline-bytes add is on the detached WS/SSE egress path beside the w.bytesWritten.Add already there; RingBytes accumulates in a worker-local ringBytesBatch in completeSend (plain uint64 +=) and is published with one atomic per event-loop iteration next to bytesWrittenBatch.

Verification

New TestSendZCCountersOnDetachedWrite (middleware/websocket/send_zc_counters_linux_test.go, io_uring only). Container: docker run --rm --cpus 4 --security-opt seccomp=unconfined --ulimit memlock=134217728:134217728 -v :/src -w /src -v /Users/fuming/go/pkg/mod:/go/pkg/mod -v gocache484:/root/.cache/go-build golang:1.27; engine log confirms workers=4 sqpoll=false send_zc=true, tier=high, kernel 7.0.12-linuxkit. Command: go test -tags=validation -race -count=10 -run TestSendZCCountersOnDetachedWrite -v ./middleware/websocket/ — 10/10 PASS, numbers identical in all 10 runs. SMALL phase (200 x 256B, each frame fully drained before the next): ZCSendsSubmitted+=0 ZCNotifs+=0 InlineBytes+=52000 RingBytes+=0; validation submits+=0 detached+=0 notifs+=0 guard_blocked+=0 notif_pending_write+=0. BIG phase (8 pipelined waves x 64 x 65536B = 32 MiB, first wave undrained, client SO_RCVBUF clamped to 8 KiB): ZCSendsSubmitted+=16 ZCNotifs+=16 InlineBytes+=2158592 RingBytes+=31400960; validation submits+=16 detached+=16 notifs+=16 guard_blocked+=0 notif_pending_write+=1. Same test without the tag (count=10): 10/10 PASS, identical engine-metric numbers, all validation deltas 0 (the production stubs), which the test asserts explicitly. Unit level: engine/iouring/send_zc_gate_test.go extended with TestPrepSendSQEWitnessesTrackTheZCArm (5 cases pinning that the witnesses move exactly with the opcode: 4095B->opSEND/0 submits, 4096B attached->opSENDZC/1 submit/0 detached, 4096B and 16384B detached->opSENDZC/1 submit/1 detached), TestPrepSendSQELinkedCountsNoSubmit, TestZCStatsNilSafe; validation-counter expectations are scaled by a build-tag const (zc_witness_tag_on/off_test.go) so the same tests run in both modes.

Negative control

ZC policy forced off via the CELERIS_IOURING_SEND_ZC knob: docker run ... -e CELERIS_IOURING_SEND_ZC=off ... go test -tags=validation -race -count=10 -run TestSendZCCountersOnDetachedWrite -v ./middleware/websocket/ — 10/10 PASS, engine log shows send_zc=false in both the "engine selected" and "engine listening" lines (20 occurrences over 10 runs). BIG phase: ZCSendsSubmitted+=0 ZCNotifs+=0, validation submits/detached/notifs all 0, while InlineBytes+=2158592 and RingBytes+=31400960 are byte-for-byte identical to the ZC-on arm — the same egress went over the same fabric with zero zero-copy, so the 16/16 above is attributable to the ZC branch and not to traffic volume. Second control, for the one counter the rig does not move: IouringInlineGuardBlockedZC stays 0 on this host because the kernel reports IORING_NOTIF_USAGE_ZC_COPIED on loopback (copy fallback), so the NOTIF follows its send CQE inside the same drain pass and a dispatch-goroutine write never lands in the window. A throwaway build with time.Sleep(2*time.Millisecond) added while detachMu is held after cs.zcNotifPending is set moved it to guard_blocked+=69 (and notif_pending_write 1 -> 14) in one of 5 runs, proving the site is live and not dead code; that sleep was reverted and is not in the commit. Third control, for the failing suite: see regression_risk.

Regression risk

Low. Production builds compile against the validation no-op stubs, so all five validation adds vanish; the always-on adds are confined to ZC-only branches, the detached egress path, and a per-iteration batch flush. adaptive sums the four new fields additively, which is correct because they are io_uring-only and cumulative. One measured caveat: under -tags=validation -race the middleware/websocket suite fails TestBackpressureInboundSequenceIntegrity/io_uring/multishot_recv. I ran the negative control for this — go test -tags=validation -race -count=5 -run TestBackpressureInboundSequenceIntegrity on an unmodified origin/main tree extracted with git archive origin/main into a separate directory, same container and flags: 1 FAIL / 4 PASS. Same command on this branch: 1 FAIL / 4 PASS. Identical rate, and the failure signature is the same (SEND_ZC returned ENOMEM (RLIMIT_MEMLOCK), falling back to regular SEND on all workers, then echo-write-on-closed-connection and frame-count mismatches). Pre-existing and unrelated to this change.

Not verified

1. The probatorium half of celeris#591 is NOT done and is outside this repo: publishing the counters through the refapp /debug/vars (probatorium debugvars) so the property loop can declare an I-ENG-IOURING-ZC predicate, and adding -tags=validation to probatorium's raceCompileGoBinary so the race tier's refapps carry them. The celeris side is complete and the values are readable via Server.EngineInfo().Metrics and validation.Snapshot(); the issue should stay open for the probatorium work. 2) IouringInlineGuardBlockedZC was never observed non-zero on an unmodified build — only under the deliberately widened window (see negative_control). On a host with true zero-copy (real NIC, no IORING_NOTIF_USAGE_ZC_COPIED) the NOTIF is delayed until DMA completes and the window should be far wider; that has not been measured. 3) No arm64 run and no cluster run — everything here is x86_64 docker on kernel 7.0.12-linuxkit. 4) No throughput benchmark was run to confirm the "no per-request hot-path cost" claim; it rests on inspection (the plain-SEND path gains zero instructions, completeSend gains one non-atomic uint64 add, the extra atomic is one per event-loop iteration). 5) Only engine/iouring and middleware/websocket suites were run; the rest of the repo was built, vetted and linted but its tests were not executed. 6) The adaptive aggregation of the four new fields is compile- and lint-verified but not exercised by a test that actually switches engines.

Adversarial review

Skeptic 1 — refuted: False, mergeable: True

The named mechanism is addressed at the cited lines and no invariant is broken. prepSendSQE (engine/iouring/worker.go:4221) bumps the submit witnesses strictly inside the useSendZC arm, so the plain-SEND and linked-send hot paths gain zero instructions; handleSend's cqeIsNotif branch (:2562) bumps notifs before taking detachMu and reads cs.writeBuf under detachMu in both the mutex and nil-mutex branches; the initProtocol guarded closure's new else if cs.zcNotifPending executes under the already-held detachMu and does not alter control flow (the fall-through to mu.Unlock and the detach-queue enqueue is unchanged), so the inline-egress exclusivity rule stated in that comment (!sending && !zcNotifPending && sendBuf/bodyBuf empty) is intact. The ring-byte add in completeSend sits AFTER the sent < 0 guard, so it cannot take the negative-zcSentBytes value that would corrupt an unsigned add, and it mirrors the pre-existing bytesWrittenBatch exactly (same accumulation site, same per-iteration flush, same absence of an end-of-loop flush). git grep confirms only two bytesWritten sites in engine/iouring (the detached inline unix.Write and the batched completeSend), so the InlineBytes/RingBytes complement claim is true rather than asserted. Production cost is real-zero: validation.Counter is struct{} with empty Add/Load in validation/disabled.go, and engine/iouring already imported validation via validation_check.go, so no new production dependency or binary weight. The rig demonstrably flipped (0 submits on the drained 256B phase, 16 submits/16 notifs on the congested 64KiB phase) and the negative control was run with numbers that actually discriminate: with CELERIS_IOURING_SEND_ZC=off the same burst produced byte-identical InlineBytes (2158592) and RingBytes (31400960) with submits/notifs at 0, which rules out traffic volume as the explanation for the 16/16. The e2e test asserts != 0 rather than exact counts, so it is not brittle to host send-buffer behaviour, and the unit tests pin that the witnesses move exactly with the opcode (4095B -> opSEND/0, 4096B -> opSENDZC/1) plus a linked-send-counts-nothing case and a nil-zcStats case. The commit contains 13 source/test files only: no scratchpad, no logs, no debug prints, and the throwaway time.Sleep used to exercise the guard counter is genuinely absent from the diff. The build-tag const pair is the standard linux&&validation / linux&&!validation shape, not a stray tag. tier.go's two other ZC arms are uninstrumented but provably dead — PrepareSend( has no callers anywhere in the repo — so this is not an undercount today.

Skeptic 2 — refuted: False, mergeable: True

I could not refute the branch. (1) The change is at the cited sites: all five validation counters land in engine/iouring/worker.go at :4221-4231 (inside the useSendZC arm of prepSendSQE, Detached split nil-guarded on h1State), :2563-2588 (the cqeIsNotif branch, with the writeBuf read taken under detachMu), and :1760-1770 (a new else if cs.zcNotifPending on the inline-egress guard, under the already-held mu); four always-on EngineMetrics fields are wired through engine/iouring/engine.go and summed additively in adaptive/engine.go. (2) The rig genuinely flipped and the negative control is present with numbers: SMALL 0 submits -> BIG 16 submits/16 notifs, and CELERIS_IOURING_SEND_ZC=off gives 0/0 with RingBytes byte-identical, which is a true single-variable flip. The reported byte totals are internally consistent, which corroborates that they were observed rather than asserted: SMALL InlineBytes 52000 = 200x(256+4), BIG 2158592+31400960 = 33559552 = 512x(65536+10). InlineBytes+RingBytes == BytesWritten exactly, because git grep shows only two byte-count sites in the engine (worker.go:1744 inline, :2772 completion) and both are instrumented. (3) No double count and no overflow: the cqeHasMore branch returns without calling completeSend so a ZC send reaches completeSend exactly once via the NOTIF, and ringBytesBatch += uint64(sent) sits after the sent < 0 early return so the -ENOMEM path cannot wrap it. (4) Hot-path claim holds structurally: plain SEND and linked sends gain zero instructions, completeSend gains one non-atomic uint64 add flushed with one atomic per event-loop iteration next to bytesWrittenBatch. (5) Nothing improper is committed: 13 files, no scratchpad/logs/debug prints, the only time.Sleep calls in the diff are the two documented settle waits in the test, and the deliberately widened detachMu window used as the guard-counter control is genuinely absent from worker.go. Build tags are correctly paired (linux && validation / linux && !validation) in both packages so neither mode redeclares, and validation/disabled.go Counter is a stateless struct with func (Counter) Add(uint64) {}, so production really is a no-op — consistent with the test asserting all validation deltas are 0 without the tag. Residual, non-blocking observations: the else if cs.zcNotifPending attributes a decline to ZC even when cs.fixedFile or an empty writeBuf is also blocking (near-theoretical with fixed files default-off, but it makes IouringInlineGuardBlockedZC an over-count rather than a precise witness); BIG shows only 16 ZC submits against 31.4 MB of ring bytes and the report never says whether w.sendZC flipped false mid-run via the ENOMEM fallback at worker.go:2729 (that exact log line did appear in the same container during the backpressure test), so 16/16 may be a pre-fallback prefix; the new rig never runs in CI because .github/workflows/ci.yml:121 excludes middleware/websocket and line 141 runs only ^TestBackpressure, which removes the memlock flake risk but also means it is not a standing gate; noteInlineBytes doubles the atomics at the inline-write site (w.bytesWritten is already engine-wide shared, not per-worker) and was not benchmarked; and the probatorium half of #591 is untouched so the issue must stay open.