Measure #465: SEND_ZC on/off A/B on the real fabric with per-scenario CPU, both arches

[FumingPower3925]

Filed from the release-readiness sweep of 2026-09-13 under the standing rule that a closure counts only when it is MEASURED. Design produced by a read-only design pass and corrected by two adversarial reviews; the corrections are binding and listed last.

Closure under test

celeris#465 — SEND_ZC default kept ON by PR #489 "pending fabric A/B (~2026-09-08)"; the A/B was never run or recorded.

What the closure claims

Issue CLOSED (milestone v1.6.0) by PR #489 (merged 2026-09-07T17:51Z, merge e468d3b). The PR fixed notifUsageZCCopied (uint32 1<<31, probe.go:74/234), split the loopback probe into a FUNCTIONAL check (copy-fallback on loopback = functional, engine.go:80-95) and kept the production default enabled: resolveSendZCPolicy(functional, "auto"/"") returns functional (probe.go:249-263), i.e. send_zc=true on every host whose kernel accepts the opcode. The PR body, doc.go:9-13 and probe.go:246-247 all say the FINAL default is "pending a measured cluster fabric A/B (SEND vs SEND_ZC on get-json-64k, ws-large-echo, streaming cell, alternating builds, >=3 trials) once the cluster is free (~2026-09-08)". As of 2026-09-13 no benchmark run has toggled CELERIS_IOURING_SEND_ZC (last successful Benchmark Tier = 33256654220, 2026-08-29, pre-PR; every later run cancelled/none), so the perf-affecting default was decided by argument, and the bench that will publish v1.6.0 would run it unmeasured.

What was argued rather than measured

(1) A loopback self-probe can never observe the NIC (kernel always copies on local delivery), NETIF_F_SG gating is unsound, so keep a functional probe + an env knob (CELERIS_IOURING_SEND_ZC=on|off|auto) and decide the default later by A/B — sound. (2) "Blast radius is narrow: unlinked sends >= sendZCMinBytes, essentially get-json-64k and ws-large-echo" — PARTLY WRONG on both facts: get-json-64k is not in the grid (probatorium scenarios/static.go:144-153 cut all 8k/16k/64k rows as wire-bound; the /json-64k handler exists in servers/celeris/server.go:172 but is unscheduled), and the PR's "sendZCMinBytes = 32768" is stale — consts.go:114 is 4096 (commit ce1073c, already at tag v1.5.8). (3) Never stated, but it determines the whole A/B: on the default bench configuration SEND_ZC is STRUCTURALLY UNREACHABLE for HTTP. Multishot recv is opt-in (doc.go:15-16) so w.bufRing == nil, and every H1 response then goes through flushSendLink (worker.go:1906, 2162, 2338) which always emits a plain linked SEND (worker.go:4034-4046, comment "never ZC"); bodies staged by writeBody go WRITEV (worker.go:3922-3950); h2 frames are written inline with unix.Write (worker.go:3441) and only the EAGAIN/partial remainder reaches the ring. The only unlinked ring SEND in the grid is the detached WS/SSE path: inline unix.Write on the dispatch goroutine (worker.go:1567-1575), remainder -> detachQueue -> drainDetachQueue -> flushSend (unlinked) -> prepSendSQE -> useSendZC(sendZC, false, n>=4096) (worker.go:3896-3991). ws-echo/ws-hub/sse-fanout frames are 256 B (< 4096), so ws-large-echo (64 KiB, scenarios/streaming.go:132) is the ONE cell where the default matters. (4) There is already an un-attributed signal exactly there: the five byte-identical published v1.5.8 amd64 runs (docs/results/v1.5.8/{20260716,0722,0729,0805,0829}/x86_64/summary.json; ZC was ON via the buggy probe, threshold 4096) show every io_uring column on ws-large-echo at 95.6-96.8 % SUT CPU (resources.summary.mean_cpu_pct) for 26.6-29.7 K msg/s (1.76-1.95 GB/s one-way), while epoll delivers the same 26.0-29.5 K at 53-58 % and std 34.8-35.4 K at 56-62 %. io_uring spends ~1.7x the CPU per byte of epoll on precisely the ZC-eligible cell; whether SEND_ZC (extra NOTIF CQE per send, buffer held across DMA, possible kernel copy-fallback via skb_copy_ubufs) is that cost is the unanswered question. The 2026-09-04 review already scored the default choice as "unmeasured perf-affecting" (memory v160-bug-plan-review); the 2026-09-13 release sweep flagged #465 as argued-not-measured.

Measurement

Observable

Per cell, from the harness artefacts already produced for every bench cell: (a) saturation_mode_rps of ws-large-echo (msg/s; 1 msg = one 64 KiB frame out + echo back, loadgen ws.go:33/138-153), (b) resources.summary.mean_cpu_pct = SUT host-wide mpstat 'all'-row busy % at 1 Hz (report/resources.go:100-143, sampler = run_bench_cell.yml:384-389), (c) derived DECISION METRIC cpu_per_GB = mean_cpu_pct / (rps * 65536 B) i.e. SUT CPU per delivered byte, (d) rated_mode_p99_at_target_rps / latency_at_slo for ws-large-echo as secondary. Arm identity is proven per cell by the engine log line 'io_uring engine selected ... send_zc=true|false' (engine.go:137-145, slog.Default -> stderr -> {{cell_dir}}/server.log, run_bench_cell.yml:271). Controls in the SAME dispatch: ws-echo (256 B < sendZCMinBytes, identical detached write path, useSendZC always false) and get-json (linked SEND only). Optional kernel-side observables (root on the SUT, outside the harness): count of io_uring submissions with opcode 47 (IORING_OP_SEND_ZC, consts.go:45) vs 26 (SEND) during the cell, and kprobe:skb_copy_ubufs hit count (= number of ZC sends the kernel COPIED because the device/bond cannot take user frags).

Where it runs

probatorium Benchmark Tier on the cluster (benchmark-tier.yml), profile=fast (35 s/10 s, saturation-only, BENCH_PUBLISH forced 0 at benchmark-tier.yml:154 so nothing reaches docs), competitors=celeris-iouring-h1-async, deploy_competitors=celeris, ONE ARCH PER DISPATCH: target=msa2-server (amd64, 2x10G LACP bond, kernel 7.0.0-30, memlock unlimited so the ENOMEM fallback at worker.go:2434/2521 cannot silently disable ZC mid-run) first, then target=msr1 (arm64, RTL8127/r8127 DKMS NIC) as a second series. NOT target=both: the 20260829 parallel-arch run depressed ws-large-echo ~8 % on EVERY amd64 column (27.6-29.7 K -> 26.0-26.9 K) because this cell runs near the loadgen bond's line rate and the two arch passes share msa2-client. Bench SUT is pinned to celeris main ededb6c (servers/celeris/go.mod:6), i.e. the v1.6.0 candidate.

Procedure

1. Land the two small harness passthroughs in code_changes_needed (env + cells), on main, before dispatching. 2. Wait for the cluster group to drain (24 h soak 34736980002 ends ~2026-09-14T04:15Z; nightly 34744788978 is pending behind it; matrix-tier-cluster serializes). 3. Dispatch SIX runs on amd64 in the order ON, OFF, OFF, ON, ON, OFF (A B B A A B cancels monotone drift), each: gh workflow run benchmark-tier.yml --repo goceleris/probatorium -f profile=fast -f target=msa2-server -f competitors=celeris-iouring-h1-async -f deploy_competitors=celeris -f publish=false -f cells='ws-large-echo/,ws-echo/,get-json/' -f sut_env='CELERIS_IOURING_SEND_ZC=on' (or =off). Use the explicit 'on', not 'auto', so the ON arm is not hostage to the functional probe. 4. For each run read benchmark-tier-results-<run_id> (results.json + 00-celeris-iouring-h1-async/server.log + run0//.json + cpu.log): assert server.log contains send_zc=true for ON and send_zc=false for OFF (a mismatch voids the run); extract rps, mean_cpu_pct, p99 for the 3 cells. 5. Form the three adjacent pairs (1-2, 3-4, 5-6), compute Delta = OFF - ON for cpu_per_GB and rps on ws-large-echo, and the same deltas on ws-echo and get-json. 6. Decision rule: WIN for ZC if cpu_per_GB(ON) < cpu_per_GB(OFF) by >= 5 % in 3/3 pairs AND rps(ON) >= rps(OFF) - 3 %; LOSS if cpu_per_GB(ON) > cpu_per_GB(OFF) by >= 5 % in 3/3 pairs OR rps(ON) < rps(OFF) by >= 3 % in 3/3 pairs; NOISE if neither (or the controls move by as much as the treatment cell, which flags run-level drift). If the three pairs disagree in sign, add one more pair (2 dispatches) before calling it noise. 7. Optional but decisive for interpretation, during ONE ON-arm ws-large-echo cell on msa2-server as root: bpftrace -e 'tracepoint:io_uring:io_uring_submit_req /args->opcode==47/ {@zc=count()} tracepoint:io_uring:io_uring_submit_req /args->opcode==26/ {@send=count()} kprobe:skb_copy_ubufs {@copied=count()}' for 30 s; plus ethtool -k bond0 and each slave | grep scatter-gather. 8. Repeat steps 3-6 with target=msr1 for arm64 (its ws-large-echo is CPU-bound at 11 K msg/s / 95.6 % CPU, so rps is the primary metric there). 9. If LOSS: change resolveSendZCPolicy 'auto' to return false (probe.go:258-259), re-run one confirming pair, repin probatorium, THEN the release bench, so the flip lands at the version boundary the issue itself asked for. If WIN or NOISE: record the numbers in io_uring: probeSendZC can never detect copy-fallback — SEND_ZC enabled on every host regardless of NIC #465 and leave the default ON.

Expected if the claim is TRUE

Claim under test = 'SEND_ZC on the real fabric is a win, so keeping the default ON is right'. Expected: in every one of the 3 amd64 pairs the ON arm shows lower cpu_per_GB on ws-large-echo by >= 5 % (e.g. mean_cpu_pct drops from the ~96 % plateau of the v1.5.8 data toward epoll's 53-58 % at the same 27-29.7 K msg/s, or rps rises above ~29.7 K at the same CPU); bpftrace shows opcode-47 submissions in the thousands per second during the cell with skb_copy_ubufs ~0 (true zero-copy on the bond); the controls ws-echo and get-json differ between arms by <= 1.6 % rps and <= 0.6 CPU points. On arm64 ON rps >= OFF rps + 3 %.

Expected if the claim is FALSE

Two distinct falsifications. LOSS: the OFF arm has lower cpu_per_GB by >= 5 % and/or higher rps by >= 3 % in 3/3 pairs — most plausibly OFF brings the io_uring column's ~96 % CPU down toward epoll's ~55-58 % at equal throughput (the NOTIF-per-send + buffer-hold cost, or a copy-fallback where bpftrace shows skb_copy_ubufs count ~= opcode-47 count, meaning the kernel copies anyway and ZC only adds a second CQE). NOISE: |Delta cpu_per_GB| < 5 % and |Delta rps| < 3 % in every pair with inconsistent sign, controls moving as much as the treatment; then the default is immaterial for the published grid and #465's residual question is closed as 'no measurable effect on this fabric', which is also a real result. Either way the ~1.7x CPU-per-byte gap between io_uring and epoll on this cell becomes attributable (to ZC, or explicitly NOT to ZC -> a separate issue on the detached-write funnel).

Negative controls

Three built into every dispatch: (1) ws-echo — same detached inline-write/detachQueue path, 256 B frames < 4096 so useSendZC is false (send_zc_gate_test.go:24-25 pins this) — must show no arm difference; (2) get-json — linked SEND only, ZC unreachable — must show no arm difference; if either control moves as much as ws-large-echo the delta is run-level drift, not ZC. (3) Arm verification: server.log must read send_zc=false in the OFF arm; the bpftrace opcode-47 count must be 0 in the OFF arm and > 0 in the ON arm (proves the env reached a root-launched SUT and that the cell actually exercises the branch — without this a 'noise' verdict could just mean the toggle never applied or the branch never fired). Existing-data sanity anchor: the ON arm at profile=fast should reproduce the v1.5.8 fast-run band (28.8-29.7 K msg/s, 96.3-96.8 % CPU) within the v1.6.0 engine changes; a large departure from that band in BOTH arms indicates something else moved (#572/#574/#525 etc.), to be reported, not folded into the ZC verdict.

Cost

Per dispatch with the cells passthrough (3 cells x 1 column x 1 arch): setup runner ~5-10 min + mage Deploy (Go build of servers/celeris only) ~5-10 min + 3 cells x ~2 min (budget model 10+35+5+12 = 62 s/cell, ~2x observed under-count) + merge/upload/cleanup/teardown ~10-15 min = ~35-45 min, i.e. ~4-4.5 h cluster wall-clock for the six amd64 dispatches; the same again for arm64 (msr1) = ~9 h total for both arches, versus ~74 h for the release bench. WITHOUT the cells passthrough (column-scoped only via the existing competitors input) each dispatch runs the column's ~29 capability-gated scenarios (~1 h at fast + overhead) = ~1.5 h/dispatch, ~9 h per arch, ~18 h both — the controls then come for free. Engineering: ~1-2 h for the passthroughs + a unit test. Statistical cost: three pairs give a 3/3 sign requirement (one-sided p = 0.125 on sign alone) — the decision rests on the magnitude thresholds being 3x the measured same-window spread (rps <= 1.6 %, CPU <= 0.6 pt over five byte-identical v1.5.8 runs), not on the sign test.

Not measurable, and why

(1) Whether the NIC/bond actually zero-copies cannot be observed from celeris: production SQEs never set IORING_SEND_ZC_REPORT_USAGE (sqe.go:175-186 sets only opcode/fd/buf), so NOTIF CQEs carry no ZC_COPIED bit; only the out-of-harness kernel probe (skb_copy_ubufs / tracepoint opcode filter, root on the SUT) can tell copy-fallback from true ZC, and the harness cannot record it. (2) The fraction of ws-large-echo bytes that go through SEND_ZC vs the inline unix.Write fast path (worker.go:1567) is not counted anywhere — EngineMetrics (engine.go:335-351) has bytesWritten but no ZC counter — so a NOISE verdict could mean 'ZC rarely fires' rather than 'ZC is free'; the bpftrace step or a small celeris counter is the only way to separate these. (3) HTTP/1.1 and h2 large responses are structurally ZC-unreachable in the bench (linked SEND, WRITEV bodies, inline h2 writes), so this A/B says NOTHING about ZC for HTTP; the only configuration where every >= 4 KiB send would be ZC is multishot recv (CELERIS_IOURING_MULTISHOT_RECV=1, bufRing != nil -> flushSend unlinked), which the bench does not run — a second A/B dimension deliberately out of scope. (4) Fixed-file mode (off by default, celeris#541/#553) and the adaptive columns' post-promotion behaviour are not covered. (5) The CI memlock-limited ENOMEM fallback path (worker.go:2434, 2521; 8 MB memlock on GitHub runners) is not exercised on the cluster (memlock unlimited). (6) Cross-dispatch rps is confounded by window length (35 s vs 90 s) and by any parallel arch pass, so only within-pair comparisons at identical profile/target are valid; a single ON run compared against the historical v1.5.8 numbers is NOT a measurement. (7) mean_cpu_pct averages the whole cell-guard window including idle edges (dilutes both arms equally) and is host-wide, so it cannot attribute CPU to the SUT process vs kernel softirq — a perf split is out of harness.

Code changes needed

Required (probatorium, harness only; no celeris change needed to run the A/B): (1) SUT env passthrough — benchmark-tier.yml: add workflow_dispatch input sut_env (KEY=VALUE, default "") -> job env BENCH_SUT_ENV; mage_bench.go Bench() buildArgs (mage_bench.go:451-483): when BENCH_SUT_ENV is set, append --extra-vars with a JSON dict {"bench_sut_env": {KEY: VALUE}}; ansible/tasks/run_bench_cell.yml 'start competitor server in background' (lines 199-272): turn the hard-coded environment: mapping (207-235) into environment: "{{ sut_base_env | combine(bench_sut_env | default({})) }}" and also log the merged env into server.log (or echo it in the launch shell) so the arm is recorded next to the send_zc= line. (2) Cell-scope passthrough — benchmark-tier.yml: input cells (default "") -> env BENCH_CELLS; mage_tier.go setBenchEnvFromProfile (lines 218-244): honour a pre-set BENCH_CELLS the same way it already honours BENCH_DURATION/BENCH_WARMUP instead of unconditionally os.Setenv("BENCH_CELLS", budget.CellsGlob(p)); FitWithin still projects p.Cells=813 so it passes trivially — acceptable for a scoped run, note it in the log. Unit tests: a table test for the merged extra-vars and one for the BENCH_CELLS precedence. Optional but recommended for interpretability: (3) celeris — two monotonic counters on Worker (zcSendsSubmitted in prepSendSQE's ZC branch worker.go:3973-3977, zcNotifs in handleSend worker.go:2370) surfaced through EngineMetrics (engine.go:335-351) and thus /debug/vars, which the bench observer already scrapes for celeris cells (run_bench_cell.yml:421) — this makes the 'did ZC actually fire' question harness-recordable instead of bpftrace-only; small, additive, covered by extending send_zc_gate_test.go. (4) Optional: record bench_sut_env in the merged Document's environment (mage_bench.go:1788-1806) so an arm is identifiable from results.json alone. If the verdict is LOSS: celeris probe.go:258-259 'auto' -> return false (default OFF; explicit on still honoured), doc.go:9-13 wording, and a repin — that is the follow-up, not part of the measurement.

Corrections from the adversarial review (binding)

Review 1: refuted

Why: Two independent defects each make the proposed A/B unable to separate "SEND_ZC is a win/loss" from "SEND_ZC never mattered", and the design's decision rule then converts the guaranteed NOISE outcome into a verdict.

1. The decision metric does not exist per cell. resources.summary.mean_cpu_pct is the mean of ONE mpstat run over the WHOLE competitor column: in the cluster harness a "cell" is (run_index, competitor_slug) (probatorium ansible/tasks/run_bench_cell.yml:2-6), the server is launched once and mpstat -P ALL 1 {{cell_guard_seconds}} runs once per column (lines 384-389), and aggregatePerCellResults stamps that single aggregate onto every scenario the runner expanded ("The same aggregate applies to every scenario ... since the observer scopes to the whole cell process", mage_bench.go:996-1004). Verified in the published data the design cites: in docs/results/v1.5.8/20260805/x86_64/summary.json every celeris column has 27 scenarios, exactly 1 distinct resources summary, byte-identical series across ws-large-echo and get-json, spanning 1334 s (8024 s in the 20260829 full run). So the motivating "existing signal" — io_uring 95.6-96.8 % vs epoll 53-58 % "on precisely the ZC-eligible cell" — is the column-wide average over churn-close, driver-*, get-json, etc., and says nothing about ws-large-echo. Under the proposed 3-cell scoping the treatment cell and BOTH controls would share one CPU scalar: the control test ("ws-echo/get-json must move <= 0.6 CPU pt") is unevaluable, and any treatment delta is diluted by two control cells plus cooldown/idle gaps. cpu_per_GB as specified cannot discriminate ON from OFF. rps cannot rescue it on amd64: at 29.7 K msg/s x 64 KiB x 2 directions the cell is near the 2x10G bond's line rate (std reaches 35 K), which is why the design itself leans on CPU.

2. Under the bench configuration the ZC branch almost never carries bytes, so the toggle is nearly inert. Detached WS egress first tries an inline unix.Write(cs.fd, cs.writeBuf) on the dispatch goroutine and only hands the PARTIAL remainder to the ring (celeris engine/iouring/worker.go:1567-1583 guarded closure; ring path = drainDetachQueue -> flushSend -> prepSendSQE -> useSendZC, worker.go:3896-3991). loadgen's ws-large-echo is strictly ping-pong — exchange writes one frame then blocks until its echo arrives (loadgen ws.go:138-163) — so each conn has at most one 64 KiB frame in flight and an empty send queue when the echo is written; the SUT runs kernel-default autotuned tcp_wmem (bench_tuning.yml touches no wmem/rmem sysctl). In steady state the whole frame goes inline; SEND_ZC only fires for the rare short-write remainder that is also >= 4096 B. The inline path landed in 0af8831 (2026-07-03) and is inside v1.5.7 and v1.5.8, so even the cited 96 % number was produced with ZC essentially unreachable. The design concedes this in not_measurable(2) but its decision rule maps NOISE to "leave the default ON, close io_uring: probeSendZC can never detect copy-fallback — SEND_ZC enabled on every host regardless of NIC #465 as no measurable effect" — the result the run would produce whether a ZC send costs 0 % or 50 % more CPU than a SEND, because there is no ZC-fired count to distinguish "ZC is free" from "ZC never executed". The bpftrace count is optional and one-arm-only, and the celeris ZC counters are optional.

Minor: neither the sut_env nor the cells workflow input exists yet, so step 3's command line cannot run today (the design acknowledges this). The arm-identity check is sound: profile.SendZC = enabled precedes SelectTier(profile) (engine.go:97-103), so the logged send_zc= is the effective post-env value.

Change to the design: Make both legs discriminating before spending cluster time:

A. Per-cell CPU. Never use the summary scalar. Compute per-scenario CPU by windowing the raw 1 Hz cpu.log (wall-clock rows; report/resources.go ParseMPStat already returns the series) against the runner's per-scenario started_at/completed_at recorded in results.json (cmd/runner/main.go:1093-1096), trimming the warmup window; or have the harness restart mpstat per scenario. Add SUT-process CPU (utime+stime from /proc//stat, the observer already owns the PID: run_bench_cell.yml:420) and mpstat's %soft so a ZC cost (kernel NOTIF/skb_copy_ubufs work) is attributable rather than host-wide. Define cpu_per_GB and the control comparisons on these windowed numbers only. Before dispatching anything, apply the same windowing to a retained v1.5.8/20260829 run artefact to learn whether io_uring's per-cell CPU on ws-large-echo actually differs from epoll's — the current 96 % vs 55 % motivation is a column average and must be retracted.

B. Make ZC carry the bytes, and prove it. (1) Promote the celeris counters from optional to REQUIRED: zcSendsSubmitted, zcNotifs, inlineBytes (the unix.Write fast path) and ringBytes (flushSend/WRITEV) on Worker, surfaced via EngineMetrics -> /debug/vars, scraped by the observer per cell; a verdict is only valid if ringBytes/(inlineBytes+ringBytes) >= ~50 % and zcSendsSubmitted is in the thousands/s in the ON arm and 0 in the OFF arm; otherwise the run has answered a different question. (2) Because the production ping-pong path takes the inline fast path, run a 2x2: {SEND_ZC on/off} x {inline egress on/off}. Add a celeris env knob (e.g. CELERIS_IOURING_INLINE_EGRESS=off) that skips the unix.Write fast path so every detached >= 4 KiB send reaches prepSendSQE; the inline-off pair is the only one that measures SEND vs SEND_ZC on the fabric, and the inline-on pair measures whether the default matters in production (expected: it does not fire — which is then a real, counted result, not an inferred one). Alternatively (no celeris knob) add a large-frame fan-out cell (ws-hub-broadcast with 64 KiB payload, the shape celeris's own broadcast_egress_correctness_linux_test.go uses) where per-conn queues fill and the ring path genuinely carries the data. (3) Keep bpftrace opcode-47/26 + kprobe:skb_copy_ubufs, but run it in BOTH arms of at least one pair and make it a precondition (opcode-47 > 0 ON, == 0 OFF), plus ethtool -k bond0 and slaves | grep scatter-gather, so copy-fallback vs true ZC is on record.

C. Keep ABBAAB ordering, one arch per dispatch, server.log send_zc= arm check, and the harness passthroughs (sut_env, cells). Restate the decision rule on the windowed per-cell metrics and the ZC-fired precondition: WIN/LOSS thresholds as before; NOISE is only a verdict when the ON arm demonstrably routed >= 50 % of ws-large-echo bytes through SEND_ZC; a NOISE result with a low ZC fraction closes #465's production question ("the default is immaterial because the branch does not fire on the echo path") but leaves the fabric question open for the inline-off pair.

Review 2: refuted

Why: The design's DECISION METRIC (cpu_per_GB = mean_cpu_pct / bytes on ws-large-echo) and the premise that motivates the whole A/B ("io_uring at 95.6-96.8 % SUT CPU on precisely the ZC-eligible cell vs epoll 53-58 %") both rest on resources.summary.mean_cpu_pct being a per-(column, scenario) number. It is not. In the harness a "cell" is one COLUMN's whole runner pass: bench.yml:266-282 builds the schedule as [(0, competitor)...], run_bench_cell.yml:42 sets cell_dir = <run_dir>/00-<competitor>, the mpstat sampler (run_bench_cell.yml:384-389) and the observer (:404-426) are started ONCE per that dir for cell_guard_seconds, and the single runner invocation (:514-525) runs every scenario matched by -cells inside it. The merge then copies the one aggregate onto every scenario — mage_bench.go:996-1004, in its own words: "The same aggregate applies to every scenario the runner expanded in this cell, since the observer scopes to the whole cell process." The published data proves it: in all six v1.5.8 summary.json files every column has exactly ONE distinct mean_cpu_pct across its 27-29 scenarios (e.g. celeris-iouring-h1-async 20260829 amd64: churn-close, get-json, ws-echo, ws-large-echo ... all 96.283 %; epoll-h1-async all 53.31 %), and the "ws-large-echo" series spans 1334 s (fast) / 8024-8850 s (90 s runs) — the whole column — not one ~35-90 s cell. So the "1.7x CPU per byte on the ZC-eligible cell" is an artefact of reading a column-wide mean (dominated by 27 other scenarios incl. get-json at 1 M rps and churn-close) as a per-cell value; the io_uring-vs-epoll CPU gap is column-wide (consistent with worker.go:1252 adaptiveTimeout()==0 whenever dirtyHead != nil, i.e. a busy loop under any pending send) and says nothing specific about SEND_ZC. Consequences for the proposed measurement as written: (a) with the planned cells='ws-large-echo/*,ws-echo/*,get-json/*' passthrough, all three scenarios run in ONE column pass and share ONE cpu.log, so the treatment cell and both "negative controls" would report the IDENTICAL mean_cpu_pct by construction — the CPU control is vacuous and cannot flag drift; (b) that shared number would be the average over get-json (CPU-saturated at ~1 M rps regardless of ZC), ws-echo, ws-large-echo and the idle/warmup edges, so a real ZC-driven CPU change on ws-large-echo (which contributes ~1/3 of the window) is diluted at least 3x before the 5 % threshold is applied, biasing the rule toward NOISE — and the design explicitly closes #465 on NOISE as "no measurable effect on this fabric"; (c) the 5 %/3 % thresholds were calibrated against "same-window spread ... CPU <= 0.6 pt over five v1.5.8 runs", which is the spread of a column-wide mean over ~27 scenarios, not the per-cell noise floor, so the threshold itself is mis-sized. Secondary, non-load-bearing errors: the "expected_if_true" anchor "mean_cpu_pct drops from the ~96 % plateau toward epoll's 53-58 %" compares column-wide means; and resources.series cpu_pct points are the same column-wide 60-point downsample. What is NOT refuted: saturation_mode_rps on ws-large-echo is a genuine per-scenario observable (loadgen ws.go:138-153 is a synchronous write-then-read per conn, so msg/s tracks SUT turnaround; std reaching 34.8-35.4 K on the same fabric shows io_uring's 26-29.7 K is not wire-capped), the send_zc=true|false log-line arm check, the code-path analysis (inline unix.Write fast path at worker.go:1567 with ring SEND -> useSendZC(n>=4096) only on the EAGAIN/partial remainder; flushSendLink 'never ZC' for H1), and the ABBAAB ordering. The design would produce numbers, but its CPU-per-byte verdict would be about the column, not the cell.

Change to the design: Make the CPU observable per-scenario before running anything, and re-base the decision rule on it. Concretely: (1) In the merge (mage_bench.go readCellResources / report.SummarizeResources) window the column's cpu.log and observer.sqlite per scenario using the per-scenario started_at/finished_at the runner already writes (cmd/runner/main.go:304/339, report/schema.go:296-297; mpstat rows carry wall-clock times, observer rows carry ts_unix), and attach the sliced ResourceStats to each scenario record instead of the shared aggregate; alternatively, and with zero merge changes, dispatch ONE scenario per column pass (cells='ws-large-echo/*' only, and separate dispatches for the ws-echo and get-json controls) so the column-wide mean IS the cell mean — but then compute the mean over the measurement window only (exclude the warmup/idle edges via the timeseries t in run0//*.json), because the guard window is ~5/4 x budget + 300 s and mostly idle for a 35 s cell. (2) Replace host-wide mpstat as the primary with per-process SUT CPU (utime+stime of the server.pid from /proc, which the observer already sidecars, or pidstat -p) so worker busy-spin/softirq is at least separable; keep mpstat 'all' as secondary. (3) Re-derive the noise floor and the 5 %/3 % thresholds from per-cell repeats (>=3 identical-arm ws-large-echo runs), not from the column-wide spread. (4) Because the io_uring worker spins whenever dirtyHead != nil (adaptiveTimeout()==0), report and reason on rps as the primary decision metric on both arches (it is per-scenario and not wire-capped), with per-scenario CPU as the secondary; do not close #465 on a CPU NOISE verdict unless the per-scenario CPU is shown to move with a known perturbation (e.g. the ws-echo vs ws-large-echo difference) so the metric is proven sensitive. (5) Make the ZC-fired counter (zcSendsSubmitted / zcNotifs via EngineMetrics -> /debug/vars, scraped by the observer) REQUIRED rather than optional, so NOISE can be distinguished from "ZC never fired" on this path; the bpftrace step can stay optional. (6) Correct the premise in the issue write-up: the 96 % vs 55 % io_uring-vs-epoll CPU gap is column-wide across all ~27 scenarios and belongs to a separate io_uring loop-CPU issue, not to #465.

[FumingPower3925]

Measured 2026-09-13 (branch feat/bench-per-scenario-cpu, commit ff71458)

Verdict
N/A for #585 itself — no benchmark was run (by instruction). The prerequisites are IMPLEMENTED and MEASURED at unit/merge level: the windowing discriminates scenarios sharing one cpu.log (45.0 vs 93.0 vs column 47.667), trims the warm-up (35.0 -> 45.0), reports no-data as absence, and the env passthrough demonstrably reaches a launched process and its log. New JSON fields: raw/.json cells[] started_at, completed_at, scenario_resources (ResourceStats), scenario_resources_status (ok|no_data|no_timestamps|empty_window|no_sidecar); results.json (schema 5.9) benchmarks[] scenario_resources map, ResourceStats window {start,end,cpu_samples,observer_samples}, summary mean_soft_pct, sut_process_cpu_pct (percent of ONE core), environment sut_env; observer.sqlite columns cpu_utime_ticks, cpu_stime_ticks. Dispatch inputs: sut_env (KEY=VALUE[,KEY=VALUE], default '') -> BENCH_SUT_ENV, cells (comma-separated '/' globs, '!' excludes, default '') -> BENCH_CELLS; ansible extra-var bench_sut_env (JSON dict) merged via combine() into the SUT environment in run_bench_cell.yml; mpstat now launched with S_TIME_FORMAT=ISO TZ=UTC.

Numbers (from assertion output)
Unit windowing (report.TestWindowResourcesTwoScenarios, synthetic ISO cpu.log 12 rows 10:00:00..11 busy 10,20,30,40,50,60,70,80,90,96,12,14 with %soft 1..12; observer ticks 50/s until 10:00:05 then 250/s; warm-up 2 s): column-wide mean 572/12 = 47.667 %; scenario A (started 10:00:00, completed 10:00:05 -> window 10:00:02..05, 4 cpu + 4 observer samples) mean_cpu_pct 45.0, mean_soft_pct 4.5, sut_process_cpu_pct 50.0, peak_rss 105 (rows 02..05 only); untrimmed A would be 35.0; scenario B (10:00:06..09 -> window 08..09, 2+2 samples) mean_cpu_pct 93.0, mean_soft_pct 9.5, sut_process_cpu_pct 250.0. AM/PM + C-locale banner + midnight rollover: row0 ts = 2026-12-31T23:59:59Z, row1 = +1 s, soft 0.5/1.5. ReduceResources over {2,40},{4,60} -> mean_soft_pct 3, sut_process_cpu_pct 50, window carried from last run; column-wide inputs keep all three nil. Merge path (mage TestAggregateWindowsScenarioResources, aggregatePerCellResults(dir, 2s) + mergeBenchResults): raw cells ws-large-echo 45.0/4.5/cpu_samples=4 status ok, ws-echo 93.0/9.5/2 ok, get-json status no_data with scenario_resources nil, gin-h1 (no sidecar) no_sidecar with resources nil; every resources entry = 47.667 with no window/soft fields; results.json schema_version 5.9, environment.sut_env CELERIS_IOURING_SEND_ZC=off, exactly 1 benchmark carries scenario_resources (2 entries, none for get-json). parseSUTEnv: 19-case table (7 accept incl. 'A=b=c' -> 'b=c' and 'A=' -> '', 12 reject incl. PATH, LD_PRELOAD, dup key, stray/trailing comma, space/quote/$ in value). Observer /proc parse: comm '(bench (sut) srv)' -> utime 250 stime 75. Ansible local render: OFF arm server.log line 1 '[sut-env] {..., "CELERIS_IOURING_SEND_ZC": "off"}' and the process sees CELERIS_IOURING_SEND_ZC=off DOTNET_ROOT=/opt/bench/dotnet; no-override render: key unset, base env intact. Test runs: report ok 0.380s, cmd/observer ok 0.685s, cmd/runner ok 1.471s, budget ok 0.735s, root -tags mage ok 0.444s/0.342s; go vet ./... and go vet -tags mage . clean; actionlint clean; gofmt clean. Diff: 18 files (13 modified + 5 new), +532/-66 in e2e553f plus the 1-line parser guard + 1 test in ff71458.

Negative control
Three, all run and asserted: (1) a scenario whose window (10:00:30..40) lies after the sampler stopped returns (nil, "no_data") — no zero-valued summary — in both the report unit test and the full merge path (get-json cell: scenario_resources absent, scenario_resources_status="no_data", and no entry in results.json benchmarks[].scenario_resources); (2) a pre-v5.9 observer DB without tick columns yields CPUTicksOK=false on every sample and sut_process_cpu_pct=nil while RSS/FD still window (3 observer / 0 cpu samples); a tick-column DB of all zeros (dead PID after respawn) likewise yields nil, not 0 (ff71458); an mpstat log with no parseable banner date yields TSUnix=0 rows that cannot be placed -> "no_data"; (3) the column-wide resources aggregate is asserted unchanged on every record (47.667 on all three scenarios, Window/MeanSoftPct nil), and the mage-tag suite's pre-existing merge tests (TestMerge*, TestAggregateReconstructsColumnMissingRollup) still pass with the new fields present. Ansible no-override render: CELERIS_IOURING_SEND_ZC=unset.

Found on the way (not the closure under test)
(1) Pre-existing and now mitigated: the observer pins the PID resolved at bind time; after the respawn supervisor replaces the SUT, every /proc field (RSS, FD, and now ticks) reads 0 for the rest of the column — RSS/FD zeros were already silently folded into the column aggregate; ticks are now treated as absent (ff71458) but RSS/FD remain a false-zero source for any respawned column (out of scope, worth an observer re-resolve on pid death). (2) Confirmed from code, not new: mage_bench.go stamped one column-wide ResourceStats on every scenario (the issue's review finding) — kept as-is under resources, complemented by scenario_resources. (3) For RATED cells the scenario window runs from started_at+warmup to completed_at, i.e. it spans the saturation pass AND the rated sweep (the runner records one pair per cell, cmd/runner/main.go:1093-1096); for profile=fast (saturation-only) this is exactly the measurement window, but a rated-profile per-scenario CPU mixes both passes — noted in code comments, not fixed. (4) go test ./... on this Mac exhausts process slots when the validation package's e2e tests run in parallel with everything else (fork EAGAIN); passes alone.

Not measured, and why
(1) Nothing was run on the cluster: run_bench_cell.yml's merged environment: + '[sut-env]' echo was proven only by an equivalent scratch task under ansible connection=local, not by the real playbook (its bench_target/cell_dir fact context needs the cluster). (2) Wall-clock joining of RETAINED (pre-ISO/TZ) cpu.logs assumes the SUT printed UTC local time and one of the three accepted banner date layouts (ISO, MM/DD/YYYY, MM/DD/YY); whether msa2-server/msr1's locale/timezone matched is unverified — the review-1 step 'apply the windowing to a retained v1.5.8/20260829 artefact before dispatching' was NOT done (no cpu.log is retained locally; it needs the benchmark-tier-results-<run_id> artifact, then aggregatePerCellResults on it). New runs are deterministic (S_TIME_FORMAT=ISO TZ=UTC). (3) readCPUTicks against a real /proc//stat was exercised only via a synthetic line + a missing PID (macOS has no /proc); the USER_HZ=100 assumption is documented, not measured on the cluster kernels. (4) Sub-second alignment: mpstat and the runner's timestamps are joined at 1 s resolution with the SUT and loadgen clocks assumed in sync (NTP) — a skew of a few seconds shifts a 35 s window's edges. (5) The celeris-side ZC counters (zcSendsSubmitted/zcNotifs/inlineBytes/ringBytes) and the inline-egress knob the reviews made REQUIRED for a valid NOISE verdict are celeris changes, not covered here. (6) The budget FitWithin projection for a cells-scoped run still counts the full grid (passes trivially, logged). (7) yamllint/ansible-lint are not installed here; the ansible YAML was validated only by ansible's own parse during the local render and by the go/actionlint tooling for the rest.

Adversarial review of this measurement

Skeptic 1 — refuted: True, accepts as measured: False
The closure the issue defines (a six-run SEND_ZC on/off A/B per arch with per-scenario CPU, ZC-fired precondition, and a WIN/LOSS/NOISE verdict recorded in #465) was not run; the branch delivers harness prerequisites only, which the report itself says. The unit-level claims DO reproduce: from a git-archive copy of origin/feat/bench-per-scenario-cpu (e2e553f, ff71458) go test ./report/ ./cmd/observer/ ./budget/ and go test -tags mage . all pass and the verbose output carries the reported assertions (A=45.0/4.5/50.0 vs B=93.0/9.5/250.0 vs column 47.667; merge log 'per-scenario resource windows: no_data=1 ok=2'; 19-case parseSUTEnv table; TestMerge* untouched). Code matches the report file-for-file; the ansible render logs are NOT committed on the branch (they exist only in the worktree .claude/worktrees/wf_fa89913b-936-6/scratchpad/logs/ and their content matches the report). Real mpstat under S_TIME_FORMAT=ISO prints rows as bare HH:MM:SS (sysstat mpstat.c), which the cpuIdx==1 parser path handles. BUT applying the branch's windowing to the retained artefact (benchmark-tier-results-33256654220, 20260829, the binding review-1 step the agent skipped) refutes two discriminating claims. (1) 'no-data as absence' is not a guard: both cluster hosts logged mpstat in LOCAL time UTC+2 (msa2-server observer first ts 1788080413 = 09:00:13Z while cpu.log row 1 reads 11:00:13; msr1 identical), so the parser places every retained row 2 h late and WindowResources returns status=ok with the wrong slice, not no_data: celeris-std-h1 ws-large-echo naive 67.41 % vs corrected 42.16 %; epoll ws-echo 64.80 vs 52.83; get-json windows land as observer-only 'ok' with cpu=nil. New runs pin TZ=UTC so this is mitigated, but nothing cross-checks the mpstat clock against the observer's unix ts, so any residual skew still reports ok. (2) The CPU observable itself does not discriminate on io_uring: mean_cpu_pct = 100-%idle counts %iowait as busy, and io_uring's CQ wait is charged as iowait. Retained 20260829 amd64: iouring column usr 17.6 sys 23.7 iowait 40.9 soft 14.1 idle 3.7 => 'busy' 96.3 but 55.3 excluding iowait; epoll iowait 0.0, busy 53.3; std 55.7. arm64: iouring 95.6 -> 70.5 vs epoll 69.3. In the ws-large-echo 90 s saturation window (TZ-corrected) iouring is usr 49.8 sys 15.0 iowait 17.5 soft 12.6 idle 5.0 (77.4 % real) vs epoll 50.3/12.2/0.0/12.4/25.0 (74.9 % real). So the issue's motivating '1.7x CPU per byte on the ZC-eligible cell' is an iowait accounting artefact, and per-scenario mean_cpu_pct on io_uring is pinned at 92-99 % on ALL three planned cells (ws-large-echo 94.98, ws-echo 95.67, get-json 98.54 sat-only; msr1 92.05/91.62/99.43): a >=5 % cpu_per_GB delta between ZC arms cannot appear in that metric regardless of what ZC costs. Only the new sut_process_cpu_pct (utime+stime excludes iowait) and mean_soft_pct could discriminate, and the report's per-run check and the issue's decision rule still lead with mean_cpu_pct. (3) The binding review corrections that make a NOISE verdict falsifiable (celeris zcSendsSubmitted/zcNotifs/inlineBytes/ringBytes via /debug/vars scraped by the observer, inline-egress knob) are absent, so the listed dispatch commands would produce an unfalsifiable NOISE. Minor: the runner's started_at..completed_at spans saturation + rated sweep (310 s for a 90 s cell in the retained data), fine for profile=fast only, as the report notes.

What would complete it: (a) Parse %iowait into CPUPoint and report a per-window busy-excluding-iowait (or usr+sys+irq+soft) alongside mean_cpu_pct; re-base cpu_per_GB and the control comparisons on sut_process_cpu_pct (primary) and mean_soft_pct, with rps as the primary metric per review 2; correct #585's premise (the 96 vs 53 gap is iowait). (b) Add a clock-consistency guard in the merge: compare the first mpstat TSUnix against the first observer ts in the same cell dir and mark windows 'clock_skew' (not ok) when |delta| exceeds a few seconds; optionally accept an explicit offset so retained artefacts can be windowed honestly. (c) Land the celeris ZC counters via EngineMetrics -> /debug/vars, have the observer record them, and make 'zc submitted > 0 in ON, == 0 in OFF, ringBytes fraction >= 50 %' a merge-level precondition on the verdict. (d) Commit the render logs, then run ONE real scoped dispatch (profile=fast, cells=ws-large-echo/,ws-echo/,get-json/*, sut_env=CELERIS_IOURING_SEND_ZC=off) and assert server.log line 1 '[sut-env]', send_zc=false, and scenario_resources_status=ok with cpu_samples ~= 35 on all three cells. (e) Only then run the 6+6 ABBAAB series and record the per-pair numbers in #465/#585 — that is the measurement the issue defines.

Skeptic 2 — refuted: True, accepts as measured: False
The closure the issue defines (celeris#585: six ABBAAB dispatches per arch on the cluster, per-scenario cpu_per_GB + rps deltas on ws-large-echo with ws-echo/get-json controls, send_zc= arm line per cell, and — per the binding review corrections — a ZC-fired counter proving the branch executed) was not measured at all: zero cluster runs, zero exposure of the SEND_ZC branch, no arm line, no pair deltas. The agent's own verdict field says "N/A". What the branch delivers is harness prerequisites, and those I could verify: origin/feat/bench-per-scenario-cpu = e2e553f + ff71458 on top of origin/main 6051f4a, 18 files (+1603/-66, not the +532 the report states — the difference is the test files); extracted with git archive into the scratchpad and run: report package all PASS, cmd/observer PASS, mage-tag TestAggregateWindowsScenarioResources/TestParseSUTEnv(19 subtests)/TestSUTEnvExtraVars/TestResolveBenchCellsPrecedence + pre-existing TestMerge*/TestAggregateReconstructs* PASS, no SKIPs. The reported numbers (col mean 572/12=47.667, A=45.0/soft 4.5/proc 50%, B=93.0/9.5/250%, untrimmed 35.0, get-json no_data, gin-h1 no_sidecar, schema 5.9, sut_env=off) are literal assertion targets in report/window_test.go and mage_bench_sutenv_test.go. I ran two mutation controls the agent did not: reverting the ff71458 ut+st > 0 guard makes TestWindowResourcesZeroTicksAreAbsent FAIL ("0/0 ticks must read as absent"), and removing the warm-up trim in ScenarioWindow makes TestWindowResourcesTwoScenarios FAIL (got 35 want 45, 6/6 samples vs 4/4) — so the tests discriminate. Runner code confirms one StartedAt/CompletedAt pair per cell around both saturation and rated passes (cmd/runner/main.go:1093-1096), the runner's filterCells honours comma globs and '!' excludes, mage Bench's competitors-vs-cells check accepts 'ws-large-echo/*,...' with competitors=celeris-iouring-h1-async, and the mpstat sampler now runs under S_TIME_FORMAT=ISO TZ=UTC. Refuting points beyond "not run": (1) the scratchpad/logs ansible-render evidence is NOT on the branch (git ls-tree finds no scratchpad path), so the run_bench_cell.yml combine()/'[sut-env]' echo claim is unverifiable from what was pushed, and the real playbook was never executed; (2) there is no PR and no CI run for the branch (gh pr list / gh run list both empty) and go test ./... never completed locally (fork EAGAIN), so the full suite has not been green anywhere; (3) the binding review corrections that make a NOISE verdict admissible — celeris ZC counters (zcSendsSubmitted/zcNotifs/inlineBytes/ringBytes via /debug/vars) and the inline-egress knob, plus windowing a retained v1.5.8/20260829 artefact before dispatch — are not done, so even with this harness the planned A/B could not distinguish "ZC is free" from "ZC never fired"; (4) sut_process_cpu_pct still rests on the observer's bind-time PID pin (a respawn now yields nil rather than 0, but RSS/FD false-zeros remain); (5) actionlint on the extracted copy reports runner-label warnings on a pre-existing line (no actionlint.yaml in the repo), so "actionlint clean" depended on the agent's local config — not a branch defect.

What would complete it: (1) Open the PR for feat/bench-per-scenario-cpu and get CI (lint + full go test on Linux) green; commit or attach the ansible render logs, or better run run_bench_cell.yml once on the cluster and show server.log line 1 '[sut-env] {...}' next to the engine's send_zc= line. (2) Land the celeris side the reviews made REQUIRED: zcSendsSubmitted, zcNotifs, inlineBytes, ringBytes on Worker surfaced via EngineMetrics -> /debug/vars (scraped by the observer), and a CELERIS_IOURING_INLINE_EGRESS=off knob; extend send_zc_gate_test.go. (3) Apply WindowResources to a retained benchmark-tier-results artefact (v1.5.8/20260829) and show io_uring vs epoll per-cell CPU on ws-large-echo actually differ, and that ws-echo vs ws-large-echo move the metric (sensitivity). (4) After the 24 h soak and nightly drain, run the six ABBAAB dispatches on msa2-server then msr1 exactly as the issue's commands specify, and for each run assert: server.log send_zc=true/false matches the arm, results.json environment.sut_env matches, scenario_resources_status=ok for all 3 cells with cpu_samples≈35, zcSendsSubmitted in the thousands/s ON and 0 OFF with ringBytes/(inline+ring) >= 50 % (else the run answered a different question); then report per-pair Delta cpu_per_GB and Delta rps on ws-large-echo plus the two controls and apply the WIN/LOSS/NOISE rule. Only those numbers close #585.

[FumingPower3925]

Lane M, 2026-09-26: harness done (goceleris/probatorium#411), the A/B is queued with its ETA

Updated 16:13Z: #411 is rebased onto probatorium main bf870f5 at 521a6a3. Its evidence and the dispatch branch were re-run at that head, and the figures below are the final head's.

#585 closes on the cluster A/B. That A/B could not answer anything until two harness defects were fixed. Both are verified at probatorium f2e4297.

1. The bench server served no /debug/vars.
   • run_bench_cell.yml scraped http://127.0.0.1:8080/debug/vars for every celeris column. Started the way the playbook starts it, the base binary answers 404 there.
   • So no engine counter ever reached a bench artifact, including the ones that say whether SEND_ZC carried a byte.
2. report/resources.go:236 counts %iowait as busy.
   • Over the retained 20260829 io_uring mode split, base reports mean_cpu_pct 96.3. The PR also reports mean_iowait_pct 40.9 and mean_cpu_ex_iowait_pct 55.4.
   • This corrects the issue's premise. The "1.7x CPU per byte on the ZC-eligible cell" was a column-wide mean (review 2), and 41 of its 96 points are io_uring's completion wait charged as iowait.

What goceleris/probatorium#411 does:

• The bench server serves /debug/vars on a side listener. It adds no route to the benched router, puts no connection on the benched engine, and does not stop the world. The engine block is EngineMetrics whole, so it builds against every pinned celeris, v1.5.8 included.
• The observer records ZCSendsSubmitted, ZCNotifs, InlineBytes, RingBytes and BytesWritten every second. A counter that was not published is written as −1, so an OFF arm's 0 stays a reading. The observer also refuses a foreign-pid document.
• Every per-scenario window now reports the deltas plus ring_bytes_fraction, along with the iowait-free CPU.
• servers.Adapter.SUTEnv gives each column its own runtime env. The port guard verifies it in /proc/<pid>/environ. ON, OFF and an A/A twin of one binary now run minutes apart in one dispatch.
• Schema 5.17, additive. deps: bump the all-go-deps group across 2 directories with 3 updates #410 took 5.16, and v1.5.8: WS-upgrade crash + Context/stream use-after-recycle + H2/overload perf (#406/#407/#408) #411 was rebased over it.

Evidence (base dir evidence/measurements-585-587-588/lane-20260926/585/):

• base-behaviour-585.sh, base bf870f5 vs 521a6a3: base gives the 404 and 96.3 readings above. The PR answers 200 from the SUT's own pid, with 70 engine fields (celeris 9f4d89b).
• mutants-585.sh at 521a6a3: 23/23 named tests pass, and 11/11 mutants are killed.
• Whole-package suites, base vs PR, with go test -v on fresh trees:
  • root: 922 → 934 PASS;
  • -tags mage: 102 → 104 PASS;
  • 0 FAIL, identical SKIPs, 0 regressions.
• ansible-render/: the column-env merge and the guard, rendered.
• CI on v1.5.8: WS-upgrade crash + Context/stream use-after-recycle + H2/overload perf (#406/#407/#408) #411 at 521a6a3 is green: Test run 36253694927, Lint, CodeQL.

The A/B (queued in evidence/_queue/cluster.tsv, not dispatched):

• Branch: perf/zc-ab-9f4d89b @ baa0b2d, which is v1.5.8: WS-upgrade crash + Context/stream use-after-recycle + H2/overload perf (#406/#407/#408) #411 @ 521a6a3 plus servers/zc_ab_arms.go. Celeris 9f4d89b comes from main's pin.
• Pre-flight: a cmd/runner -dry-run gives 63 cells, in A1/B/A2 order per engine (585/preflight-20260926T155752Z-baa0b2d).
• Columns: per io_uring engine (h1-async, h1-sync, auto+upg-async), <n> (ON), <n>-zc1-off (OFF) and <n>-zc2-on (ON twin), run back to back.
• Scenarios: ws-large-echo, ws-echo, ws-hub-broadcast-128/1024, sse-fanout-128/1024 and get-json, so 63 cells per arch.
• Dispatches: one per arch, msa2-server and then msr1. Running both arches in parallel depressed ws-large-echo about 8 %.
• ETA: about 72 min each, about 2.4 h in total.
• Verdict: pre-registered in 585/zc_ab_verdict.py. Its synthetic controls pass 5/5.
  • A cell votes only if both ON arms show zc_sends_submitted > 0 and ring_bytes_fraction >= 0.5.
  • WIN, LOSS or NOISE is judged against each arch's own A1/A2 floor.
  • "NOT EXERCISED" is a named outcome, not a silent NOISE.
• Release rule: LOSS on both arches flips resolveSendZCPolicy auto to off before the release bench. Any other outcome keeps ON and records the table here.

One exposure reading already exists. It comes from the celeris#588 container leg: a plain build of auth_session_ratelimit, whose 64 KiB echo walker was running on an adaptive cell promoted to io_uring. That cell submitted 985 SEND_ZC sends, with ring bytes 8.8 GB against 264 MB inline. Streaming load does reach the arm. Whether ws-large-echo's ping-pong reaches it is what the A/B's exposure column will show.

[FumingPower3925]

SEND_ZC on/off A/B, both arches: NOT EXERCISED, so the auto default stays ON

Decision. The pre-registered rule flips the default only on LOSS on both arches. The verdict is NOT EXERCISED on amd64 and NOT EXERCISED on arm64, so resolveSendZCPolicy keeps auto = enabled when the functional probe passes, for every size and all three io_uring engines. sendZCMinBytes = 4096 and the linked-send exclusion stay as they are. No code change follows from this measurement. The rule says to record the table in #585 and #465; it is recorded here, and this mention links it from #465.

Runs

arch	run	SUT host	bench job (UTC, 2026-09-27)	cells
amd64	Benchmark Tier 36325718467, success	msa2-server	14:25 to 15:28	63, all ok, 0 client errors
arm64	Benchmark Tier 36330506266, success	msr1	15:57 to 16:57	63, all ok, 0 client errors

• Both runs: branch perf/zc-ab-9f4d89b @ 8871d72, celeris 9f4d89b (v1.5.12-0.20260920100717-9f4d89b171db in all 18 columns), loadgen v1.4.13, results schema 5.17.
• Inputs, from the job logs: profile fast (35 s windows), competitors all, deploy celeris, publish 0, sut_env empty, cells ws-*/celeris-iouring-*,sse-*/celeris-iouring-*,get-json/celeris-iouring-*.
• One arch per dispatch, run serially. No other cluster job overlapped either run. The only cluster job between them was Nightly Validation 36325737189 (15:33 to 15:52).

Design

• For each io_uring engine (auto+upg-async, h1-async, h1-sync) there are three columns of the same build. They differ only in the column env: A1 CELERIS_IOURING_SEND_ZC=on, B off, A2 on (the A/A twin). They run back to back, with A1 and A2 11.7 to 11.8 min apart.
• 7 scenarios: ws-large-echo (64 KiB frames), ws-echo, ws-hub-broadcast-128/1024, sse-fanout-128/1024, and get-json (the control: linked SEND, never ZC).
• The functional probe passed in all 18 columns (functional=true), so auto == on on both hosts. The A arms are the shipped default.
• Metrics:
  • rps = saturation_mode_rps.
  • cpb = SUT-process core-seconds per GB written.
  • Secondary: host CPU excluding iowait (mean_cpu_ex_iowait_pct). It has no floor and does not vote.
• d = (B − mean(A1, A2)) / mean(A1, A2), signed so that d > 0 means OFF is better.
• Floor per arch and metric = max(p90 of the 21 A/A values of |A2 − A1| / mean, 3 % for rps, 5 % for cpb).
• A cell votes only if it is exposed: both A arms have zc_sends_submitted > 0 and ring_bytes_fraction >= 0.5. An arch with no exposed cell is NOT EXERCISED.

Arm proof

• --artifact read 9/9 column server.logs per arch and found 0 problems. The A columns log "CELERIS_IOURING_SEND_ZC": "on" and send_zc=true. The B columns log "off" and send_zc=false.
• B arms: zc_sends_submitted == 0 in all 21 B cells per arch, with the field present. The field is also present in all 42 A cells per arch.
• inline_bytes + ring_bytes == bytes_written holds in 124 of 126 cells. The other two (arm64) are off by 9 bytes each.

Exposure: why nothing voted

Exposed cells: 0/21 on amd64 and 0/21 on arm64.

• ws-large-echo was expected to decide. On both arches 100 % of its bytes went through the detached inline unix.Write path, with 0 ring bytes and 0 ZC sends:
  • amd64: 73.7 to 75.8 GB per 35 s window (16.8 to 17.3 Gbit/s, network_bound=true);
  • arm64: 28.8 to 31.0 GB (6.6 to 7.1 Gbit/s).
• ws-echo, ws-hub-broadcast-128/1024 and sse-fanout-128: 100 % inline on both arches.
• get-json: 100 % ring bytes and 0 ZC sends. H1 responses go out as linked SENDs, which useSendZC excludes.
• ZC fired in one scenario only: arm64 sse-fanout-1024.
  • 1,887 sends across the 6 ON columns (A1/A2 per engine: 287/341, 236/316, 505/202).
  • Ring share in the A arms: 0.75 to 2.03 % (0.41 to 0.48 % in B).
  • amd64 submitted 0 ZC sends in every cell.

Floors

arch	rps floor	cpb floor	A/A rps median / p90 / max	A/A cpb p90	A/A host CPU ex-iowait p90
amd64	3.00 %	5.00 %	0.74 / 2.23 / 3.55 %	2.73 %	9.28 %
arm64	6.16 %	5.00 %	1.44 / 6.16 / 12.90 %	3.92 %	13.75 %

The DRIFT flag (get-json moving beyond the floor) was not raised on either arch:

• amd64: 0 get-json crossings.
• arm64: 1 crossing, h1-sync rps +6.22 %. There A2 is the outlier: A1 394.9k, B 394.1k, A2 347.1k.

Deltas per scenario

Every scenario was eligible to vote; none was exposed. Each cell lists auto+upg / h1-async / h1-sync, in %, with d > 0 meaning OFF is better.

amd64 (floors: rps 3.00 %, cpb 5.00 %)

scenario	ZC sends, A arms	ring share, A arms	d rps	d cpb	d host CPU ex-iowait
ws-large-echo	0	0 %	−0.04 / +1.27 / −1.18	−0.40 / −0.79 / −0.40	−0.03 / −2.06 / +1.24
ws-echo	0	0 %	−0.83 / +0.68 / +0.80	−1.61 / +0.24 / +0.16	−0.49 / −0.31 / +0.28
ws-hub-broadcast-128	0	0 %	−0.10 / +1.68 / −0.26	−1.16 / −0.18 / +0.03	−3.28 / +15.34 / +5.47
ws-hub-broadcast-1024	0	0 %	−0.76 / +3.21 / −0.46	+1.48 / −0.98 / −1.20	−1.42 / −7.04 / −0.51
sse-fanout-128	0	0 %	−0.92 / −0.08 / −0.66	+0.75 / +0.90 / +3.62	+11.48 / +24.76 / −35.18
sse-fanout-1024	0	0 %	+0.37 / −0.49 / +0.02	−0.48 / −1.98 / −1.11	+2.73 / −2.99 / −2.35
get-json	0	100 %	−0.50 / +0.05 / +1.55	+0.31 / −0.18 / −0.05	+0.28 / −0.52 / −1.42

arm64 (floors: rps 6.16 %, cpb 5.00 %)

scenario	ZC sends, A arms	ring share, A arms	d rps	d cpb	d host CPU ex-iowait
ws-large-echo	0	0 %	+4.02 / −0.87 / −3.34	+1.09 / −0.78 / −3.28	−2.46 / +0.19 / +0.74
ws-echo	0	0 %	+1.18 / +0.77 / +1.05	+0.32 / −0.11 / +0.77	−2.05 / −0.68 / +0.66
ws-hub-broadcast-128	0	0 %	−0.94 / −0.69 / −4.29	+0.66 / −0.20 / −0.60	+9.23 / −5.03 / −2.18
ws-hub-broadcast-1024	0	0 %	−0.88 / +0.63 / +1.43	+1.73 / +1.25 / +1.38	+0.35 / +0.07 / +1.31
sse-fanout-128	0	0 %	+0.87 / +0.21 / +3.06	+1.99 / +0.49 / −1.98	+23.60 / −29.43 / −43.85
sse-fanout-1024	1,887	0.75 to 2.03 %	+4.32 / +3.63 / +4.51	+1.50 / +5.75 / +5.76	−3.11 / +0.92 / +0.75
get-json	0	100 %	−3.65 / +3.53 / +6.22	−3.50 / +0.61 / +0.71	−0.09 / −1.12 / −4.03

What did not reach significance

• No cell voted on either arch, because none was exposed.
• Three single-engine floor crossings had B beyond both A arms (the tool's vote condition, which counts only on an exposed cell). All three are unexposed:
  • amd64 ws-hub-broadcast-1024 h1-async, rps +3.21 %, with 0 ZC sends;
  • arm64 sse-fanout-1024 h1-async, cpb +5.75 %;
  • arm64 sse-fanout-1024 h1-sync, cpb +5.76 %.
• Every other rps and cpb delta is inside its arch's floor, apart from the get-json drift crossing above.
• Host CPU ex-iowait goes beyond its own A/A p90 in 7 cells, all in sse-fanout-128 or ws-hub-broadcast-128. Those are cells at 6 to 39 % host CPU, the signs are mixed, and ZC sent nothing in any of them.
• arm64 sse-fanout-1024 is the only scenario where ZC fired, and OFF was better in all 3 engines:
  • rps +3.6 to +4.5 %, under the 6.16 % floor;
  • cpb over 5 % in 2 of 3 engines.
  • It does not vote, because ZC carried at most 2.03 % of that cell's bytes.
• The same 3/3 same-side pattern also appears where ZC sent nothing: amd64 sse-fanout-128 (rps favours ON, cpb favours OFF) and arm64 ws-echo (rps favours OFF). The chance expectation is about 1.0 per arch; observed: 2 on amd64 and 3 on arm64.
• It could not change the decision either way: amd64 fired ZC in no cell, and the rule needs LOSS on both arches.
• Resolving that hint would take an arm64-only, sse-fanout-1024-only run with several A/B/A repetitions.

Deviations from the plan in the 2026-09-26 comment

• Dispatch head. The runs used 8871d72, not the baa0b2d named above. The arms file (servers/zc_ab_arms.go) was rebuilt on v1.5.8: WS-upgrade crash + Context/stream use-after-recycle + H2/overload perf (#406/#407/#408) #411 @ 82819d1. Between baa0b2d and 8871d72, the bench path (observer, report, bench mage, cell playbook) has these non-comment changes:

  • an egress-counter delta now needs at least two samples and no decrease;
  • the observer refuses a /debug/vars document that lacks the SUT's pid;
  • the port guard also checks the dispatch-level sut_env, and sut_env may no longer override PROBATORIUM_DEBUG_ADDR;
  • become: true was dropped from the port-free task.

  None of them changes how rps, cpb or the exposure fields are defined. The artifact shows the planned 63 cells, in A1/B/A2 order.

• Two failed attempts. Runs 36306263697 and 36306273363, at bef603d, failed in mage BenchTier about 3 minutes in, before any cell ran. Their result artifacts hold only bench-vars.json. On the bench path, bef603d and 8871d72 differ only in the cell playbook (become: true dropped from the port-free task, plus comments) and in go.mod (golang.org/x/sys v0.48.0 moved from indirect to direct). report/, cmd/observer/, cmd/runner/, servers/ and the mage files are identical.

• Tool vs its docstring. The tool counts DRIFT as at least 2 get-json engine-metric crossings, while its docstring says "2 of 3 engines". With 0 and 1 crossings, neither reading raises the flag. Its --artifact mode checks the env and send_zc= lines but not whether the counters are present; that was checked separately (above).

Left open

• The fabric question is not answered: does SEND_ZC beat SEND for ring-bound large sends on this NIC? With inline egress on, these workloads almost never produce such sends. The pre-registration leaves this for a celeris inline-egress-off knob, and no issue tracks it yet.
• Stale wording on main. engine/iouring/doc.go (the CELERIS_IOURING_SEND_ZC entry) and the resolveSendZCPolicy comment in engine/iouring/probe.go still describe the auto default as an open measurement owned by Measure #465: SEND_ZC on/off A/B on the real fabric with per-scenario CPU, both arches #585. That wording needs an update once this issue closes.

Reproduce

• Tool: evidence/measurements-585-587-588/lane-20260926/585/zc_ab_verdict.py, sha256 adfb2adb189ebb4ebea397b8140a8eaf84db79958538afaca8b5c4b70b58a2c8. Its docstring is the pre-registration. The file has not changed since 2026-09-26T12:28:43Z, which is before the rule was posted here (13:06Z) and before any dispatch.
• Full write-up: evidence/measurements-585-587-588/zc-ab-verdict/VERDICT.md.

gh run download 36325718467 --repo goceleris/probatorium -n benchmark-tier-results-36325718467 -D amd64
gh run download 36330506266 --repo goceleris/probatorium -n benchmark-tier-results-36330506266 -D arm64
T=evidence/measurements-585-587-588/lane-20260926/585/zc_ab_verdict.py
python3 $T --selftest                                            # 5/5 PASS
python3 $T --artifact amd64/*/ && python3 $T --artifact arm64/*/ # 9 server.logs, 0 problems, each
python3 $T amd64/*/results.json arm64/*/results.json             # VERDICT results.json: NOT EXERCISED (twice)

[FumingPower3925]

Measured on both arches; see above.