Measure: capture enough at the next h2c_hang / ws_handshake_fail to root-cause it from the artifact, and try to force recurrence

[FumingPower3925]

Filed from the release-readiness sweep of 2026-09-13 under the standing rule that a closure counts only when it is MEASURED. Design produced by a read-only design pass and corrected by two adversarial reviews; the corrections are binding and listed last.

Closure under test

v1.5.11 24h soak (probatorium run 34003151551, celeris v1.5.11 pins, probatorium f8e5d3c): auth_session_ratelimit/iouring/arm64 tier_1.h2c_churn.h2c_hang=1 (cause=timeout, max_elapsed=10000 ms) and auth_session_ratelimit/epoll/arm64 tier_1.ws_torture.ws_handshake_fail=1 (pre-cause-split); neither root-caused; 0 recurrences in the 96 cells of soaks 34368601903 and 34616620237.

What the closure claims

NOT closable as root-caused, and NOT closable as "gone". The two original events are permanently unattributable: the artifact kept only the counters (verified: the msr1 cell-03 directory in the downloaded v1.5.11 artifact held nothing but incidents/20260906-043231-I-MEM-1; no tally sidecar, no error string, no timestamp, no engine log). 0-in-96 later cells is statistically compatible with the observed per-cell rate (~1 h2c hang per ~120k reads, ~1 WS failure per ~47.6k handshakes at concurrency 50 — the memory's own "0-in-55k and 1-in-55k are the same rate" lesson applies). What CAN be closed by measurement is the apparatus claim "the next occurrence will be root-causable from the artifact alone" — that claim is FALSE today, and the measurement below turns it TRUE and verifies it with a fault-injected negative control before the next soak. Separately, a directed 6 h cell can force the rate into a measurable regime (~6 events per cell per counter if the observed rate holds).

What was argued rather than measured

1. "The soak's single h2c_hang was a plain responsiveness stall" (memory absolute-gate-v159-release.md, posted to probatorium#298): argued from config only — every refapp pins Protocol: celeris.HTTP1 (validation/refapp/auth_session_ratelimit/main.go:185) and config.go:289-294 infers EnableH2Upgrade=false, so the preamble is a plain GET / (h2c.go:337-358) that hits session+ratelimit (main.go:215-245; / is not routed, so 404 or 429 = "declined"). Correct that it is not an upgrade path, but stall-vs-deaf-listener-vs-server-header-timeout was never discriminated: h2c.go:185-194 says in so many words that at the then-10 s read budget "it is not recoverable from the artifact whether the server stalled or merely hit its own header deadline" (ReadHeaderTimeout default 10 s, config.go:92-102; enforced in iouring worker.go:1479/2796 and epoll loop.go:297). The budget was raised to 20 s in feat: per-handler async dispatch (sync/async per route + group, all engines × all protocols) #302 (merged 2026-09-07, AFTER this soak).
2. The WS failure's cause class is unknown: the eof/timeout/reset/status/other split (ws.go:56-72, 132-136) also landed in feat: per-handler async dispatch (sync/async per route + group, all engines × all protocols) #302 after the soak. By construction it cannot be a 401/429 (/ws skips both middlewares, main.go:214-245) and the walker's request is well-formed and Origin-less over http, so websocket.go:64-101 yields 101; a non-101 would have to be a 400/403/426 from a corrupted request — nothing in the artifact can say.
3. The strongest prior is celeris#493 (closed v1.5.10): the v1.5.9 soak's 6 h2c hangs + 40 WS + 8 SSE handshake failures on the SAME cell type/arch were a once-a-minute worker-event-loop freeze (store sweep under shard lock with inline dispatch), proven by goroutine dumps inside a stall. The bounded sweeps shipped (middleware/store/memory.go:300-310 sweepBatch=2048; middleware/ratelimit/shard.go:164-213 with runtime.Gosched). v1.5.11 still recorded 1+1 on that cell type — residual sub-threshold stall or a different mechanism; no evidence either way. celeris#284 (v1.4.8) is the same symptom class never reproduced in isolation, whose "suggested next step" — log per-attempt elapsed and failure mode — is still not implemented.
4. What the harness discards today (all read): h2c.go:286-295 calls recordHang(err, elapsed) which keeps only a cause class and a GLOBAL max elapsed (h2c.go:204-226); the mode, walker seed, err.Error(), conn.LocalAddr(), wall-clock instant and per-event dial/write/read durations are dropped; dial failures (h2c.go:242-246) and write failures (252-258) return with NO counter. ws.go: fireWSTorture runs dial+write+status+headers under ONE 2 s deadline (wsMaxHold, ws.go:192, 233-241) versus h2c's 20 s — so a 2-10 s server stall fails WS but not h2c; the non-101 status text is counted (ws.go:272) but not kept; dial failures dropped (236-239). Neither counter is in runner.go's reactive fire() list (runner.go:953-975 covers only adv_wrong_accepted, h2c_crashed, ws_accepted_bad_frame, ws_hang_no_close, liveness, hang), so no incident dossier — no goroutine.pprof, /proc//stack, fd list, dmesg (forensics.go:31-114) — is ever taken at the instant. The refapp's stdout+stderr goes through remote/local.go:63-64 into superviseStderr's 80-line ring (liveness.go:376-380) which is attached ONLY on process death (liveness.go:457-475); the engine Warn/Error lines that would name a mechanism — epoll "accepted fd exceeds conn table cap; dropping" (loop.go:770), EMFILE/ENFILE (loop.go:740-748), io_uring "re-create listen socket" (worker.go:653) — are discarded for a live refapp (the refapp sets no Config.Logger, main.go:181-191, so they go to slog.Default → stderr). EngineMetrics.ErrorCount (engine/engine.go) is bumped on those paths (loop.go:747/753/766; worker.go:1293/1333) but NOT exported by debugvars (debugvars.go:262-290 exports only active_conns/adaptive_switches/engine) and not in the 1 Hz series (series.go:21-25: ts, goroutines, heap*, rss, accepted, closed, active). The io_uring SQ-full accept drop (worker.go:3586-3591) sets acceptRearmPending without bumping any counter. The cell document has no per-cell start/ready time (ValidationCellResult in report/schema.go; StartedAt is run-level, matrix.go:493), so the io_uring worker event loops freeze once a minute: session store sweep holds shard locks for seconds while the inline session Set blocks every worker (24h soak gate failure) #493 "stall at the refapp's start second every minute" phase signature cannot be tested even if a timestamp existed.
5. The gate (report/gate.go:303-326, 452-463) fails on h2c_hang and ws_handshake_fail totals and folds the cause counters into the message (causeSuffix) — correct behaviour, kept.

Measurement

Observable

Two things, in order. (A) Attributability: for every h2c_hang and ws_handshake_fail event the cell document must carry a per-event record (bounded ring of 16, modelled on #327's sse_early_errs at sse.go:73-79,107-145 / schema.go:618-623): UTC ts (RFC3339Nano), ms since refapp ready, walker seed+id, churn/torture mode, cause class, verbatim err.Error(), non-101 status line (WS), walker conn.LocalAddr() (the engine<->client join key from celeris#562), THIS event's dial/write/read durations and bytes read, plus validator-side GC pause since last fire (runtime/metrics) — and, alongside it, an incident dossier taken within 2 s (goroutine.pprof, /proc//stack, fd list, dmesg), the refapp stderr ring flushed to /refapp_stderr_tail.txt, celeris.engine_error_count added to /debug/vars and the 1 Hz series, and ready_at in the cell doc. (B) Rate/mechanism: count of events per 6 h directed cell per counter, their timestamps, and the series columns accepted/active/engine_error_count in the +/-10 s window around each event.

Where it runs

(A1) Unit, probatorium validation/ under -race on Linux: extend newFakeH2CServer (h2c_test.go:89-113) and newFakeWSServer (ws_test.go:163-217) with policies {silent 25 s, accept-then-close, RST, slow-101 at 3 s} and assert the per-event record fields. (A2) Apparatus negative control in a Linux container with --security-opt seccomp=unconfined --ulimit memlock=-1 (the imem1 rig at scratchpad/imem1/{drive.sh,run.sh}): one auth_session_ratelimit io_uring cell, concurrency 50, 20 min; at a known instant T a sidecar sends SIGSTOP to the refapp for 3 s (and once for 25 s), then SIGCONT. (B) Probatorium cluster weekend tier, scoped: VALIDATE_MATRIX=1 VALIDATE_MATRIX_REFAPPS=auth_session_ratelimit VALIDATE_MATRIX_ENGINES=iouring,epoll,std SOAK_DURATION=18h VALIDATE_TARGET=both VALIDATE_CONCURRENCY=50 -> cellBudget = 18h/3 = 6 h per cell (cmd/validator/matrix.go:403), both hosts in parallel (VALIDATE_PARALLEL=1). The dispatch currently exposes only duration/resume_from (matrix-weekend-tier.yml:15-30), so this needs either a workflow input or a mage Soak from the runner host; VALIDATE_GATE_EXPECT_CELLS must be 6 (not 48) for that run.

Procedure

1. Land the code changes (see code_changes_needed) with the unit tests; run go test -race ./validation/... ./report/... on Linux. 2. Run the fault-injected container cell: verify (a) the 3 s SIGSTOP produces ws_handshake_fail_timeout events whose ts is within [T, T+3 s], h2c_hang=0 (3 s < 20 s), sse early closes/timeouts at the same T, series accepted flat for 3 rows, a dossier dir incidents/-I-WS-HANDSHAKE with goroutine.pprof, refapp_stderr_tail.txt present; (b) the 25 s SIGSTOP additionally produces h2c_hang_timeout with per-event read elapsed ~20000 ms and I-H2C-HANG dossier; (c) with no injection over 20 min all new rings are empty and the gate still PASSes (guard test like deps: bump the all-go-deps group across 3 directories with 5 updates #327's fails if the rings are dropped from schema or gate). 3. Dispatch the scoped 18 h soak (6 h x 3 engines per host). 4. From the artifact: tally events per cell per counter; for each event join ts against the series and against the SSE/adversarial rings; compute (ts - ready_at) mod 60 s; read the dossier's goroutine.pprof for the worker goroutines' state; read refapp_stderr_tail for engine Warn/Error lines. 5. Compare iouring vs epoll vs std per host.

Expected if the claim is TRUE

TRUE = a server-side mechanism exists at the observed rate. In each 6 h iouring/epoll cell expect ~6 h2c hangs (5 walkers x 10/s x 21600 s x 2/3 non-RST reads = ~720k reads at ~1/120k) and ~6 WS failures (2 walkers x 6.67/s x 21600 s = ~288k at ~1/47.6k) with 95% bands roughly [2,12]; events CLUSTER in time across slices (h2c, WS, SSE early-close, adv_hang_until_timeout within the same few seconds); h2c per-event read elapsed ~20000 ms with cause=timeout and WS ~2000 ms timeout for a stall, OR small-elapsed eof/reset for a drop; series accepted flat (or active spiking) across the event window; goroutine.pprof shows worker goroutines off the io_uring_enter/epoll_wait syscall (blocked on a lock or running a sweep) for the stall class, or the listen worker idle with no accept SQE for the deaf-listener class; celeris.engine_error_count steps at the event for the fd-cap/EMFILE class; if #493-residual, (ts - ready_at) mod 60 clusters within a few seconds of one phase; std shows 0 (no worker event loop). The error string discriminates further: 'i/o timeout' = never answered; 'EOF' at ~10000 ms = server ReadHeaderTimeout closed an accepted-but-unread conn; 'connection reset by peer' small elapsed = fd dropped at accept.

Expected if the claim is FALSE

FALSE = apparatus/transport artefact (the ~1/9000 dial-loss class the SSE analysis already found on all engines). Events are uniformly spread, do not coincide with SSE/adv events, accepted never flattens, goroutine.pprof is unremarkable, engine_error_count does not move, the phase histogram is uniform, the new h2c/ws dial_fail and write_fail counters show a comparable rate on std, and the error strings are client-side (dial i/o timeout, write: broken pipe) with validator GC pause > 0 at the instant. Also possible: 0 events in 6 h x 3 cells per host (~2.2M h2c reads, ~860k WS handshakes total) which bounds the rate below ~1/1M and says the v1.5.11 pair was a rate that #574/#572-era changes removed — that is a measured statement, unlike today's 0-in-96.

Negative controls

Three: (1) the fault-injected SIGSTOP cell above (known instant, known duration) must produce exactly the predicted counters/records/dossier with timestamps within 2 s of T — this proves the capture path works before it is relied upon; without injection the same cell must produce empty rings and a PASS (proves no false positives; run it also with the rings assertion inverted to confirm the guard test fails when the fields are removed, per RULE: always run the negative control); (2) the std-engine 6 h cell in the same dispatch: std has no worker event loop and no fd-cap table, so a server-side stall/deaf-listener mechanism must NOT appear there while an apparatus artefact appears at the same rate; (3) unit fake-server policies assert the record content for each cause class so a future refactor cannot silently drop the error string again (the #327 guard-test pattern).

Cost

Unit tests: ~1-2 min per package under -race. Container fault-injection: ~30 min wall (2 x 20 min cells) on any Linux box with seccomp=unconfined. Directed cluster run: 18 h wall on both hosts in parallel (6 cells total, 3 per host), exclusive cluster time; if 0 events, one repeat at VALIDATE_CONCURRENCY=100 (h2c 10 walkers, WS 5) for another 18 h before bounding the rate — note that changes the load profile so its I-MEM verdicts are not comparable to the standard soak. msr1 is the runner host: nothing may touch it during the run. Code: ~1 day incl. schema bump and tests.

Not measurable, and why

(1) The two original v1.5.11 events: no error string, timestamp, stderr tail or dossier exists anywhere (artifact verified; /tmp/soak3 is empty; the run artifact 340 KB is downloadable but contains only the run document and I-MEM-1 dossiers per the memory's own audit). They stay unattributed forever. (2) Stalls shorter than the 2 s tallyCB interval (runner.go:1001) will have a dossier taken after the stall ended; only the per-event record + 1 Hz series catch them, and a <1 s stall cannot fail either walker anyway (WS budget 2 s). (3) Kernel accept-queue occupancy at the instant (ss -ltn / /proc/net/tcp) is not captured by forensics.go; 'deaf listener' vs 'worker stalled after accept' is inferred from elapsed+EOF patterns and goroutine state, not observed directly — adding an ss -ltnp snapshot to the dossier would close that but is a further change. (4) The validator shares the host and loopback with the refapp, so a validator-process scheduler stall is indistinguishable from a server stall from the walker's clock alone; the record's validator GC-pause field and the cross-slice clustering bound this but cannot exclude a host-wide stall (both processes freeze together). (5) A rate below ~1/1M per counter is not resolvable in one 6 h cell; only repeated directed runs can push the bound down.

Code changes needed

probatorium (all under validation/ unless noted): (1) h2c.go: recordHang(err, elapsed) -> recordHang(ev h2cEvent) with a mutex-guarded ring of 16 {ts, since_ready_ms, walker, mode, cause, err, local_addr, dial_ms, write_ms, read_ms, n_read, validator_gc_pause_ns}; count dial/write failures as h2c_dial_fail / h2c_write_fail (informational, not gated); snapshot() emits h2c_hang_events. (2) ws.go: same for recordHandshakeFail (+ status line text), ws_dial_fail / ws_write_fail; keep the 2 s wsMaxHold but split the dial deadline from the handshake read deadline so the record can say which leg expired. (3) runner.go Tier1Summary projection + report/schema.go: new H2CHangEvents / WSHandshakeFailEvents []struct fields next to SSEEarlyErrs (schema.go:618-623), schema minor bump; report/gate.go causeSuffix prints the first event's err/elapsed (informational). (4) runner.go tallyCB: fire(properties.IH2CHang.ID, ..., snap.H2CChurn.Hang > 0) and fire(properties.IWSHandshake.ID, ..., snap.WSTorture.HandshakeFail > 0) with RecordOnly=true so the cell runs on but a dossier is written; add the two Specs to properties/i_tier1_walker.go and registry.go. (5) liveness.go: expose the stderr ring via a tailSnapshot() on livenessTally; runner.go writeIncidentDossier writes refapp_stderr_tail.txt into the dossier and tier1 end-of-cell writes /refapp_stderr_tail.txt unconditionally (80 lines, bounded); optionally tee the refapp's merged output to /refapp.log with a size cap in remote/local.go. (6) refapp/internal/debugvars: export celeris.engine_error_count from EngineMetrics.ErrorCount; checker/poll.go Snapshot field; series.go add the column; record the ready-banner time as ready_at in the cell doc (runner.go/matrix.go). (7) .github/workflows/matrix-weekend-tier.yml: dispatch inputs refapps/engines mapped to VALIDATE_MATRIX_REFAPPS/ENGINES and VALIDATE_GATE_EXPECT_CELLS derived from the selection. (8) Tests: h2c_class_test.go / ws_test.go per-event assertions for silent/close/RST/slow policies; a gate guard test that fails if the event rings vanish from schema or the gated totals stop being gated. celeris (optional but raises signal): bump errCount when prepareAccept hits a full SQ ring (engine/iouring/worker.go:3586-3591 currently only sets acceptRearmPending) and expose fdCapDrops (engine/epoll/loop.go:129-135) through EngineMetrics.ErrorCount, so the exported counter moves on exactly the suspected deaf-listener paths.

Corrections from the adversarial review (binding)

Review 1: refuted

Why: Part (B) cannot discriminate its two hypotheses at the sample size proposed. The whole basis is ONE h2c event and ONE WS event; a single Poisson observation has a 95% rate interval of roughly [0.025, 5.6] x the point estimate, so the 'expected_if_true' band '[2,12] events per 6 h cell' is fiction — the honest expectation for a 6 h cell (720k h2c reads / 288k WS handshakes) is anywhere from 0.05 to ~11 events, and the pooled data already in hand (1 in 3 same-config cells, i.e. ~1/360k h2c and ~1/143k WS; or 1 in 5.8M h2c / 1 in ~1M WS if uniform across cells) gives lambda ~2 with P(0)=0.14, or lambda 0.04-0.3 with P(0)>=0.74. A 0-event result is therefore compatible with 'mechanism still present', 'mechanism removed' and 'apparatus artefact' alike, and the design explicitly plans to read 0 events as the FALSE branch ('bounds the rate below ~1/1M ... says the v1.5.11 pair was removed') when it bounds nothing the 96 routine cells have not already bounded. The std 'negative control' is powerless for the same reason: at lambda 0.04-0.3 std shows 0 under BOTH hypotheses (P>=0.74), so 'must NOT appear on std' is satisfied by chance. It is also mis-specified: the h2c GET / goes through session+ratelimit on std too (main.go:214-245 skips only /ws,/events), so a lock-hold stall WOULD show on std for h2c and not for WS. Part (A)'s key observable does not survive contact with the capture path: /debug/vars and /debug/pprof are served THROUGH the engine under test (debugvars.go:217 srv.Pre(v.Handler(), pprof.New())), so for the process-wide worker-freeze class the design says expected_if_true hinges on ('goroutine.pprof shows workers off the io_uring_enter/epoll_wait syscall'), the 500 ms poll and the 5 s pprof fetch both fail during the stall and the dossier holds goroutine.pprof.missing; celeris#493 was only proven because the diag build had a pprof SIDE-listener and a 200 ms ss sampler — neither is in this design (ss is explicitly deferred as 'a further change'). Even with a side listener, alertedCounters fires the dossier once per counter per cell and only after the walker's budget expired (>=2 s WS, >=20 s h2c, +<=2 s tallyCB), so for the observed 1.5-8 s stall class the h2c dossier is always post-stall and the WS one is post-stall for anything under ~4 s: the record says 'timeout', the dossier shows a healthy process, and 'server stall' vs 'host-wide stall' vs 'validator stall' (the design's own not_measurable #4) remains undecided — which is exactly the state today plus an error string. The SIGSTOP negative control does not validate what it claims: SIGSTOP freezes every thread (unlike the one-lock/one-worker mechanisms listed), returns .missing pprof by construction, cannot exercise the deaf-listener / fd-cap / SQ-full classes at all, and the 25 s variant trips the existing I-HANG oracle (4 x 5 s probes 2 s apart, liveness.go:243) into a hard fail that competes with the predicted I-H2C-HANG dossier for the same 30 s forensics budget. So passing the control proves the ring gets written, not that the next event will be root-causable. Minor premise errors: the weekend tier is now 64 cells/4 engines (adaptive) since 2026-09-13, so '3 engines / EXPECT_CELLS=6' is stale; the LocalAddr join key joins against nothing because the refapp logs nothing per-conn (io.Discard recovery logger, no engine per-conn log).

Change to the design: 1) Drop the 18 h directed run as a discriminator. State the rate honestly (one event => 95% interval [0.025,5.6]x) and pre-register a stopping rule on EVENTS, not hours: keep the instrumentation on in every routine soak and stop when >=10 attributed events have accumulated per counter, or when a pre-registered denominator (e.g. 5M h2c reads and 2M WS handshakes on the same config) has 0 events — only that bounds the rate below the v1.5.11 point estimate with power. A directed cell adds nothing a routine soak does not give for free once capture is unconditional. 2) Make the mechanism observable independent of the SUT: add a loopback net/http side-listener in debugvars (ephemeral port announced in the ready banner, the #493 diag pattern) serving /debug/vars + /debug/pprof, and point BOTH the 1 Hz property poll and the dossier at it; keep the engine-served path too, so 'engine path timed out, side path answered' is itself the process-wide-stall signal. 3) Take the dossier INSIDE the stall, not after: have the h2c/WS walkers raise a pre-fail probe when a read has gone >1 s (WS) / >3 s (h2c) with no bytes — goroutine dump via the side listener, ss -tnp/ss -ltnp (listen Recv-Q + the walker's LocalAddr on the server side: recvq>0 = accepted-unread, absent = never accepted — the discriminator that actually split 16/26 vs 10/26 in #493), and a /debug/vars conn-table snapshot to join the walker's LocalAddr against; fire per event (drop alertedCounters for these two, bounded to e.g. 16 dossiers per cell). 4) Replace SIGSTOP with mechanism-shaped fault injections behind an env gate in the refapp: (a) a route that holds one store shard's lock for N s, (b) an inline worker sleep of N s, (c) an accept-side drop (ulimit -n to force EMFILE, or an env that skips the accept re-arm once) — and assert that each yields the DIFFERENT expected classification (a/b: side-listener dump shows workers in Mutex.Lock / off the syscall, ss shows backlog; c: engine_error_count steps, listen Recv-Q>0 with idle workers). Keep SIGSTOP only as the host-wide case and set its duration <20 s so it does not collide with I-HANG. 5) Fix the std control: WS (middleware-free) is the engine-loop control; the h2c GET / is not, and must be compared per path. 6) Update the premises: 4 engines / 64 cells; include adaptive or exclude it explicitly; derive EXPECT_CELLS from the selection.

Review 2: refuted

Why: The design's rate/mechanism leg (B) and its main discriminators measure the wrong thing; the attributability leg (A) is fine in intent but the instruments it names cannot deliver the signatures it says will separate TRUE from FALSE.

1. The h2c channel is calibrated on an instrument that no longer exists. The v1.5.11 event was a timeout at the then 10 s read budget; the budget is now 20 s (validation/h2c.go:185-198, feat: per-handler async dispatch (sync/async per route + group, all engines × all protocols) #302). The strongest prior the design itself cites, celeris#493, was stalls of 1.5-8 s once a minute (issue text: "stalls of 1.5-8 s happen once a minute at the refapp's start second", goroutine-dump proven). A io_uring worker event loops freeze once a minute: session store sweep holds shard locks for seconds while the inline session Set blocks every worker (24h soak gate failure) #493-residual stall can NEVER produce an h2c_hang at a 20 s budget: the read simply returns late as declined. So the design's TRUE prediction "~6 h2c hangs per 6 h cell at ~1/120k, read elapsed ~20000 ms" extrapolates a 10 s-budget rate to a 20 s-budget walker, and its FALSE reading "0 h2c events bounds the rate below ~1/1M / fix(engine): clear the hand-off queue slots so a drained conn can be collected #574-era changes removed it" would be exactly what the mechanism being present looks like through the new instrument. Only the WS channel (still 2 s, ws.go fireWSTorture) is comparable across the two soaks. The walker records nothing about slow-but-answered reads (h2c.go:286-322 only branches on err/n==0), so the 1-19 s tail the prior predicts is invisible by construction.

2. Whole-server stalls >= 20 s are already a different oracle. I-HANG is 4 consecutive 5 s /healthz timeouts at 2 s spacing (liveness.go:243-247) = ~20 s, the same instant the h2c read budget expires, and I-HANG is a HARD fail: writeIncidentDossier + captureForensics then cancel() and abort (runner.go:640-660; only RecordOnly incidents keep the cell running, 629-638). Consequences: (a) the proposed 25 s SIGSTOP negative control trips I-HANG at ~T+20-22 s and aborts the cell — it validates the hang oracle, not the h2c capture path; (b) an h2c_hang_timeout in a cell that did NOT also I-HANG (the v1.5.11 case) implies a PARTIAL stall (some workers) or a deaf listener, i.e. exactly the class a whole-process SIGSTOP does not model. The control injects the wrong fault class for the surviving hypothesis space.

3. The dossier cannot discriminate the prior-supported class. The design routes dossiers through tallyCB fire(), which is deduped "AT MOST ONCE per counter per run" (runner.go:922-940) — one dossier per cell, not one per event, so "events CLUSTER" and "read each dossier's goroutine state" are unmeasurable for events 2..n. Worse, for the 1.5-8 s io_uring worker event loops freeze once a minute: session store sweep holds shard locks for seconds while the inline session Set blocks every worker (24h soak gate failure) #493 class the WS event is recorded 2 s after handshake start and the tick adds up to 2 s more, so goroutine.pprof lands after the stall ended and is "unremarkable" — which the design lists as the FALSE signature. What actually proved io_uring worker event loops freeze once a minute: session store sweep holds shard locks for seconds while the inline session Set blocks every worker (24h soak gate failure) #493 was a 200 ms ss -ltn backlog sampler triggering goroutine dumps INSIDE the stall (memory absolute-gate-v159-release.md:160-171, stall-probe3); the design demotes exactly that instrument to "not measurable / further change".

4. The cross-slice time-join is not measurable with the proposed changes. The deps: bump the all-go-deps group across 3 directories with 5 updates #327 SSE ring the design models itself on is earlyErrs []string with no timestamp (sse.go:73-79, 138-144), and there is no adversarial ring at all (adversarial.go: bare atomic counters). The code_changes_needed add timestamped rings only for h2c/ws, so "join ts against the SSE/adversarial rings" — the headline TRUE-vs-FALSE discriminator — has no data on the other side of the join.

5. The validator-side confound is measured with the wrong metric. A 2 s WS timeout cannot come from a Go GC pause; the plausible client-side stall is scheduler/CPU starvation of the validator process sharing the host, or a host-wide freeze. runtime/metrics GC pause would read ~0 in exactly the case it is meant to flag, and no control ever injects a validator-side stall, so the record's client/server attribution is never validated in that direction.

Minor: the SIGSTOP(3 s) assertion "ts within [T, T+3 s]" is wrong even for the intended case — WS failures are recorded at deadline expiry, i.e. [T+0, T+5 s]; the weekend workflow now expects 64 cells (4 engines incl. adaptive), not 48.

Change to the design: Keep the per-event rings, stderr tail, engine_error_count export and ready_at (leg A), but change what the run measures and how the controls are built:

1. Measure the read-latency DISTRIBUTION, not only the timeout tail. In h2c.go and ws.go record every fire's dial/write/read elapsed into fixed buckets (<100 ms, <1 s, <2 s, <5 s, <10 s, <20 s, timeout) plus a bounded ring (16-64) of every fire whose read exceeded 1 s (ts, elapsed, outcome, err, local_addr). This makes the v1.5.11 10 s-budget event comparable (it is the >=10 s bucket), makes a io_uring worker event loops freeze once a minute: session store sweep holds shard locks for seconds while the inline session Set blocks every worker (24h soak gate failure) #493-residual 1.5-8 s stall visible as a once-per-minute burst in the 1-10 s buckets regardless of walker budget, and turns "0 h2c_hang" into a statement about the whole tail rather than about one threshold. Compute the mod-60 phase histogram over these slow-read events, not over hangs.

2. Replace the fire()-triggered dossier with a stall-triggered sampler that runs continuously in the cell: sample ss -ltn (Recv-Q on the refapp's LISTEN sockets) every 200 ms and, on backlog >= 2 for >= 400 ms, take goroutine.pprof + /proc//stack + the ss snapshot, rate-limited to N per cell and independent of tallyCB dedupe. This is the instrument that root-caused io_uring worker event loops freeze once a minute: session store sweep holds shard locks for seconds while the inline session Set blocks every worker (24h soak gate failure) #493 and it captures the state INSIDE a 1.5-8 s stall, which no post-event dossier can. Per-event dossiers via fire() may stay as a fallback but must not be the discriminator.

3. Fix the negative controls to the surviving hypothesis classes, not whole-process freeze: (a) partial stall — a validation-only refapp flag (e.g. -fault=store-lock:<dur>@<period>) that holds one session-store shard lock for 3 s (and once for 8 s) on a 60 s period, reproducing io_uring worker event loops freeze once a minute: session store sweep holds shard locks for seconds while the inline session Set blocks every worker (24h soak gate failure) #493's shape: healthz stays alive (no I-HANG), WS fails at 2 s, h2c reads land in the 2-10 s buckets, backlog sampler fires, phase histogram peaks; (b) deaf-listener/fd-cap — run the refapp under ulimit -n small enough that accepts hit EMFILE, asserting engine_error_count steps and the epoll fd-cap Warn line lands in refapp_stderr_tail; (c) keep SIGSTOP only at 3 s (below I-HANG), asserting WS timeouts in [T, T+5 s], and drop the 25 s SIGSTOP (it measures I-HANG). (d) validator-side control: SIGSTOP the VALIDATOR for 3 s and assert the events it produces are flagged client-side.

4. Replace the validator GC-pause field with a heartbeat-skew monitor in the validator (a 100 ms ticker whose observed gap is recorded; any gap > 500 ms in the +/-5 s window of an event is stamped into the record) and/or runtime/metrics /sched/latencies:seconds; this is what actually discriminates a client-process or host-wide stall from a server stall.

5. Give the SSE and adversarial slices the same timestamped event ring (ts, err, elapsed, local_addr) so the cross-slice time-join exists on both sides; alternatively one shared per-cell event log (JSONL, bounded) that all four walkers append to.

6. Restate the rate expectation for the directed run per channel: WS (unchanged 2 s budget) inherits the v1.5.11/v1.5.9 rate and is the only counter whose 0-in-N is a bound on the old rate; h2c is judged on the >=1 s slow-read ring and the >=10 s bucket, never on h2c_hang alone. Set VALIDATE_GATE_EXPECT_CELLS to the actual cell count of the scoped dispatch (the workflow now assumes 64, four engines).

[FumingPower3925]

Measured 2026-09-13 (branch feat/hang-capture, commit 234a1ad)

Verdict
INCONCLUSIVE for the closure under test (the v1.5.11 h2c_hang/ws_handshake_fail pair): nothing was run on the cluster and no fault-injected cell was run, so the rate/mechanism question is untouched. CONFIRMS, by measurement against fault-shaped fake servers, that the apparatus now captures what the corrected design lists: a 1.2 s server close produces a ring record with err='EOF', read_ms>=1200 and both addresses; a 1.1 s answered read is kept as 'declined' (the class the hang counters are blind to); a silent WS server produces handshake-fail-timeout with the READ leg identified as the one that expired (dial/write <=500 ms, read ~2000 ms, timeout bucket=1); fast fires stay out of the ring (0/10 false positives across both walkers); the 8-bucket edges are exact; the ring holds the last 32 of 101 in order; the stderr tail is readable while the refapp is alive (80 of 163 lines); the gate message names the first failed record; all under Linux -race with 0 races. DIRECTED 6 h STRESS (asked): implementable as a walker mode — a tier1Config knob (e.g. WalkerMix 'h2c,ws') in driveTier1's fan-out setting markovCount=1, adv/sse=0, h2c/ws from Concurrency with a ~20 ms tick (about 20 lines in tier1.go) plus plumbing Config->Orchestrator->tier1Config, a VALIDATE_WALKER_MIX env in cmd/validator, and workflow inputs (refapps/engines/mix) with VALIDATE_GATE_EXPECT_CELLS derived from the selection (the weekend tier now assumes 64 cells / 4 engines). What it needs: (a) auth_session_ratelimit only — the only refapp routing /ws; (b) a fresh-conn rate ceiling: every fire is a client-closed TCP conn, so ~28k ephemeral ports / 60 s TIME_WAIT caps ~450 fires/s before EADDRNOTAVAIL (now visible as h2c_dial_fail / ws_dial_fail); today's concurrency-50 cell is ~50 h2c/s + ~13 ws/s, a naive 20+10-walker/20 ms mode would be ~1500/s and exhaust ports; (c) per-path comparison, since h2c GET / traverses session+ratelimit (429s at high rate; the ratelimit shard lock is the #493 mechanism, so on-target) while /ws bypasses both; (d) its I-MEM verdicts are not comparable to the standard soak; (e) per Review 1 it does not discriminate the rate hypothesis (one event => 95% interval [0.025,5.6]x), so its honest use is the fault-injection negative controls, which additionally need the refapp -fault=store-lock:@ flag, a ulimit -n launch option, a 3 s SIGSTOP sidecar, the 200 ms ss -ltn backlog sampler and a debugvars side listener for /debug/pprof — none of which are in this PR.

Numbers (from assertion output)
go vet ./... : exit 0 (also -tags mage). go test (darwin, foreground): run go-test-full-2: validation ok 121.719s, report ok 0.898s, cmd/conformance ok 0.439s, cmd/loadgen ok 0.627s, cmd/observer ok 2.351s, cmd/runner ok 2.510s, cmd/smoketest ok 1.732s, cmd/validator ok 3.055s, cmd/validator-checker ok 2.057s, cmd/validator-replay ok 2.618s, validation/checker ok 3.031s, validation/properties ok 9.039s; debugvars module ok 0.464s; mage root ok 0.372s; auth_session_ratelimit refapp builds. FINAL gate run (go-test-final.log): report ok 0.562s; validation FAIL 139.706s on exactly ONE test, TestRun_HardFailCapturesForensicsBeforeTeardown (20.00s, 'hard fail must surface as the I-PANIC error, got '); every other validation test passed. Reruns of that test: 3/3 PASS (3.25s, 3.25s, 2.44s); count=10 on my tree: 10/10 PASS (28.0s); count=10 on the unchanged base tree 6051f4a (via stash): 10/10 PASS. Overall on my tree: 1 failure in ~24 runs of that test. Linux docker -race (golang:1.27, seccomp=unconfined): validation ok 22.909s, report ok 1.061s, 70 '--- PASS', 0 '--- FAIL', 0 'DATA RACE', exit=0 (no engine ran in the container; only harness tests against fake servers, so no 'workers=' line applies). New-test assertions (darwin and Linux -race identical): TestLatencyBucketsEdges 16/16 edge cases (99ms->lt_100ms, 100ms->lt_1s, 999ms->lt_1s, 1s->lt_2s, 2s->lt_5s, 5s->lt_10s, 10s->lt_20s, 20s->ge_20s, timeout regardless of elapsed); TestSlowFireRingIsBoundedAndOrdered: 101 adds -> 32 kept oldest-first, total=101; TestFireCaptureThreshold: 999 ms read NOT in ring, 1000 ms read in ring with err 'read tcp: i/o timeout', local/remote addr, dial_ms=2 write_ms=1, read.timeout bucket=1; TestFireH2CChurn_SlowCloseLandsInRingWithError (1.20s): h2c_hang=1 hang_eof=1, ring len 1, outcome=hang-eof, err='EOF', read_ms>=1200, n_read=0, read.lt_2s=1, dial/write legs filed, max_elapsed>=1200; TestFireH2CChurn_SlowButAnsweredLandsInRingAsDeclined (1.10s): h2c_hang=0 declined=1, ring outcome=declined err='' n_read>0; TestFireH2CChurn_FastFireDoesNotLandInRing: 5 fires declined=5, ring nil, slow_reads_total=0, read.lt_100ms+lt_1s=5, timeout=0; rst-before-read files dial+write only; TestFireH2CChurn_DialFailureIsCountedAndBucketed: dial_fail=1, only dial leg filed, hang=0; TestFireWSTorture_SlowRejectLandsInRingWithStatus (httptest, 1.20s): handshake_fail=1 status=1, ring outcome=handshake-fail-status status='HTTP/1.1 400 ...', read_ms>=1200, err=''; TestFireWSTorture_SilentServerLandsInRingAsTimeout (2.00s): handshake_fail_timeout=1, ring err contains 'i/o timeout', dial_ms<=500 write_ms<=500 read_ms>=1900, read.timeout=1 dial.timeout=0, fire bounded under 3s; TestFireWSTorture_FastHandshakeDoesNotLandInRing: 5 upgrades, ring nil, total 0; fast 400 also out of the ring; TestLivenessTailIsBoundedAndReadableWhileAlive: 163 post-ready lines -> 80 kept, pre-ready lines excluded; TestHeartbeatMaxGapSince: gaps <500ms dropped, window query correct, nil-safe, ring bounded at 64; TestGate_HangViolationCarriesTheFirstSlowFire: 2 events -> exactly 2 violations, h2c message carries '(timeout=1)' + ts + read=20000ms + outcome + err + local addr and skips the answered slow read; pre-5.9 doc message unchanged; TestTier1SummaryCaptureSurvivesJSON: keys h2c_latency/ws_latency/h2c_slow_reads/ws_slow_reads/ready_at/refapp_stderr_tail present, 8 buckets present, h2c_hang and ws_handshake_fail still in gatedTier1Keys, the 6 new informational counters NOT gated, empty summary omits all capture keys; TestTier1SummaryExportsEveryTallyField (existing guard) passes with the new fields; TestSeriesWriterRecordsEveryColumn passes with engine_error_count=7 as the 14th column; TestDebugVars_DocumentShape passes with celeris.engine_error_count numeric.

Negative control
Run and reported at three levels. (1) Threshold: fires whose read is under 1 s must NOT enter the ring — asserted for h2c (5 fast 200s + rst-before-read: ring==nil, h2c_slow_reads_total==0, histogram still files all 5 in lt_100ms/lt_1s) and for WS (5 fast 101s + one fast 400: ring==nil, total==0); unit level: a 999 ms read is absent, a 1000 ms read is present. (2) Arming: an unstamped tally leaves since_ready_ms at 0 (no 1970-relative value); a stamped one reads ~90000 ms. (3) The one test failure in the final gate: the same test 10/10 on the unchanged base tree and 10/10 on my tree, and the fixture (404 on /ws and /, so the WS slice is probed absent and h2c is 'declined') cannot trigger either new record-only fire — so the failure cannot be attributed to the change, and equally the 10-run controls cannot exclude a rare interaction.

Found on the way (not the closure under test)
(1) TestRun_HardFailCapturesForensicsBeforeTeardown (validation/incident_forensics_test.go:90) failed once in ~24 runs on my tree with 'hard fail must surface as the I-PANIC error, got ' after the full 20 s cell budget, i.e. the property loop's first I-PANIC violation was never delivered; 0/10 on base 6051f4a, 0/10 on my tree in isolation, 3/3 on rerun. Mechanism not found: the property loop sends only v.First with a non-blocking default: drop into a cap-1 channel (propertyloop.go:308-319), so any earlier occupant of the channel loses the hard fail for the whole cell; the fixture cannot trigger my new record-only fires, so I could not attribute it to this change. Worth a directed look (log the dropped send). (2) Design items NOT implemented here (scope): the 200 ms ss -ltn backlog-triggered sampler (Review 2 item 2), the debugvars loopback side listener for /debug/pprof (Review 1 item 2), refapp fault-injection flags (store-lock / ulimit -n / SIGSTOP 3 s), timestamped rings for the SSE and adversarial slices (Review 2 item 5; sse_early_errs still has no timestamp), workflow dispatch inputs for refapps/engines/EXPECT_CELLS, and the celeris-side ErrorCount bumps on the io_uring SQ-full accept path (worker.go:3586-3591) and epoll fdCapDrops — without those two, celeris.engine_error_count does not move on exactly the suspected deaf-listener paths. (3) Note on behaviour: the new I-H2C-HANG / I-WS-HANDSHAKE incidents are deduped once per counter per cell (alertedCounters) and skip gcore (core.skipped marker) so the dossier cannot pause the refapp and manufacture the next event; RecordOnly property incidents (I-MEM-1) still take gcore mid-cell as before.

Not measured, and why
No cluster run, no soak, no fault-injected container cell (no SIGSTOP / store-lock / EMFILE), so: engine_error_count was verified present and numeric (0) via debugvars_test but never observed to STEP; validator_skew_ms (heartbeat) never observed nonzero; since_ready_ms only unit-tested against a synthetic stamp; the record-only dossier path for I-H2C-HANG / I-WS-HANDSHAKE (refapp_stderr_tail.txt + core.skipped + pprof) was not driven end-to-end through Orchestrator.Run against a slow refapp (only the pre-existing RecordOnly I-PANIC path runs under -race); refapp_stderr_tail was not observed carrying a real engine Warn line (no engine ran); no real celeris engine was exercised at all (docker ran harness unit tests against fake TCP/httptest servers, so no 'workers=' evidence exists or applies); the dial-timeout branch of isDeadline was skipped on this host (blackhole dial did not time out in 1 ns) — only the conn-deadline and context shapes were asserted; the mod-60 phase histogram is an offline computation over ts/since_ready_ms and was not implemented or run. Logs: scratchpad/logs/{go-test-validation-report.log, go-test-full-2.log, go-test-final.log, docker-race-linux.log, hardfail-rerun.log, hardfail-x10-mytree.log} in the worktree (untracked, not committed); base-tree x10 log at the session scratchpad hardfail-x10-basetree.log.

Adversarial review of this measurement

Skeptic 1 — refuted: True, accepts as measured: False
The closure the issue defines is "the next h2c_hang / ws_handshake_fail is root-causable from the artifact alone, verified by a fault-injected negative control before the next soak" plus the binding review corrections (side listener, in-stall ss sampler, mechanism-shaped fault flags, std control, SSE/adv rings, inverted guard run). None of that was run: no fault-injected cell, no inverted-guard run, no soak, engine_error_count never observed to step, the I-H2C-HANG/I-WS-HANDSHAKE record-only dossier path never driven end-to-end; the agent's own verdict is INCONCLUSIVE. Worse, one of the three reported negative controls never executed: validation/walker_capture_arm_test.go carries a _arm_test.go suffix = GOARCH=arm build constraint, go list puts it in IgnoredGoFiles, and TestArmWalkerCaptureStampsReadyAndHeartbeat appears in none of the six logs (darwin arm64 or Linux amd64 -race), yet the report asserts its since_ready_ms results as measured. So armWalkerCapture / ready_at stamping / since_ready_ms / validator_skew_ms (the heartbeat the corrected design mandates) have no passing assertion anywhere. The Linux -race run was a -run regex subset, not the ./validation/... ./report/... the issue's step 1 requires. The final foreground gate FAILED (1 test) while the commit headlines the earlier green run. The logs are not committed on the branch (only untracked in the worktree); the debugvars-module run has no log. What IS sound: the branch matches the 25-file report, the ring/histogram/gate/schema unit tests exist in source, the reported numbers correspond to their t.Fatalf assertions, exposure was real (1.20 s / 1.10 s / 2.00 s test durations match the fake-server holds), fast-fire controls genuinely discriminate (ring must be nil while the histogram is filed), and no SKIP was counted as PASS (the report even understates: TestIsDeadline's dialer branch PASSed on both hosts). That confirms walker-level capture, not the closure.

What would complete it: (1) Rename validation/walker_capture_arm_test.go (e.g. walker_capture_arming_test.go), confirm it appears in go list -f '{{.TestGoFiles}}' ./validation/, run it with -v on darwin and under Linux -race, and add a CI/vet guard that fails on any *arm_test.go / *_test.go filename in the repo. (2) Run the full go test -race ./validation/... ./report/... on Linux, not a -run subset, and commit the logs under scratchpad/logs/ on the branch (or state that they are worktree-only). (3) Execute the issue's negative controls as written: a fault-injected container cell (3 s SIGSTOP below I-HANG, plus the refapp -fault=store-lock:@ and ulimit -n EMFILE variants) asserting ws_handshake_fail_timeout ts within [T, T+5 s], ring records with err/addresses, incidents/-I-WS-HANDSHAKE with refapp_stderr_tail.txt and core.skipped, engine_error_count stepping in the 1 Hz series; an un-injected 20 min cell with empty rings and a PASS; and the guard test run inverted (drop the ring fields) to show it fails. (4) Implement the remaining binding corrections before claiming the corrected design: debugvars loopback side listener for /debug/vars + /debug/pprof, 200 ms ss -ltn backlog-triggered sampler, timestamped SSE/adversarial rings, workflow inputs deriving VALIDATE_GATE_EXPECT_CELLS. (5) Then dispatch the routine soak with capture on and pre-register the event-count stopping rule from Review 1; only an attributed event (or the pre-registered zero-event denominator) closes the issue.

Skeptic 2 — refuted: True, accepts as measured: False
The closure celeris#588 defines is not measured. Its measurement has three legs — (A1) unit tests under Linux -race, (A2) a fault-injected Linux container cell whose corrected negative controls are a store-lock flag, ulimit -n EMFILE, a 3 s SIGSTOP and a validator-side SIGSTOP (plus the 200 ms ss -ltn sampler and a debugvars side listener, both binding per the reviews), and (B) a run-level count with a pre-registered stopping rule on events. Only A1 ran. No container cell, no soak, no fault injection, no ss sampler, no side listener, no refapp fault flags, no workflow inputs; the record-only I-H2C-HANG / I-WS-HANDSHAKE dossier path (SkipCore, core.skipped, refapp_stderr_tail.txt into the dossier) has zero test coverage (no *test.go references SkipCore or the new fires), and engine_error_count was never observed to step. So the apparatus claim 'the next occurrence is attributable from the artifact alone' is verified only for the ring/histogram at the walker level against fake servers, not for the dossier leg and not against any real engine. The agent's own verdict says INCONCLUSIVE, and the numbers support exactly that. The numbers themselves are genuine and match the untracked logs in the worktree (branch 234a1ad = worktree HEAD; darwin first pass FAIL only on the 5.8 schema pin; full-2 all ok, validation 121.719 s; final gate FAIL solely on TestRun_HardFailCapturesForensicsBeforeTeardown 20.00 s 'got '; reruns 3/3, x10 mine ok 28.044 s, x10 base ok 25.901 s; docker log 70 top-level PASS + 39 subtests, 0 FAIL, 0 SKIP, 0 DATA RACE, ok validation 22.909 s / report 1.061 s). Exposure at the unit level is asserted in code (h2c_hang=1/hang_eof=1, ring len 1, err=='EOF', read_ms>=1200; ws handshake_fail_timeout=1 with dial/write<=500 ms and read>=1900 ms; 5 fast fires leave ring nil and slow_reads_total=0), so the threshold negative control did run and invert. Discrepancies: (1) logs are NOT committed on the branch, only untracked in .claude/worktrees/wf_fa89913b-936-5/scratchpad/logs/; (2) the docker log has no command header, so -race, the image, seccomp and memlock flags are unverifiable from the artifact; (3) the report says the isDeadline dial-timeout branch was skipped on darwin, but both logs show '--- PASS: TestIsDeadline' with 0 SKIPs, i.e. the branch ran (report understates); (4) no log backs the debugvars-module / mage / refapp-build claims; (5) the issue's explicit guard-inversion negative control ('run it also with the rings assertion inverted') was never run — TestTier1SummaryCaptureSurvivesJSON was only shown passing; (6) 'no real celeris engine was exercised' is slightly wrong: debugvars_test starts a celeris.Std server on darwin, which is where engine_error_count=0 came from (still no epoll/io_uring engine); (7) DebugVarsKeys lost a space ('celeris.engine,celeris.oldest…') — doc-only constant, cosmetic; (8) the two new record-only fires add occupants to the cap-1 violations channel behind which propertyloop's default: drops a hard-fail v.First — an unmeasured risk the 1-in-~24 flake sits next to (0/10 base, 0/10 mine in isolation cannot exclude it).

What would complete it: 1) Commit the logs under scratchpad/logs/ on the branch with the exact command lines at the top of each (docker run flags, -race, image) so the Linux -race claim is auditable. 2) Run the guard inverted: drop the h2c_slow_reads json projection (or the ring from Tier1Summary) and show TestTier1SummaryCaptureSurvivesJSON and TestGate_HangViolationCarriesTheFirstSlowFire FAIL, then restore. 3) Add an end-to-end test driving Orchestrator.Run against a fake refapp whose /ws answers after >2 s (and a fake h2c path that closes after >1 s) and assert incidents/-I-WS-HANDSHAKE exists with core.skipped, refapp_stderr_tail.txt and incident.json carrying the ring record, that the cell keeps running, and that a concurrent I-PANIC hard fail is still delivered (covers the cap-1 channel risk). 4) Implement the binding corrections that are missing: refapp -fault=store-lock:<dur>@<period> flag, a ulimit -n launch option, the 200 ms ss -ltn backlog-triggered sampler, the debugvars loopback side listener, timestamped SSE/adversarial rings, workflow refapps/engines inputs with EXPECT_CELLS derived, and the celeris-side ErrorCount bumps on the io_uring SQ-full accept path and epoll fdCapDrops. 5) Run the Linux container negative controls (seccomp=unconfined, memlock=-1, one auth_session_ratelimit io_uring cell, concurrency 50): store-lock 3 s and 8 s @60 s → WS timeouts in [T, T+5 s], h2c reads in the 2-10 s buckets, phase histogram peak, healthz alive; ulimit -n → engine_error_count steps and the fd-cap Warn line in refapp_stderr_tail; 3 s SIGSTOP of the refapp; 3 s SIGSTOP of the validator → validator_skew_ms>0 on the records; and a 20 min no-injection cell → empty rings, gate PASS. 6) Then keep capture on in every routine soak with the pre-registered stopping rule (>=10 attributed events per counter, or 0 events at 5M h2c reads / 2M WS handshakes on the same config) and judge h2c on the >=1 s ring and >=10 s bucket, WS on ws_handshake_fail, per path (WS is the engine-loop control; h2c GET / traverses session+ratelimit on std too).

[FumingPower3925]

Lane M, 2026-09-26: the capture is now proven on every engine in a fault-injected control (goceleris/probatorium#412); the cluster run that closes this is queued

What #412 does.

• It injects the io_uring worker event loops freeze once a minute: session store sweep holds shard locks for seconds while the inline session Set blocks every worker (24h soak gate failure) #493 shape at a known instant: PROBATORIUM_REFAPP_FAULT holds a lock every request on a path needs. /ws held 8 s fails the WS handshake; / held 40 s is an h2c_hang.
• mage ValidateFaultControl then judges the artifact alone against the refapp's own hold/release log. For each injected path it requires:
  • the gated counter moved;
  • a slow-fire record inside the hold (read leg expired, deadline error, both addresses, validator skew 0);
  • a dossier taken inside the hold whose goroutine dump names the lock holder and the blocked requests and whose stderr tail holds the hold line;
  • every un-injected path clean.

Three live runs (laptop container, linux/arm64, auth_session_ratelimit × {iouring, epoll, std, adaptive}) found three sets of capture defects. All are fixed in #412:

1. Run 1. The dossier was taken on the tally tick after the stall had ended, so its dump named nothing. A walker hard fail also ended every cell early.
   • Fixed with an in-stall dossier (taken while the read still waits), a socket snapshot first, and record-only walker oracles on a fault run.
2. Run 2 (at the celeris 9f4d89b pin). std named holder and waiters every time. The event-loop engines did not. On io_uring, epoll and adaptive a handler runs on its loop, so the stall parks the loops, and the dossier's /debug/pprof went through that engine:
   • io_uring's in-stall dump was written 2.9 s late, after the release, with no waiter;
   • inside the 40 s hold every profile timed out, the incident loop fell 28–37 s behind, and I-HANG ended the cell first;
   • the dossier's stderr tail was stale;
   • adaptive's first records were overwritten in the ring.
   • The capture as landed could not have root-caused a io_uring worker event loops freeze once a minute: session store sweep holds shard locks for seconds while the inline session Set blocks every worker (24h soak gate failure) #493-class stall on the engines this issue is about.
   • Fixed with a debug side listener in every refapp (net/http, outside the engine, announced on a debug addr= banner and used by the dossier's pprof leg), the text dump fetched first, a live stderr tail, and the first 16 failures pinned in the ring.
3. Run 3. Every in-hold dossier on every engine was written within 0.1 s of its trigger and named 2–35 blocked requests and the holder. The checker had one false negative left: it required the / hold line in a dossier taken before the / hold. That is fixed in the checker.
   • Re-judged, the run passes 4/4.
   • A fresh end-to-end run at perf(h2,overload): fix write-queue sharding + Reap GC storm + realize inline-response bypass (#406 #407 #408) #412's final head 5ef1943 passes too: fault control 4/4 and no-fault control 4/4. All 35 of its dossiers took their dump from the side listener. Corrected 2026-09-27: all 35 dossiers aimed their pprof leg at the side listener; 33 got a goroutine dump, all 33 within 0.1 s of their trigger (31 at +0.0 s, 2 at +0.1 s, DOSSIER-TIMING.txt); the 2 I-HANG dossiers got none (the refapp was already gone).

Supporting checks:

• mutants-588.sh at perf(h2,overload): fix write-queue sharding + Reap GC storm + realize inline-response bypass (#406 #407 #408) #412's head: 44/44 named tests pass and 25/25 mutants are killed, with failing-first tests on the base and on the previous checker.
• On linux, with every epoll loop held, the engine's own /debug/pprof does not answer and the side listener's dump names the holder and the blocked requests.

Evidence: evidence/measurements-585-587-588/lane-20260926/588/ (MANIFEST.txt; DOSSIER-TIMING.txt per run).

The event class, caught naturally and now attributed. Two of the no-fault legs caught a real ws_handshake_fail on the adaptive cell only, after its epoll→io_uring promotion (1 of 2983 at +22 s in run 1; 2 of 2971 at +26 s and +62 s in run 3). Run 1's artifact could not say more. Run 3's dossiers say:

• the connection was accepted by the refapp, and its request bytes were taken off the socket (server Recv-Q 0, every accept queue 0);
• both io_uring workers were runnable and nothing was blocked;
• no response came within 2 s.

That points at the adaptive hand-off lineage (#657/#624/#686), not at a handler stall. It is an inference from the artifact, not a root cause. Not filed: gh search issues for "handshake" and "ws_handshake_fail adaptive" finds only this issue, #633/#566, #662/#675 and #667, none of which covers it.

Close path. This issue closes when the queued cluster control passes. It is in evidence/_queue/cluster.tsv and has not been dispatched: matrix-nightly-tier.yml after #412 merges, refapps=auth_session_ratelimit engines=iouring,epoll,std,adaptive refapp_fault=/ws:8s@30s,/:40s@60s fault_expect_cells=8. That is 8 cells, about 35 min.

What the release soak must also show:

• Either no I-*-STALL, I-H2C-HANG or I-WS-HANDSHAKE dossier in any cell, or every one of them attributed by its ss.txt and side-listener dump, as above.
• On adaptive, which is where the only natural events so far have come from, any such event must carry pprof_source=side-listener in forensics_status.txt.
• A 0-event soak bounds the rate only at the denominator review 1 pre-registered (5M h2c reads / 2M WS handshakes). It does not prove absence.

Update 2026-09-27 (probatorium#412 review round 2): the stall capture (in-stall dossiers and the side listener) is now on only in a fault-control run (PROBATORIUM_REFAPP_FAULT) or with PROBATORIUM_STALL_CAPTURE=1, not in routine validation runs; and the checker now credits a dump to a hold only if the dump was FETCHED inside it (the dossier stamps stacks_from/stacks_to). Details on probatorium#412.

[FumingPower3925]

Checkptr tier 36283577085, 2026-09-27: a natural h2c_hang on adaptive/arm64, closed at the 10 s header deadline; mechanism not attributed

Run. probatorium run 36283577085 (checkptr tier) ran on probatorium main 6f5521c with celeris pinned at 9f4d89b.

• What passed: all 64 cells ran, with 93,405 property evaluations and 0 property violations. There were no checkptr, fatal, panic or race findings.
• What failed: one gated walker counter, tier_1.h2c_churn.h2c_hang=1, in auth_session_ratelimit/adaptive/arm64 (msr1, cell-07).
  • Validate job 108526811356, log lines 128498-128512.
  • ValidateDiff's four MED rows (lines 128526-128534) are this same event compared four ways. Their printed arch labels are wrong (ValidateDiff prints every row under the file-level arch headers: cross-engine rows read as cross-arch, a cross-arch row comes out with its arches inverted, and no row names its refapp probatorium#429).

The record. From artifact matrix-checkptr-results-36283577085, cell-07 tier1_tally.json:

• The connection: a fresh one, 127.0.0.1:32738 -> 127.0.0.1:6861. Dial took 0 ms and writing the GET / preamble took 0 ms. Then 0 bytes came back, and an orderly EOF arrived after 10,004 ms (read ended 01:55:46.600Z).
  • Outcome counters: eof=1, with timeout, reset and other all 0.
  • No validator_skew_ms, so the validator had no heartbeat gap of 500 ms or more.
• Read histogram for the cell:
  • 6,004 reads under 100 ms;
  • 0 between 100 ms and 10 s;
  • this one in the 10-20 s bucket.
• Timing: the write was at 01:55:36.596Z, 79.5 s after ready. That is 48.6 s after the epoll -> io_uring switch at 01:54:48Z (refapp log, 1 s resolution).

What the artifact and the code at the pins establish. This is the reading this issue pre-registered for "'EOF' at ~10000 ms": the header deadline closed an accepted connection before its request was parsed.

• The 10 s timer: the refapp sets Read, Write and Idle timeouts but no ReadHeaderTimeout (validation/refapp/auth_session_ratelimit/main.go:181-191 @6f5521c). So celeris's 10 s default applies (resource/config.go:269 @9f4d89b).
• The walker's budget: 20 s, so a server-side timeout arrives as a close (validation/h2c.go:212-224).
• How io_uring closes on it:
  • It arms the header deadline when it sets up the connection's protocol state (initProtocol, engine/iouring/worker.go:2190-2199).
  • handleHeaderTimer closes only while that deadline is still set (1796-1802), with a plain closeConn (1821). The checkTimeouts fallback does the same (5269-5270).
• Which engine held the connection: the io_uring engine, not the epoll standby.
  • All 12 1 Hz rows across the read window show engine_standby_active_conns=0.
  • engine_standby_close_count stayed at 277, unchanged since 01:54:59Z.
• The hand-off was quiet for over 30 s before the write:
  • transplant detached and adopted both 69, unchanged since 01:54:49Z;
  • sweep passes 313 and residual busy 0, unchanged since 01:54:59Z.
• Clean witnesses: every engine zero witness reads 0 (recv_double_armed, recv_cqe_unaccounted, recv_sq_full, recv_stall_episodes, close_missing_conn_state, adopt_slot_occupied). requests_error and requests_5xx are 0.

Excluded: a parked ring (#704).

• The cell averaged 53.7 h2c reads/s, about 537 in any 10 s, across 2 io_uring workers (refapp_stderr_tail.txt:9: workers=2).
• None of its 6,005 reads took between 100 ms and 10 s. A worker parked for a large part of those 10 s would have pushed the other reads it owned past 100 ms.

Not attributed. Three mechanisms fit, and this artifact cannot separate them:

• (a) celeris#685: a stale RECV left by a closed connection resolves the reused fd number and consumes this connection's request.
• (b) Unarmed recv: this connection's recv was never armed, or ended without a re-arm, so the bytes stayed in its Recv-Q.
• (c) Unparsed bytes: its own recv read the bytes, and they never reached the parser.

The stale counter does not single out (a). engine_stale_recv_data_closed=65 in this cell says nothing specific: the counter is nonzero in every auth_session_ratelimit io_uring and adaptive cell, whether or not a request was lost.

• This run: 37 (io_uring arm64), 26 (io_uring amd64) and 47 (adaptive amd64), none with a loss.
• Nightly 36247560882, same celeris pin: this same cell (adaptive/arm64) had 156, with 0 h2c_hang and 0 ws_handshake_fail.
• Race tier 36280496987, same pins: its loss cell, adaptive/amd64 with ws_echo_timeout=1, had 0.

_transplanted and _unattributed are 0 in this cell.

Why the artifact cannot say more.

• The dossier is post-stall. The I-H2C-HANG dossier was taken at 01:55:47.094Z, 0.493 s after the EOF. That is post-stall by construction, because the counter moves when the read leg ends. It has no ss.txt, and its goroutine dump shows the process after the connection was gone. Its incident.json snapshot is all zeros (Walker incidents write an all-zero snapshot into incident.json (PID 0, EngineName "", CheckptrBuild false) beside a forensics_status.txt that names the real pid probatorium#430).
• No stale read can be matched to this connection. engine_stale_recv_data_closed is a cell total with no series column (report/engine_counter.go:154-159 @6f5521c). celeris keeps no per-read record, so no stale read can be joined to local port 32738.
  • A 1 Hz column would only place a stale read to the second.
  • What would decide (a) is a bounded per-read exemplar from the engine (time, fd, peer address), joined to the walker's local_addr.
• An in-stall ss would help only partly. Recv-Q > 0 means (b). Recv-Q 0 cannot tell (a) from (c).
• feat(validation): the celeris#588 capture control -- inject an h2c hang and a WS handshake failure at a known instant, and prove the artifact names them on every engine (probatorium#351) probatorium#412 at its current head (22a5490) does not change this for a routine tier.
  • Its in-stall dossier (I-H2C-STALL at 3 s) and debug side listener turn on only when PROBATORIUM_REFAPP_FAULT is set or PROBATORIUM_STALL_CAPTURE=1 (validation/liveness.go:113-127 and runner.go:1443-1444 at that head).
  • At that head, ansible/validate.yml forwards only PROBATORIUM_REFAPP_FAULT to the cluster validator (line 252). None of the nightly, weekend, checkptr or race tier workflows sets PROBATORIUM_STALL_CAPTURE.
  • A routine run does gain ss.txt in its live dossiers (forensics.go:62-63). For an event like this one, though, it is taken after the connection has ended.
  • The release-soak criterion in the 2026-09-26 comment above needs dossiers attributed by ss.txt and a side-listener dump, with pprof_source=side-listener on adaptive. A routine soak cannot meet it at that head unless the soak turns the stall capture on.

Rate. One event cannot resolve a rate.

• In this cell it is 1 in 6,005 h2c reads, and in the run 1 in 178,901.
• The same cell in nightly 36247560882 had 0 in 6,080.
• This is the first h2c_hang this issue has recorded on an adaptive cell. The natural adaptive events in the 2026-09-26 comment were ws_handshake_fail.

Evidence. evidence/cluster-tiers-20260927/checkptr-36283577085/ (see MANIFEST.txt). The files that regenerate every figure above:

• scripts/census.py and scripts/checkptr_scan.py, which write census.txt and checkptr_scan.txt: the cell census and the clean checkptr scan;
• publish/numbers.py, which writes publish/numbers.txt: every other number;
• publish/code_cited.sh, which writes publish/code-cited.txt: the cited lines at 9f4d89b and 6f5521c;
• publish/fetch_pr412_files.sh, which writes publish/live/pr412-22a5490/cited-lines.txt: perf(h2,overload): fix write-queue sharding + Reap GC storm + realize inline-response bypass (#406 #407 #408) #412's lines.

[FumingPower3925]

Lane M, 2026-09-27 (probatorium#412 review round 3): the release soak can now carry the capture, and the natural events are #715

1. The release-soak criterion, amended

Before (probatorium#412 at ba0a3e0): the capture (in-stall dossiers and the debug side listener) turned on only with PROBATORIUM_REFAPP_FAULT or PROBATORIUM_STALL_CAPTURE=1, and only REFAPP_FAULT reached the cluster validator. As the 04:27Z checkptr comment above says, no routine soak could meet the criterion of the 2026-09-26 comment.

Now (probatorium#412 at 5a97133, on main cd8dedf):

• PROBATORIUM_STALL_CAPTURE crosses the same three hops as the fault: a mage extra-var, ansible/validate.yml's environment, the validator.
• The nightly, weekend, checkptr and race tiers take a boolean stall_capture dispatch input, default false: scheduled and routine runs stay off.
• The proof: three tests FAIL on the head before the change, and four mutants are killed (the call site gone, the playbook line gone, the soak step never setting it, the input defaulting on). ansible-core renders the validator's environment from the exact extra-var mage emits (probatorium#412 comment, round3/588/r3-20260927T100258Z-5b6b53a).

The criterion, amended. The release soak is dispatched with the capture on, after #412 merges:

gh workflow run matrix-weekend-tier.yml --repo goceleris/probatorium --ref main -f stall_capture=true

• The rest of the criterion stands: every I-*-STALL, I-H2C-HANG or I-WS-HANDSHAKE dossier attributed by its ss.txt and side-listener dump, and on adaptive pprof_source=side-listener.
• A soak dispatched without stall_capture cannot meet it and does not count toward it.
• The capture adds no gate: the in-stall dossiers are record-only, and the soak is judged by ValidateGate and ValidateDiff as usual.
• In a routine run (capture off) a record-only dossier no longer takes the text goroutine dump (probatorium#412 5b6b53a): runtime.Stack(all) is a stop-the-world in a cell that keeps running. The soak with stall_capture=true still takes it in every dossier.

What the capture costs a running cell, measured on the capture-on container runs (round3/588/DOSSIER-COST.txt, 121 dossiers with a dump): the goroutine?debug=2 fetch, an upper bound on its pause, median 1.6 ms, p95 6.2 ms, max 25 ms at up to 148 goroutines; ss plus the /proc snapshots median 16 ms, p95 41 ms (host-side, the refapp is not stopped). The pause grows with the goroutine count, so the soak's first dossiers are worth a look.

2. The natural events are #715

The 2026-09-26 comment said "not filed". #715 (bug, v1.6.0) is now the record for the class: seven events, all adaptive, all 20-73 s after the first epoll→io_uring promote of their cell (six from the lane's container no-fault legs, plus the checkptr h2c_hang above).

New since the 04:27Z comment: five of the seven were caught by an in-stall socket snapshot, and all five show the same state (round3/588/NATURAL-EVENTS.txt, SIGNATURE 5/5):

• the server socket, ESTAB and owned by the refapp, had received every byte of the request (bytes_received 126 or 156 = the client's bytes_sent), its Recv-Q was 0, and it had sent no data (no bytes_sent, no data_segs_out; the client's bytes_acked = bytes_sent + 1);
• every accept queue was 0;
• the side-listener dump, fetched 45-50 ms after the trigger where stamped, showed nothing blocked (no sync.(*Mutex).Lock frame).

That excludes (b), the unarmed recv, of the three candidates the checkptr comment listed. (a), a stale completion on a reused descriptor number (#685's class), and (c), bytes read but never handed to the parser or the async handler, remain; #715 says what would separate them.

3. The no-fault control on adaptive

The container no-fault control (capture on, nothing injected) failed its adaptive cell on #715 events in 3 of its 5 legs across the lane's rounds (NATURAL-EVENTS.txt, DENOMINATORS). The capture did not make them: the class occurred on the cluster with no #412 code, and each in-stall dossier was triggered by its event's own stall. There is no discriminating rate control, though (capture-off routine legs: 0 events in 4, two of them this round). This round's two container runs, at probatorium#412 5b6b53a and 1c00c1c, caught no #715 event in their no-fault legs. That makes 3 of 7 legs across the lane, and the count of seven #715 events stands. At 1c00c1c the capture control itself passed all three legs 4/4 (round3/588/20260927T110420Z-1c00c1c). The control that closes this issue is unchanged: the queued cluster fault run (refapp_fault), judged by ValidateFaultControl.

Evidence: evidence/measurements-585-587-588/lane-20260926/round3/ (MANIFEST.txt).

[FumingPower3925]

Closing: the capture is proven on the cluster. probatorium's celeris#588 capture control, the one probatorium#412 added, passed on bare metal on both arches. The run is 36325737189: main 6c3d7f1, refapp auth_session_ratelimit, refapp_fault=/ws:8s@30s,/:40s@60s, 8 injected cells (epoll, io_uring, std and adaptive on amd64 and arm64).

The step mage ValidateFaultControl succeeded:

ValidateFaultControl: PASS -- 8 injected cell(s): every injected stall was counted, recorded and named from the artifact; every un-injected path stayed clean.

• Each injected h2c hang and WS stall produced an in-stall dossier observed INSIDE the hold. For example, "I-H2C-HANG observed at +22.003s after the hold's start, dumped between +22.07s and +22.072s". Its goroutine dump names the request goroutines blocked in the fault handler (2 to 35 per dossier).
• Dossiers observed outside the hold window are correctly reported as unable to name the stall (for example "I-WS-HANDSHAKE observed at 8.001s, outside the hold [+0, +8s]"). So the checker cannot pass on a lucky late dump.
• The run's overall conclusion is red by design: mage Validate fails on the injected cells. It is recorded as a deliberate red run.

So the next natural h2c_hang or ws_handshake_fail can be root-caused from the artifact alone. The capture is gated to fault and stall-capture runs (probatorium#412's round-2 design). To root-cause the natural adaptive events of celeris#715, run a nightly with PROBATORIUM_STALL_CAPTURE=1 on the adaptive cells.