fix(cluster): close four-node PCM-X liveness campaign#63
Closed
sqlrush wants to merge 56 commits into
Closed
Conversation
added 30 commits
July 18, 2026 04:51
The runtime fail-closed transition was silent: pcm_x_runtime_fail_closed()
only flipped the gate and bumped a counter, so a fused node in a multi-node
wedge (t/400 L3) could not be attributed to any of its ~77 call sites, and
the t/400 probe only summed pg_cluster_state values across all four nodes.
- Record the winning call site ("file:line") in PcmXShmemHeader and expose
it as the text-valued pg_cluster_state key pcm_x_runtime_fail_closed_site;
emit one LOG line per fuse outside critical sections.
- Route every existing call site through a __FILE__/__LINE__ macro; call
sites are textually unchanged (source-scan asserts unaffected).
- t/400: per-node runtime probe (state / generation / recovery_blocked
deltas / fail-closed site) so the L3 diag names the fused node and arm.
- Unit: PcmXShmemHeader ABI pin 36400 -> 36480; new winning-site test;
CritSectionCount + accessor stubs for the standalone links.
t/024 L1, t/108 L1, and the unit exact-key-set dump test pin the pcm category row count (58 -> 59) and the exhaustive key list; the new pcm_x_runtime_fail_closed_site row joins both.
The static pcm_x_runtime_fail_closed() has ~170 zero-argument internal call sites in cluster_pcm_x_convert.c (the majority of all fail-closed arms; the previous commit only routed the extern seam). Rename the impl and mirror the __FILE__/__LINE__ macro file-locally so every internal arm records its site with the call sites textually unchanged; also fixes the stale forward declaration that broke the build.
…uges A wedged writer holds no observable state post-mortem (kill_kill), so the existing after-leg probe cannot distinguish where the stalled writers wait. Snapshot pg_stat_activity wait events and the live queue gauges per node at start_at+8s while the leg is still running; bounded, failure-tolerant, diag only.
Sampling every pcm key per node at start_at+8s and +12s and printing the per-node movement between the samples separates the live retry loop (still ticking) from the wedged pipeline stages (frozen) during a pure L3 stall.
… deltas The terminal-leg retry path stages IC frames without touching any pcm counter, so a frozen pcm fingerprint cannot distinguish a silent resend loop from a dead pump. Sample cluster aux process wait events and pg_cluster_ic_peers msg send/recv counts at both probe instants.
…dump A wedged tag whose terminal legs stall silently leaves no counter or log trace (the terminal path stages IC frames without stats). Emit one text-valued pg_cluster_state row per live master tag slot (FIFO shape: head/tail/active/queued bitmap) and per occupied ticket slot (state, leg bitmaps, reliable-leg opcode/phase/retry/responder) so the stuck leg and the missing ACK node are directly readable; racy-tolerant reads, rows present only while slots are occupied. t/400 mid-leg probe prints them.
Per the t/400 root-cause audit (pgrac scratchpad review): - The per-slot dump iterators now revalidate and copy under the matching master partition shared lock (the same lock every in-place field mutation holds) instead of a generation double-read, so a printed combination is a state that actually existed. - Every non-OK acquire_writer_impl exit records its source line; the nested-wait guard records the held content-lock tag and result that closed it. The bufmgr writer failure report prints fail_line and guard evidence, closing the silent BARRIER_CLOSED(13) client-escape attribution gap. - t/400 mid-leg probe samples at t+1/t+3/t+8 to catch the terminal handoff mid-flight, with movement printed per sample pair.
bufmgr.c keeps upstream pgindent formatting and is outside the cluster clang-format scope; restore it wholesale and re-apply only the writer failure-report evidence hunk.
…ines Completes the evidence surfaces for the t/400 L3 chain per the root-cause audit: - pcm_x_ltag_<i> rows: local tag slot flags/round/refs/drain generations under the local partition lock, exposing the foreign REVOKE_BARRIER shape on remote participants. - pcm_x_terminal_last_note row: the arm and result that terminated the most recent terminal kick (detach / leg arm / stage / local retire apply / ack resolve) with a same-shape repetition count, closing the counter-free terminal path. - The 33 goto requester_fail_closed sites now record their source line too, so a client-visible failure names the arm even through the fail-closed fallthrough (this run proved cleanup PRESERVE_FAIL_CLOSED at cluster_gcs_block.c:8226 fusing node2 after a STALE acquire). - PcmXShmemHeader 36480 -> 36504 with pins synced.
…VOKE_BARRIER t/400 L3 four-node hot-block dead-pump: a fully-DRAINED external holder/blocker terminal (older granted cohort) could not retire its own holder lane because the (REVOKE_BARRIER && !same_ref_dual) guard in pcm_x_local_retire_candidate_at and pcm_x_local_holder_detach_terminal_exact treated a newer QUEUED writer REVOKE_BARRIER as blocking. The RETIRE watermark never advanced (retire_acked stuck at master own bit) and the FIFO wedged. Exempt a cross-lane older holder terminal via pcm_x_local_cross_lane_holder_terminal_retirable (writer_flags==0 so not same-ref dual, HOLDER_TERMINAL_MASK fully drained, grant_generation!=0); retiring it clears HOLDER_TERMINAL_MASK so the empty-frozen-round path drops REVOKE_BARRIER and promotes the writer, while detach_terminal_common preserves every writer ref/round/barrier/FIFO field. TDD RED->GREEN: test_local_cross_lane_holder_terminal_retires_under_revoke_barrier. Spec: spec-2.36a
…ghten exemption to distinct tickets The t/400 cross-lane regression test previously modelled the newer writer by registering an S holder and OR-ing REVOKE_BARRIER by hand, leaving tag_slot->ref zero and the writer-preservation assertions vacuous. Rebuild it through the production chain (join -> claim -> revoke-cutoff -> ENQUEUE -> ADMIT_ACK/CONFIRM, then the older cohort's grant-time PROBE / blocker ACK / DRAIN), pin the closed round of one, the ticketed writer ref and the FIFO/round/reliable bytes across the retire, and drive the unwedged writer through its full production grant chain to the queue baseline. Tighten pcm_x_local_cross_lane_holder_terminal_retirable() to take the tag slot and refuse a writer-lane ref that aliases the external ref or collides with its master-unique ticket id: such states cannot be an older cohort and keep the pre-exemption NOT_READY verdict. Deliberately no ticket-order requirement beyond that -- a cancelled predecessor leader's duplicate-ACK tombstone legally lingers in tag_slot->ref with an older ticket until the successor's ENQUEUE arm retires it, and refusing there could wedge the empty-frozen path permanently if the successor exits before arming. Verified: rebuilt test RED against pre-fix production (retire NOT_READY, HOLDER_TERMINAL_MASK stuck, drain generation retained); new defense-probe test RED against the flags-only helper (alias retired as OK); both GREEN after the change; cluster_unit 182 binaries all green; check-format clean.
… preflight Any non-OK remote-reservation preflight used to fail the whole PCM-X runtime closed (t/400 fuse family, requester exit line 8854). BUSY there only means a revoke/grant lifecycle flag is still live on this node's own tuple -- ordinary contention the next iteration re-snapshots -- so route it through the requester retry table as a wait, never a runtime fuse. STALE keeps the fail-closed verdict for now: it means an interleaved revoke consumed the queued identity's enqueue-time base_own_generation, which the grant/final-ack chain hard-pins at every hop (master final-ack prepare/finalize, local final-ack arm, writer-holder authority, bufmgr claim); absorbing it needs a master-visible rebase amendment. Capture the preflight own-tuple evidence (live/base generations, pcm_state, flags) and print it in the requester fail-closed LOG so the next t/400 run names the exact STALE arm. Verified: retry-table case RED (BUSY mapped fail-closed) then GREEN; driver source-scan pin RED (no preflight retry dispatch) then GREEN; cluster_unit 182 binaries all green; full macOS build clean; check-format clean.
…ALL_READY grant rebase A queued PCM-X writer that held S/X at ENQUEUE loses its enqueue-time base_own_generation when an interleaved revoke commits while it waits; the grant then arrived un-installable (reservation preflight STALE) and fused the whole runtime -- the t/400 clean-N drift fuse (evidence: live_gen=base+1, pcm_state=N, flags=0). Rebase design (A'): the immutable PcmXWaitIdentity and every directory key never change. The requester publishes the live clean-N generation exactly once as an effective grant base -- locally pre-reservation (cluster_pcm_x_local_grant_rebase_publish_exact) and to the master on the INSTALL_READY V2 frame (trailing rebased_own_generation, 104->112). Master and local planes persist it independently (ticket/tag slot grant_base_own_generation) with identical one-shot semantics: same-value replay is DUPLICATE, a different second value is evidence divergence and fails closed. Every exact committed==base+1 proof now runs against the effective grant base under its owning lock (master final-ack prepare/finalize, local final-ack arm, writer-holder authority, bufmgr grant-snapshot binding via the claim copy, image-fetch reservation check); the wire ingress checks relax to a monotonic floor only. Mixed-version safety: PGRAC_IC_HELLO_CAP_PCM_X_REBASE_V1 (0x400) is advertised unconditionally, but publication activates only when the whole bound formation advertised it at activation (runtime-lifetime flag, sampled by the formation tick). The V2 frame exists on the wire only to carry a nonzero rebase; every other INSTALL_READY stays the V1 104-byte exact frame, and receivers accept both exact lengths. Without coverage a drifted requester keeps the fail-closed STALE verdict, now with rebase_active in the preflight evidence LOG. Verified: master one-shot/conflict/no-drift and local publish/effective tests RED under neutralized persistence or effective helper, then GREEN; HELLO reference bytes and capability-set tests RED on the new bit, then pinned; ingress dual-length and monotonic-floor arms covered; layout pins updated (ticket slot 384->392, local tag slot 752->760, progress 240->248, writer claim 144->152, shmem header 36504->36512, INSTALL_READY 104->112 with V1-prefix offset). cluster_unit 182 binaries all green; full macOS build clean; check-format clean.
…nstead of a client ERROR t/400 L3 item 3. A writer holding a heap content lock whose tag sits under a frozen REVOKE_BARRIER hits the nested-wait guard when its pre-crit VM LockBuffer enters the PCM-X queue (heap_update -> LockBuffer(vm) -> fail_line 8555, captured live via backtrace_functions in t/400): the refusal escaped as an OBJECT_IN_USE ERROR, killing a pgbench client and violating the S3 errors=0 gate. The refusal is now caller-owned: - bufmgr: cluster_bufmgr_pcm_x_writer_prepare() grows a barrier_refused out-param; LockBuffer body becomes LockBufferInternal() and the new ClusterLockBufferExclusiveBarrierAware() returns false with nothing held on BARRIER_CLOSED (all other paths keep the historical error surface). bufmgr never releases the outer foreign content lock itself. - heapam: the reachable pre-crit VM sites unwind their OWN heap lock(s), resolve the map-page conversion via cluster_heap_vm_barrier_warm() while holding no content lock (blocking legal, node X retained by hold-until-revoked), then re-enter a proven point: heap_update requalifies at l2 when the old tuple carries no temporary lock (ITL pick idempotent, unstamped undo record unreachable), re-enters the stock page-reacquire loop when the lock-only xmax already shields it (foreign newbuf unlocked and unpinned so RelationGetBufferForTuple re-picks), and heap_delete requalifies at l1. Bounded by CLUSTER_HEAP_VM_BARRIER_MAX_RETRIES, after which the plain LockBuffer restores the fail-closed ERROR. heap_insert/heap_multi_insert/heap_lock_tuple keep the explicit fail-closed ERROR: their pre-crit VM shape is unreachable under the t/400 and S3 workloads (no all-visible insert targets without vacuum, no FOR UPDATE); follow-up may extend the same arm. - observability: pcm_x_queue_barrier_unwind_count (pg_cluster_state pcm row 60); PcmXStats grows 176->184, header 36512->36520, and PCM_X_SHMEM_LAYOUT_VERSION bumps 12->13 covering the A' slot growth that had landed without a version bump. TDD: source-scan pin test_precrit_vm_barrier_refusal_unwinds_to_caller RED (not ok 18) before the arms existed, GREEN after; full cluster_unit suite RC=0 with zero not-ok; check-format 0 violations. e2e t/400 errors=0 verification follows with the queued review-gap fixes.
…ame-round writer claim t/400 L3 review P0-1 + P1-1. cluster_pcm_x_local_writer_claim_exact() unconditionally zeroed claim_out->grant_base_own_generation, so only the leader (whose claim is backfilled after the preflight rebase publish in the gcs requester) carried the effective base. A same-round FIFO follower claiming after the leader's one-shot rebase kept its enqueue-time identity base and the bufmgr grant-snapshot proof then verified the rebased X against stale base+1 -- a deterministic fail-closed ERROR for every follower of a rebased round (direct hit under multi-client-per-node load). The claim now inherits tag_slot->grant_base_own_generation under the same partition lock; zero (no rebase live) keeps the enqueue-time math bit for bit. TDD: test_local_follower_claim_inherits_published_rebase drives the production chain leader join->claim->same-round follower join (pre-cutoff) ->rebase publish (5->8)->leader INSTALL/COMMIT/FINAL on 8+1->claim release ->follower claim. RED on the pre-fix code: claim OK but inherited base 0 != 8 and cluster_pcm_x_writer_grant_snapshot_exact refused rebased+1; GREEN after (257/257). Also repairs the unit TAP plans that had drifted from the executed test count (review P1-1): pcm_x_convert 247->257, gcs_block 76->78 (78/78).
…ce V2 admission at ingress t/400 L3 review P0-2 + P1-3. P0-2: the formation collector read cluster_sf_peer_supports_pcm_x_rebase() in a separate lock acquisition from the session authentication, so a peer reconnect inside one collection could bind the OLD connection's REBASE bit to the NEW connection's session and wrongly activate formation-wide V2 coverage. The collector now takes one lock-coherent (CONVERT, REBASE, capability-record generation) triple per member -- new pure cluster_sf_peer_cap_family_sample() + locked cluster_sf_peer_pcm_x_capability_sample() -- and re-samples after the membership/epoch after-pass: any reconnect moves the record generation and the tick is refused instead of activated. P1-3: the INSTALL_READY handler accepted a 112-byte V2 frame on length alone; the sender-side coverage gate is not a receiver invariant. cluster_gcs_pcm_x_install_ready_ingress_valid() now requires, for the V2 length, BOTH the local activated formation's rebase_wire_active and the source connection's REBASE HELLO; V1 frames stay independent of the pair. TDD: gcs_block truth-table rows for the V2 admission pair RED (3 bites) against the ungated validator, GREEN after (79/79 incl. the new collector source pin test_pcm_x_formation_samples_capability_family_atomically); sf_dep family-sample record-coherence matrix 12/12.
…rship object t/400 L3 review P1-4. The transient-BUSY preflight fix shipped with a classification table, preflight truth rows and a driver source pin, but no test EXECUTED the arm end to end. The new pcm_own case drives the real reservation object: a live GRANT_PENDING (then REVOKING) lifecycle preflights BUSY and the RESERVATION_PREFLIGHT site classifies WAIT; after the lifecycle clears, the SAME identity's re-snapshot preflights OK and the real grant reservation begins on the idle tuple (token monotonic, GRANT_PENDING re-armed). Formation/session drift across the wait fails the exact runtime proof on gate generation, master session and runtime state, refusing the re-entry before any reservation mutation. 46/46.
t/400 L3 review R2 P0-1 — the deterministic liveness regression behind the current t/400 RED. The receiver-side V2 admission consulted cluster_sf_peer_supports_pcm_x_rebase() for every 112-byte frame, but a SELF-loopback frame (master == requester node) has no HELLO capability record, so a legal self V2 INSTALL_READY was silently dropped. The master ticket froze in ACTIVE_TRANSFER at the PREPARE_GRANT leg, the remote retained source (REVOKING descriptor) was never released, and every legacy S/X entry on the source node then surfaced 'conflicts with an active reservation' client ERRORs (t/400 runs 1-2 on this branch; base commit shows zero such conflicts — bisect-confirmed regression of this branch). Mirror gcs_block_pcm_x_source_capable: self IS rebase-capable (the local binary), while the activation-bound formation coverage flag still applies. Pinned by test_pcm_x_install_ready_v2_self_loopback_is_admissible (80/80); behavioral proof is the t/400 e2e that exercises self V2 on every drift.
…local Review closure for the BARRIER_CLOSED caller-owned unwind (opus CRITICAL + codex R2 P0-3 minimal set): a re-entered pass must not consume attempt-local state from the abandoned pass. - ITL slot picks: both pairs reset before derivation, so a cross-page pass's new-page slot index can never be stamped onto a different page chosen by the retry (the stale-active false-visible class). - HOT / summarized-update verdicts: reset before the same-page check, so a cross-page retry can never carry a same-page HOT verdict and skip index entries. - cid/iscombo: the entry cid is restored on every requalify arm, so a pass-1 combo cmax neither feeds HeapTupleSatisfiesUpdate nor recombines into a combo-of-combo; the temp-locked reacquire arm deliberately keeps the pass-1 combo it already stamped. - heap_delete now frees the abandoned replica-identity copy like heap_update. Behavioral repeat-barrier tests are deferred with the rest of the non-blocker review queue until after the S3 gate (user scope ruling).
… reservation t/400 L3 review R2 P0-2 — the second live t/400 blocker. A reader (or the rare un-valid-page writer) entering the legacy LockBuffer reservation while the tuple carries a live GRANT_PENDING/REVOKING lifecycle got an immediate 'conflicts with an active reservation' client ERROR, even though a normal revoke/grant window is a ms-scale transient (t/400: three node0 client errors against a retained-source window). cluster_bufmgr_pcm_begin_grant_reservation_wait() re-begins against the re-sampled complete ownership tuple with the holder backoff ladder and the PCM convert wait event; it never touches another lifecycle's token/flags. Sleeping is refused under the nested-wait guard (a held frozen-tag content lock) and bounded at 64 waits — both fall back to the historical fail-closed report. BUSY observations surface through pcm_x_own_busy_count. Both legacy begin sites (master reservation + W1 revalidate) route through the helper; the source-scan contracts follow the new call.
…ier-aware caller
t/400 L3 — the next liveness layer surfaced once the self-V2 wedge fell:
the pre-crit VM acquire got its writer X granted, then the holder-gate
register hit a transient (busy holder lane) and its retry wait consulted
the nested guard while the backend still held the frozen-tag heap lock.
The guard refusal escalated to a client ERROR ('holder operation is not
ready, operation=retry wait nested guard result=13'), killing one t/400
writer per run.
The holder lane now speaks the same caller-owned-unwind protocol as the
writer prepare: cluster_bufmgr_pcm_x_holder_retry_wait() reports a
BARRIER_CLOSED guard verdict back through the new barrier_refused
out-param (counting it in pcm_x_queue_barrier_unwind_count) instead of
throwing; holder_prepare unwinds its untouched ledger slot; and
LockBufferInternal rolls the already-granted writer claim back through the
exact ERROR-path abort before handing the refusal to heapam, which
releases its own outer lock, warms the conversion unlocked and re-enters.
Every other guard verdict and every non-aware call site (deferred detach,
unregister, release, W1 reacquire) keeps the historical fail-closed
report.
…he node The t/400 drain-leg wedge froze with the master DRAIN leg armed at the requester node forever: that node fused at the self-source DRAIN proof arm (cluster_gcs_block.c fail-closed site), swallowed the ack, and every later poll died silently on the runtime gate. The fail-closed record only names file:line, so the refusal sub-arm - generation mismatch versus BUSY versus an evicted buffer - is invisible, and each sub-arm needs a different repair. Capture a one-shot descriptor snapshot (generation, token, own flags, pcm_state, buffer_type, BM_VALID, tag match) inside the existing header lock in cluster_bufmgr_pcm_own_self_handoff_x_exact and log it at the caller right before the fuse. Snapshot fill only under the spinlock; the ereport runs after both locks are released. No shmem or wire ABI change; the sample struct is process-local.
The t/400 wedge dumps print leg_retry from the master ticket, but every re-drive path re-arms its leg through the DUPLICATE case without touching retry_count, so a frozen dump shows leg_retry=0 whether the retry pump is re-staging every tick or died entirely - the two failure families that need opposite repairs. Count each exact DUPLICATE re-arm (terminal DRAIN/RETIRE legs and the transfer REVOKE leg, saturation-guarded like the local resend counter) so leg_retry becomes that discriminator, and give the periodic master drive a log-once note that fires when the same non-progress result repeats consecutively for the same ticket, naming the refusing arm. Progress results stay silent and reset the streak. Unit: the wire 49-56 and terminal-ack tests now pin the increment (recorded red at 0 != 1 / 0 != 2 before the counting change).
…VOKE refusal The transfer wedge shape is now proven live on the master side (the REVOKE leg re-arms every drive pass, leg_retry climbing) while the source node refuses the apply forever with own_busy ticking at exactly the resend cadence. Every source-side refusal arm - the ingress own snapshot, the materialize holder snapshot, and the materialize begin - returns silently, so the lifecycle that is holding the descriptor never names itself. Give those arms a shared log-once note that fires when the same refusal shape repeats consecutively, carrying the own snapshot (pcm_state, flags, generation, token). The flags value names the stuck lifecycle directly. A successful revoke apply resets the streak.
t/400 froze with a proven cross-ticket interlock: the oldest ticket sat in RETIRE_CREDIT with its RETIRE leg re-armed 114 times while the responder refused every poll, because that node's holder lane was still bound to the NEXT ticket's transfer (holder terminal not drained) and the RETIRE preflight's cross-lane guard correctly returns NOT_READY for an undrained holder. The younger ticket's own DRAIN leg sat armed with zero retries: the terminal retry tick drove only the single oldest terminal ticket, so the very DRAIN evidence the head was waiting for could never be collected once its one-shot event chain lost a message. Give the periodic driver a floor-scoped scan (cluster_pcm_x_master_terminal_work_next_after) and walk the terminal set in ticket-id order each tick, bounded by a small budget. The oldest ticket is still kicked first, so the contiguous retirement frontier keeps its priority; younger tickets merely regain the same at-least-once re-drive the head already had. Unit: a three-ticket fixture pins oldest-first, floor iteration, and NOT_FOUND past the tail (recorded red as a missing-symbol build before the implementation).
added 26 commits
July 19, 2026 05:18
…etion certificate The self-source DRAIN proof demanded the live descriptor still hold the freshly-committed shape (generation == source + 1, X/XCUR), but on a hot block the descriptor legitimately moves on between FINAL_ACK and the delayed DRAIN - an X->S downgrade serving a reader was caught on t/400 with live_gen = source + 2, pcm_state = S, buffer_type = XCUR - so the proof returned STALE and fused the whole node, swallowing the DRAIN ack forever (104 futile master retries) and freezing the queue until CSSD eviction. The descriptor is the wrong authority anyway: own generations are per-buf_id, not per-BufferTag lineage, so an evicted-and-reloaded tag could alias a larger generation without proving anything. Release the immutable record against the exact protocol ledger instead. The first (writer-lane) DRAIN consumption now captures a completion certificate under its own tag lock - ticket ref, image token, committed_own_generation, master identity - through the new cluster_pcm_x_local_drain_poll_certificate_exact variant (the plain call wraps it), and the release requires the certificate to match the holder binding exactly with committed == source generation + 1, the same invariant FINAL_ACK proved against the effective grant base. An inexact certificate preserves the record and refuses without fusing; the descriptor is only probed read-only for a structurally malformed flags/token shape, which alone still fails closed. Legitimate successor lifecycles, well-formed in-flight reservations, and eviction no longer fuse a delayed DRAIN. Unit: the transfer/final-ack lifecycle test pins the captured certificate fields and that a DUPLICATE replay does not fabricate one (recorded red at the valid assertion with the capture neutralized); the source-scan contracts pin the certificate policy and refusal arm ordering ahead of the record release, and the descriptor-probe contract replaces the old exact-shape pins.
…ed exit Every corruption arm of the local retire collect funnels to one fail-closed line (the retire_failed label), so the fuse that now leads two of four wedge autopsies - followed by thousands of futile RETIRE retries against the fused node - cannot say which surprise it hit or what the uncoerced result was. Log a one-shot note at each arm (preflight, allocator view, second-pass candidate, holder/writer detach, wake match, wake count) carrying the slot index and the raw result before the CORRUPT coercion.
…g it t/400 loop3/loop4b first-fuse autopsy (dump flags=0x3d, writer ref ticket 3, holder ticket 4, drain generations 7/7, one unlinked terminal membership, every FIFO index invalid): a granted writer terminal awaiting RETIRE and a newer revoked holder terminal legally share one tag under the writer round's REVOKE_BARRIER with two DISTINCT master tickets, but pcm_x_local_same_ref_dual_retire_state coerced any unequal-ref dual to CORRUPT, so RETIRE_UP_TO(3) closed the runtime and the master's RETIRE leg retried forever (leg_retry 6665/6662). Classifier: validate each lane's complete DRAIN evidence first; only ref-equal duals run the same-ref drain-generation/image rules; unequal refs under distinct ticket ids are the legal independent form (not a dual); a ticket-id collision under a different ref stays CORRUPT (master-unique ticket ids, the cross-lane alias class). Writer detach: the exact proven narrow shape (one unlinked terminal membership, all indexes invalid, clear writer leg, holder lane exact) retires ONLY the writer lane -- prehandle/ref/image/reliable/committed/ base/terminal/membership -- keeping the holder lane, the barrier and the master binding byte-exact on the LIVE tag. Any wider distinct-ticket dual stays NOT_READY with zero mutation, in both the RETIRE preflight and the detach. The holder ticket's own later watermark then consumes the holder lane and closes the still-frozen empty round (round advance + barrier drop) before the tag detaches. RED (recorded pre-fix): the new production-path unit test built the 0x3d shape and RETIRE_UP_TO(3) returned CORRUPT with the runtime RECOVERY_BLOCKED at test_cluster_pcm_x_convert.c:10636-10637. GREEN: 260/260 unit, full-suite 182 binaries, check-format clean. Alias / half-mask / poisoned-drain-leg probes keep the fail-closed verdict.
… lane retires Adversarial review of the independent dual-terminal fix found the narrow retain arm proved too little and too asymmetrically: - A next-round follower joining BEFORE the writer's RETIRE pushed the shape out of the members==1 eligibility, and the follower waits for exactly this RETIRE to drop the barrier: a deterministic cycle. - A follower joining BETWEEN the two watermarks hit the opposite hole: the holder detach consumed the last terminal authority, answered OK and left the barrier standing over the follower forever. - A cancelled external lane (zero grant generation on the blocker locator, reachable while the writer cohort is still terminal) passed the preflight but failed the detach's gen!=0 check, escalating a retryable disagreement into the collect's fail-closed exit. - A single watermark covering both tickets never got past the holder lane's cross-lane refusal even though the same episode retires the writer lane first. One shared predicate now decides writer-lane eligibility for both passes (barrier standing, one closed round of exactly the target, round/cutoff invariants, FIFO holding at most next-round followers, clear writer leg, exact holder lane; cancelled externals eligible with their staged second half defined in the cross-lane guard). The holder detach rehearses the frozen-round close read-only BEFORE consuming any evidence -- reusing follower_chain_ready for the promotion plan -- and applies round-advance/barrier-drop/promotion infallibly afterwards; the RETIRE preflight runs the same rehearsal (projected past the writer retire when one watermark covers both lanes), so no fallible check ever runs after the first mutation. The writer preflight stops projecting a final-member promotion the retain arm never performs. RED (recorded on detached 1be1998 with the pure test diff): all five new production-path legs fail exactly on their gap -- pre-joined follower NOT_READY(7) and never promoted; between-watermarks join left REVOKE_BARRIER=1 and role=FOLLOWER after the holder RETIRE; cancelled external CORRUPT(10)+RECOVERY_BLOCKED; single watermark NOT_READY(7); invariant-break answered CORRUPT instead of a byte-exact NOT_READY. GREEN: 265/265 unit, full-suite 182 binaries, check-format clean.
… evidence Adversarial review P0-12: a next-round follower cancelled around the independent dual's watermarks leaves an unlinked CANCELLED membership on the tag, and the generic detach refuses to touch anything while terminal evidence occupies it. Cancelled before the writer watermark, the membership wedges writer-lane eligibility forever (members>1 with an empty FIFO); cancelled between the watermarks, the holder RETIRE's frozen-round close read membership=1/empty-FIFO as corruption and fused the runtime. An exact unlinked next-round CANCELLED member now detaches by releasing only its own membership -- the dual terminals, the holder lane, the barrier and the master binding stay byte-exact -- and the frozen-round close plan treats a nonzero membership behind an empty FIFO as the legal transient it is (NOT_READY until those members' own detaches land) instead of guessing it away as corruption. RED (recorded on detached 0d26b44 with the pure test diff): cancel-before-watermark leaves detach NOT_READY(7) and the membership stuck at 2 with every later RETIRE refusing; cancel-between-watermarks turns the holder RETIRE into CORRUPT(10) + RECOVERY_BLOCKED. GREEN: 267/267 unit, full-suite 182 binaries, check-format clean.
…ake batch Review P1 on the independent dual-terminal close: the holder RETIRE's frozen-round close promotes the next-round follower but the promotion never reached the wake batch -- the preflight rehearsal dropped its close plan's candidate, and the detach collected its wake output before the promotion applied. Production wakes only what the batch carries, so a promoted hot-block leader degraded to its 2-32ms poll interval. The preflight (both the holder arm and the writer arm's projected single-watermark rehearsal) now projects the close plan's candidate into the wake batch, and the holder detach backfills the promoted identity into its wake output after the infallible apply. A zero capacity batch therefore refuses the whole episode with NO_CAPACITY before any mutation. RED (recorded on detached ad67e65 with the pure test diff): both new collect legs fail -- capacity 0 answered OK and mutated the tag, capacity 1 delivered count=0 with the promoted identity missing. GREEN: 269/269 unit, full-suite 182 binaries, check-format clean.
…sh fails The fast-fail finish family (result=1 STALE at the exact grant finish, now the leading first cause in loop8: three of four runs, all four clients at the same instant, queue draining cleanly afterwards) rolls back a durable master grant because the live ownership tuple no longer matches the reservation base -- but the error never said WHICH of the five compared terms (tag, generation, token, flags, pcm_state) moved, or to what, so the producer of the divergence cannot be named. Thread the classification-time live sample out of cluster_pcm_own_finish_grant_reservation (the post-rollback descriptor is useless: the rollback itself mutates flags and possibly the generation) and carry live-vs-base into the existing error detail. Cold path only; no behavior change.
…h shape The fast-fail finish family (S3 errors=0 hard blocker; leading first cause in loop7/loop8) is not a race at all: an S holder upgrading to X through the legacy master acquire begins a fresh-token GRANT_PENDING reservation from S/flags=0 -- the reservation begin snapshots whatever coherent state it finds, and the S-base abort path has always demoted S to N on rollback for exactly that shape -- but the grant-reservation classifier enumerated only N_NEW and the three revoke handoffs. Every such upgrade therefore failed its finish deterministically, rolled a durable master grant back and surfaced result=1 to the client; the loop9 evidence detail shows the live tuple matching the reservation identity exactly (S, same generation, token+1, GRANT_PENDING) against the S/flags=0 base. Add the S_UPGRADE arm for that shape. The finish gate already demands PCM_STATE_X plus a held content lock for every non-N_NEW kind, the grant commit is state-agnostic (one generation bump, flags cleared), and the queue-lane consumers compare against explicit expected kinds, so nothing else widens. A fresh-token X base stays deliberately unenumerated: a live X cover never re-acquires through this path. RED (recorded on detached 9b651a9 with the pure test diff): the shape cannot even be named -- use of undeclared identifier CLUSTER_PCM_X_GRANT_RESERVATION_S_UPGRADE at the first assertion. GREEN: 47/47 pcm_own unit, full-suite 182 binaries, check-format clean.
…nt finish shape" Adversarial review caught a protocol hazard in f40e54a before it could regress S3: legalizing the fresh-token S-base finish shape lets the stale-cover fallback bypass the convert queue's FIFO/WFG writer arbitration entirely -- the original unordered multi-writer defect the queue exists to prevent. The reviewed production chain is: cached-X cover sampled -> in-window X->S downgrade -> revalidate detects the stale cover -> the fallback WRONGLY enters the legacy master acquire from the S base and mints the "S_NEW" tuple. loop10b (clean rebuild) confirmed f40e was not a closure anyway: the same shape now surfaced result=6 (INVALID) from the finish gate's next check. Restore the classifier to refuse the shape, and flip the unit leg into a protocol pin: S_NEW (and the X-base analogue) stay INVALID. The real fix is queue re-entry at the stale-cover fallback -- evidence first via the cached-X stall TAP leg -- never a finish-side swallow of STALE/INVALID.
… writer re-acquire
The t/400 fast-fail finish family's root: the cached-X cover fast path
skips the convert queue join, and when the post-content-lock re-verify
found the cover stale (a BAST X->S downgrade raced the window), the
fallback re-acquired through the LEGACY master begin from the S base.
That both bypasses the queue's FIFO/WFG writer arbitration (the
original S3 unordered multi-writer defect) and mints the fresh-token
S-base reservation shape ("S_NEW") the finish classifier refuses by
design -- so every such re-acquire rolled back a durable master grant
and surfaced result=1 to the client.
The stale-cover fallback for an X request now detaches the ACQUIRING
holder it prepared, re-enters the queue through the same
writer-prepare gate as a fresh conversion (honoring a frozen-barrier
refusal through the existing post-PG_TRY unwind exit), binds a new
holder and activates the writer under the retaken content lock. The
legacy begin remains only for share-mode refreshes and unmanaged
relations. No finish-side result is swallowed; the S_NEW protocol pin
from the preceding revert stands.
RED (recorded on the parent commit 7755173, unmodified test):
t/394 L4c failed with exactly the production signature -- writer
outcome err=[could not finish exact cluster PCM grant reservation:
result=1], cover_stale_detected delta=1, reverify_reacquire delta=1.
GREEN: t/394 all tests successful with err=[] and the same two
deltas; full-suite 182 unit binaries; check-format clean (bufmgr.c is
PG-core hand-styled). Follow-up legs registered: queue
enqueue/commit counter growth, old-holder detach and new-holder bind
assertions on the t/394 leg, each with its own baseline RED.
…etail loop11 shows the S_NEW tuple surviving the queue re-entry fix on a SECOND producer chain; the detail line carried the live/base tuples but not whether the failing finish targeted S or X -- the one fact that separates an S->S cover refresh from an S->X upgrade. Cold path only.
…reservation A legacy grant reservation that opens from a non-N base is the protocol-violating shape behind the deterministic S_NEW finish refusals: an X convert must ride the convert queue, and an S refresh over a held S must be served by the local cover gate. The finish-side errdetail already names the rolled-back five-tuple and the target mode, but cannot say which acquire entry path minted the reservation. Log the complete begin-time context (site, acquire mode, buffer id, tag, BM_VALID, and the exact ownership tuple the reservation snapshotted) at both legacy begin call sites, gated on the non-N base shape so the hot path stays silent. Zero behavior change.
…BM_VALID under a pin t/400 loop14 root cause (P0-15): cluster_bufmgr_pcm_own_release_retained_image unconditionally cleared BM_VALID and retagged CURRENT when the exact retained write-fence token was released, even while pre-existing PG pins were live. A pin holder that then went straight to LockBuffer (never back through ReadBuffer) ran the PCM acquire ladder over invalid bytes: the legacy master reservation opened from the invalid base, the finish stamped an S mirror onto the invalid descriptor, and the next X convert was bounced from the convert queue (writer-prepare BM_VALID gate) into a legacy S-base begin whose S_NEW finish shape is refused by design -- the deterministic "could not finish exact cluster PCM grant reservation" client errors. Fix, upstream of the master reservation and converged with the proven passive-pin invalidate release shape (cluster_bufmgr_pcm_own_release_pinned_s_for_gcs): - new shared pin-aware retag helper cluster_pcm_x_retained_release_retag: refcount==0 drops the image (!BM_VALID + CURRENT, next ordinary read reloads); refcount>0 keeps the frozen PI+BM_VALID never-write/never-serve N mirror byte-exact, so the next S acquire installs over it via an exact GRANT_PENDING (content_write_permitted already admits exactly that), an X convert rides the convert queue, and eviction retags after the last pin drains. No image swap and no vanishing bytes under a pin holder. - grant finish fails closed (CORRUPT) instead of committing a durable S/X mirror over !BM_VALID or a PI mirror (Rule 8.A: never silently cover stale or absent bytes); S_NEW/X-base stay refused, no queue bypass. - release evidence LOG names dropped vs kept-pinned-pi; the direct own-transition mirror keeps its S/X-over-!BM_VALID evidence LOG. RED (recorded on parent + this tree pre-fix): missing-symbol build failure of test_retained_release_retag_respects_pin_contract (real BufferDesc, both refcount arms, byte-exact pinned assertion) and absent scan needles in test_retained_release_and_finish_never_cover_invalid_bytes (retag routed under the same header-lock hold; finish valid-image gate; reload-in-place helper pinned as forbidden). Stale drain-retag scan expectation updated to the pin-aware contract. GREEN: 49/49 + 269/269, 182 unit binaries, full build 0 errors, check-format 0.
…ge gate t/400 loop15 caught a regression in the new finish gate: the EXTEND / READ_MISS direct-init path legitimately commits its X grant between StartBufferIO and TerminateBufferIO(BM_VALID) -- the descriptor still owns BM_IO_IN_PROGRESS and the zeroed page is published only after the caller initializes it under content EXCLUSIVE (first INSERT into a fresh table failed with finish result=CORRUPT at warmup). Narrow the refusal to a PI mirror or a dropped image with no IO in progress; the loop14 poison shape (free-floating !BM_VALID descriptor after a retained release) stays refused. GREEN: 49/49 pcm_own, full build 0 errors, check-format 0.
…urrency proof The retained release now keeps a frozen PI+BM_VALID N mirror for pre-existing pins, and the grant finish refuses to commit S/X over a PI mirror. The legacy grant lane, however, historically republished CURRENT only implicitly through the finish's SCUR/XCUR stamp -- so a legitimate re-acquire over the kept mirror (t/400 loop16: hot-page S finishes refused with CORRUPT at gen=4) was failing closed even after its bytes were proven current. Give the legacy lane an explicit republish at each byte-currency proof point, mirroring the queue lane's publish_installed_x_image: - new header predicate cluster_pcm_x_grant_pending_republish_shape: only a pcm N + live GRANT_PENDING + nonzero-token + BM_VALID + PI descriptor may flip, and only to CURRENT; - new bufmgr helper cluster_bufmgr_pcm_own_republish_grant_pending_image applies it under header authority; - call sites: the shipped-image install memcpy (same content-EXCLUSIVE hold), the storage-fallback direct SCN PASS proof, and the refresh-then-PASS proof. The SKIP arms leave PI frozen, so the finish valid-image gate keeps failing closed on unproven bytes. RED (recorded pre-fix): missing-symbol on the predicate truth-table legs in test_retained_release_retag_respects_pin_contract, and scan-needle failure of test_legacy_byte_proof_sites_republish_kept_pi_mirror (81st, gcs_block). GREEN: 50/50 + 81/81 + 269/269, full build 0 errors, check-format 0.
… handler t/400 loop17 (post P0-15): the client-error families are gone, but a form-B wedge remains -- the master re-sent REVOKE 29 times to the source node with zero refusal evidence in its log, because the streak note covered only the ingress-snapshot arm while the handler refused at one of five other silent returns. Route every refusal arm through the log-once streak note (ingress auth, holder-ledger stale, holder-progress error, image reserve, and the apply result), and keep the dedup-slot release on a failed apply inside the same refusal branch. Observability only; no protocol change. GREEN: 82/82 gcs_block (new arm-coverage scan 82nd), full build 0 errors, check-format 0.
…commit t/400 L3 asserts pcm_x_own_begin_count and pcm_x_own_commit_count advance during the four-writer workload, but production never called the stats API: both counters sat at 0 deterministically. Wire the begin counter at every successful GRANT_PENDING linearization in bufmgr.c -- the legacy grant reservation, the queue X reservation, the REVOKING->GRANT_PENDING handoff, and the direct-init gate -- always after UnlockBufHdr and only on CLUSTER_PCM_OWN_OK. The idempotent duplicate PREPARE that merely observes GRANT_PENDING stays uncounted, as do all BUSY/STALE/CORRUPT refusals, so replay and failure semantics stay honest. Wire the commit counter once, at the single exact-commit funnel shared by the queue finish, the LockBuffer finalize, and the direct-init grant, gated on the target state being X so an S-grant finish stays silent. Pin the wiring with a source-contract test in test_cluster_pcm_own (four begin sites, one commit funnel, counters outside the header lock). t/400 L3: own_begin/own_commit "advanced" flip red->green (begin=20 delta=9, commit=10 delta=3); own_abort/own_corrupt deltas stay 0.
Stabilize exact PCM-X queue ownership, transfer, retirement, visibility, and fail-closed handling across concurrent four-node writers. Add deterministic unit and TAP coverage for data-plane routing, dirty-image retention, queue drainage, data conservation, and injected FlushBuffer failures. Validation: t/400 228/228 PASS and ten consecutive acceptance runs.
Apply the repository clang-format 18 rules to the PCM-X liveness implementation and unit coverage so the fast-gate validates the published tree. Validation: format gate 580/580 and t/400 228/228 PASS.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Superseded by #64, which contains the same validated final tree as a single conventional commit based on the latest
main.