Group slots: group-safe WAL retention for Spock multi-master clusters. - #546
Open
ibrarahmad wants to merge 1 commit into
Open
Group slots: group-safe WAL retention for Spock multi-master clusters.#546ibrarahmad wants to merge 1 commit into
ibrarahmad wants to merge 1 commit into
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThe PR adds opt-in group replication slots with durable SQL state, safe advancement and repair controls, background workers, node lifecycle integration, documentation, and TAP coverage. It also adds large-transaction guidance, Zodan failover-slot documentation, and updates the Spock version to 6.0.1. ChangesGroup replication slots
Large transaction apply guidance
Zodan failover slot documentation
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Up to standards ✅🟢 Issues
|
| Category | Results |
|---|---|
| Complexity | 2 medium |
🟢 Metrics 0 complexity · 0 duplication
Metric Results Complexity 0 Duplication 0
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
There was a problem hiding this comment.
Actionable comments posted: 14
🧹 Nitpick comments (4)
sql/spock--6.0.0--6.0.1.sql (3)
68-72: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueDead exclusion clause.
spkgrp_…can never matchLIKE 'spk\_%'(the escaped_requires a literal underscore at position 4), so theslot_name <> COALESCE(spock.local_group_slot_name(), '')guard is unreachable. Harmless, but it costs alocal_group_slot_name()call per row and obscures intent.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@sql/spock--6.0.0--6.0.1.sql` around lines 68 - 72, Remove the unreachable slot_name <> COALESCE(spock.local_group_slot_name(), '') predicate from the logical replication slot query, leaving the database, plugin, slot_type, and slot_name LIKE filters unchanged.
11-54: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winConsider marking the new state tables as extension config so they survive
pg_dump.Without
pg_catalog.pg_extension_config_dump(),group_slot_state,group_slot_membership, andgroup_slot_member_progressare recreated empty on restore, losing membership generation and freeze state. Other Spock catalog tables use this pattern.♻️ Suggested addition after the table definitions
SELECT pg_catalog.pg_extension_config_dump('spock.group_slot_state', ''); SELECT pg_catalog.pg_extension_config_dump('spock.group_slot_membership', '');🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@sql/spock--6.0.0--6.0.1.sql` around lines 11 - 54, Mark all three new state tables—spock.group_slot_state, spock.group_slot_membership, and spock.group_slot_member_progress—as extension configuration tables by adding pg_catalog.pg_extension_config_dump calls after their definitions, so their data is included in pg_dump and restored.
781-788: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick win
group_slot_refresh_members()is left executable by PUBLIC.It is an internal mutator of the group-slot catalog tables; revoke it alongside the other maintenance functions for a consistent surface.
🔒️ Proposed addition
REVOKE ALL ON FUNCTION spock.group_slot_complete_part(name) FROM PUBLIC; +REVOKE ALL ON FUNCTION spock.group_slot_refresh_members(bigint, interval) FROM PUBLIC;🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@sql/spock--6.0.0--6.0.1.sql` around lines 781 - 788, Add a REVOKE ALL statement for spock.group_slot_refresh_members() FROM PUBLIC alongside the existing group-slot maintenance function revocations, preserving the same privilege-restriction pattern.docs/managing/group_slots.md (1)
58-72: 🩺 Stability & Availability | 🔵 TrivialAdd WAL-retention guidance for a persistently blocked horizon.
While
stale_progress(or a join/part) holds, the slot never advances andpg_walgrows without bound. Worth documenting monitoring ofspock.group_slot_status()/pg_replication_slots.restart_lsnplus the interaction withmax_slot_wal_keep_size(which can invalidate the group slot and requirespock.repair_group_slot('recreate')).🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/managing/group_slots.md` around lines 58 - 72, The “When advancement is refused” section should add WAL-retention guidance for persistently blocked horizons, especially stale_progress, join, and part states. Document monitoring spock.group_slot_status() and pg_replication_slots.restart_lsn, and explain that max_slot_wal_keep_size may invalidate the group slot, requiring spock.repair_group_slot('recreate').
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/large_transaction_apply.md`:
- Around line 14-15: Update the compound modifier in the reproduction
description to hyphenate “600,000-row INSERT,” preserving the surrounding
wording and meaning.
- Around line 102-125: Revise the “Turn off spock.check_all_uc_indexes” section
to state that ALTER SYSTEM changes persistent subscriber-wide server
configuration, affecting all relevant apply workers rather than only the load
session. Instruct operators to record and explicitly restore the previous
setting after the load, and warn that an interrupted load may leave
secondary-unique conflict checking disabled.
- Around line 127-136: Update the unique-index conflict explanation in the
TRANSDISCARD section to identify TRANSDISCARD as the default
spock.exception_behaviour. Describe that failed apply rows are skipped and
logged, and only state that the subscription is disabled when configuration
explicitly sets sub_disable; remove the claim that sub_disable is the default.
- Around line 64-76: Update the transaction batching example in
docs/large_transaction_apply.md to state that the DO block must execute
top-level with auto-commit enabled, not inside an existing transaction. Document
that transaction-wrapped scripts or framework migrations should use client-side
batching instead.
In `@docs/managing/group_slots.md`:
- Around line 48-51: Update the safe LSN description to state that
spock.group_slot_min_downstream_lsn() considers the minimum confirmed_flush_lsn
across every spk_* logical slot in the database, including leftover or slow
subscription slots. Add spock.group_slot_safe_horizon() and the relevant
join/part functions to the Functions section where they are referenced.
In `@docs/modify/zodan/failover_slots.md`:
- Around line 115-118: Update the explanatory comment above primary_conninfo to
remove the blanket claim that pg_basebackup -R does not include dbname, and
accurately state that PostgreSQL 17+ records dbname only when it was explicitly
provided through the connection string or environment.
- Around line 8-15: Update the introductory mechanism statement in
failover_slots.md to say that PostgreSQL introduced native failover-slot
synchronization in PG17 and retained sync_replication_slots in PG18, then
separately identify the PG18 Spock worker registration change. Keep the numbered
PG15/16 and PG17/18 scope descriptions unchanged.
- Around line 210-218: Update the standby slot readiness query to exclude
temporary slots, evaluate only rows whose invalidation_reason is null, and
require both synced and failover to be true for the remaining logical slots.
Ensure the PG17/18 readiness check fails when any eligible slot violates these
conditions.
In `@samples/Z0DAN/zodremove.sql`:
- Around line 490-492: Update the RAISE NOTICE statement in the group-slot
completion flow to generate the recovery command with a valid SQL-literal
argument. Replace the unsupported %L placeholder usage by pre-formatting or
quote-escaping target_node_name with format(... %L ...) or quote_literal(),
while preserving the existing notice text and command.
In `@sql/spock--6.0.0--6.0.1.sql`:
- Around line 751-753: Update the generation-handling flow around the
group_slot_state lookup and v_newgen assignment to explicitly detect when no
metadata row exists, matching the guard and error behavior of
group_slot_begin_part(). Raise that error before evaluating v_gen + 1, while
preserving the existing increment path when metadata is present.
- Around line 476-499: Guard against a NULL v_target after assigning
COALESCE(target_lsn, v_safe) in the slot-advance flow. Before the
pg_replication_slots lookup and spock.group_slot_state UPDATE, raise a clear
exception when both target_lsn and the evaluated safe_lsn are NULL, preserving
the existing advance and update behavior for non-NULL targets.
- Around line 624-632: Update group_slot_begin_join so its missing-metadata
branch does not call spock.group_slot_worker_tick(). Seed the required
group_slot_state row directly using the existing metadata initialization path,
or raise the established directive error, then reread membership_generation
before computing v_newgen. Preserve the existing generation increment behavior.
In `@src/spock_group_slot.c`:
- Around line 161-176: Update the teardown SQL in the SPI_execute block so
group_slot_state rows are deleted only for replication slots successfully
selected and dropped by the NOT s.active cleanup query. Preserve metadata for
active slots and emit a warning when any slot remains active or otherwise was
not dropped, preventing orphaned slots from losing their cleanup metadata.
- Around line 133-140: Guard the metadata seed INSERT in the surrounding
initialization path before calling SPI_execute_with_args, using a to_regclass()
check for spock.group_slot_state. If the catalog table is absent, skip the
INSERT and emit only a warning so spock.create_node() continues; preserve the
existing result-code warning for executed INSERTs.
---
Nitpick comments:
In `@docs/managing/group_slots.md`:
- Around line 58-72: The “When advancement is refused” section should add
WAL-retention guidance for persistently blocked horizons, especially
stale_progress, join, and part states. Document monitoring
spock.group_slot_status() and pg_replication_slots.restart_lsn, and explain that
max_slot_wal_keep_size may invalidate the group slot, requiring
spock.repair_group_slot('recreate').
In `@sql/spock--6.0.0--6.0.1.sql`:
- Around line 68-72: Remove the unreachable slot_name <>
COALESCE(spock.local_group_slot_name(), '') predicate from the logical
replication slot query, leaving the database, plugin, slot_type, and slot_name
LIKE filters unchanged.
- Around line 11-54: Mark all three new state tables—spock.group_slot_state,
spock.group_slot_membership, and spock.group_slot_member_progress—as extension
configuration tables by adding pg_catalog.pg_extension_config_dump calls after
their definitions, so their data is included in pg_dump and restored.
- Around line 781-788: Add a REVOKE ALL statement for
spock.group_slot_refresh_members() FROM PUBLIC alongside the existing group-slot
maintenance function revocations, preserving the same privilege-restriction
pattern.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 8dd116ca-d70d-4222-ad11-ff3477c37afa
📒 Files selected for processing (18)
docs/large_transaction_apply.mddocs/managing/group_slots.mddocs/managing/index.mddocs/modify/zodan/failover_slots.mddocs/spock_release_notes.mdinclude/spock.hinclude/spock_group_slot.hinclude/spock_worker.hsamples/Z0DAN/zodan.sqlsamples/Z0DAN/zodremove.sqlsql/spock--6.0.0--6.0.1.sqlsrc/spock.csrc/spock_functions.csrc/spock_group_slot.csrc/spock_manager.csrc/spock_worker.ctests/tap/scheduletests/tap/t/105_group_slots.pl
ibrarahmad
pushed a commit
that referenced
this pull request
Jul 26, 2026
- large_transaction_apply: default spock.exception_behaviour is transdiscard, not sub_disable; correct the duplicate-key outcome. - large_transaction_apply: note the DO/COMMIT batching example needs top-level autocommit execution, not a transaction-wrapped context. - zodan/failover_slots: pg_basebackup -R only records dbname when it was passed explicitly; reword the blanket claim. - zodan/failover_slots: PG17 introduced native slot sync; PG18 keeps it and the change there is Spock dropping its own worker. - zodan/failover_slots: PG17/18 readiness check also verifies the synced slot is not invalidated (invalidation_reason null). - managing/group_slots: document spock.group_slot_complete_part().
ibrarahmad
force-pushed
the
GROUP_SLOTS
branch
2 times, most recently
from
August 29, 2026 20:06
b3cb105 to
42a88f8
Compare
Each database can now maintain a single internally managed logical slot that retains WAL at the oldest position still needed by any member of the replication group. Without such a slot, a node that falls behind, or one that is midway through joining or leaving, can find that WAL it still requires has already been discarded. The slot is never decoded from and stays inactive. A per-database worker recomputes the group-safe horizon, the minimum confirmed position across downstream members, and advances the slot only once the group has converged: every required member must belong to the current membership generation and have reported progress recently enough. When that does not hold, the slot keeps its position rather than advancing on incomplete information. Node addition and removal drive the same machinery. A joining node freezes advancement until it has caught up, and a departing node pins the horizon at a recorded boundary until its removal completes, so neither transition strands the group behind a member that has since gone away. The feature is disabled by default; when off, no slot is created and replication behaves as before.
ibrarahmad
force-pushed
the
GROUP_SLOTS
branch
from
September 2, 2026 08:19
42a88f8 to
fd5c2fb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This change adds group replication slots to Spock: one internally managed, permanently inactive logical slot per Spock database that pins WAL at the group-safe position for a multi-master mesh. It protects WAL retention across membership changes so a node can rejoin and catch up after downtime without WAL being removed too early, while avoiding over-retention when a downstream is merely slow. The feature is off by default (spock.group_slots_enabled) and adds no overhead unless enabled. A per-database group-slot background worker is started and stopped by the Spock manager based on the toggle, and the manager reloads configuration on SIGHUP so the setting takes effect without a restart.
Catalog state tracks group membership, membership generations, and per-member replay progress, and explicit node lifecycle handling (active, joining, parting) ensures membership churn does not lose or strand WAL. Testing adds the TAP test 105_group_slots and runs against the existing suite. The branch also includes a small commit that aligns include ordering in src/spock_manager.c and the entries in tests/tap/schedule with main, so it merges into main with no conflicts.