Create Failure Recovery Runbook

[Abd-Standard]

Description:
Document practical recovery procedures for common NotifyChain failures.

Include:

Listener interruption.
RPC failures.
Database downtime.
Stuck notification jobs.
Failed notification delivery.
Event backfill.

[I-Am-Finite]

@I-Am-Finite has applied to work on this issue as part of the Stellar Wave Program's 9th wave.

Hi! I've reviewed the requirements for create Failure Recovery Runbook and checked the codebase structure. I can deliver a clean, focused implementation that strictly adheres to your repo's patterns with unit tests included. Ready to open the PR promptly once assigned.

ℹ️ Repo Maintainers: To accept this application, review their application or assign @I-Am-Finite to this issue.

[Tsubashimo-Nanato]

@Tsubashimo-Nanato has applied to work on this issue as part of the Stellar Wave Program's 9th wave.

Hi, I'd like to work on #808. I reviewed the existing incident and notification recovery guides and noticed that the notification guide still describes Discord delivery as having no automatic retry, while the current subscriber uses NotificationRetryQueue.

After assignment, I would update and cross-link the existing documentation into a practical recovery path for listener interruption, RPC failures, database downtime, stuck jobs, failed deliveries, and event backfill. Each case would distinguish diagnosis, supported recovery steps, and checks that confirm recovery, with particular care around replay and duplicate notifications. I would verify commands and configuration against the current source and clearly document any validation limitations.

I plan to use AI assistance, review the resulting documentation, and take responsibility for its accuracy. I will wait for assignment before starting.

ℹ️ Repo Maintainers: To accept this application, review their application or assign @Tsubashimo-Nanato to this issue.

[Rayhab2000]

@Rayhab2000 has applied to work on this issue as part of the Stellar Wave Program's 9th wave.

Please assign this issue to me

ℹ️ Repo Maintainers: To accept this application, review their application or assign @Rayhab2000 to this issue.

[MaybeSomeone-arc18]

@MaybeSomeone-arc18 has applied to work on this issue as part of the Stellar Wave Program's 9th wave.

Tree-checked: NOTIFICATION_FAILURE_RECOVERY.md explains retry concepts and INCIDENT_RESPONSE_RUNBOOK.md covers severity triage, but neither gives practical per-failure recovery procedures for the six cases named here - listener interruption, RPC failures, database downtime, stuck jobs, failed delivery, event backfill. Plan: an operator-facing runbook with one playbook per case - detection signals, exact commands/queries to confirm, recovery steps, and prevention hooks - cross-linked from the two existing docs rather than duplicating them. I know this tree hands-on: four merge-damage repairs plus a lockfile resync committed on my working branch. I will implement and run everything locally before opening the PR.

ℹ️ Repo Maintainers: To accept this application, review their application or assign @MaybeSomeone-arc18 to this issue.

[drips-wave]

Congratulations, @Rayhab2000! 🎉 Your application was accepted by the repo's maintainers, and the issue is due on September 30, 2026.

🧑‍💻 @Rayhab2000: Please resolve the issue such that the repo's maintainers have enough time to review your contribution before the due date. You'll earn Points for completing the issue on-time, which will make you eligible for a share of the Stellar Wave Program's reward pool.

Warning

When opening a PR, please link it to this issue to ensure it gets tracked accurately. Points are awarded when this issue is marked as completed by the maintainer.

🤠 Repo maintainers: Please keep an eye on the contributor's progress and review their work before the due date. You can manage this issue, including adjusting its complexity and points, here.

🌊 Happy Wave 🌊

[grantfox-oss]

🎉 This issue has been marked as completed on GrantFox!

@Rayhab2000's PR #875 was approved and merged by @Abd-Standard.

🏆 @Rayhab2000: You earned 40 FoxPoints for this contribution! Your current tier: Builder (1,182 total points). Track your full progress on GrantFox.

👏 Great work, @Rayhab2000! Keep contributing to Core-Foundry.