Skip to content

docs: add failure recovery runbook for issue #808 - #875

Open
Rayhab2000 wants to merge 1 commit into
Core-Foundry:mainfrom
Rayhab2000:feat/808-failure-recovery-runbook
Open

Rayhab2000 wants to merge 1 commit into
Core-Foundry:mainfrom
Rayhab2000:feat/808-failure-recovery-runbook

Conversation

@Rayhab2000

Copy link
Copy Markdown
Contributor

Document practical recovery procedures for common NotifyChain failures:

  • Listener interruption: safe restart, stale lock release, prevention
  • RPC failures: detect, diagnose, fallback RPC switching
  • Database downtime: integrity checks, migration, corruption recovery
  • Stuck notification jobs: lock release, bulk re-queue, config tuning
  • Failed notification delivery: Discord webhook fix, retry backoff
  • Event backfill: cursor reset, selective range backfill, dedup safeguards

Closes #808

Overview

Related Issue

Closes #

Changes

Verification

# e.g.
cd dashboard && npm test
cd listener && npm run typecheck && npm test
cd contract/contracts/hello-world && cargo test

How to Test

Checklist

  • Branch is up to date with main
  • Tests added/updated and all pass locally
  • cargo fmt --all run (if Rust changes)
  • npm run lint passes (if TypeScript changes)
  • Documentation updated if behavior changed

Document practical recovery procedures for common NotifyChain failures:
- Listener interruption: safe restart, stale lock release, prevention
- RPC failures: detect, diagnose, fallback RPC switching
- Database downtime: integrity checks, migration, corruption recovery
- Stuck notification jobs: lock release, bulk re-queue, config tuning
- Failed notification delivery: Discord webhook fix, retry backoff
- Event backfill: cursor reset, selective range backfill, dedup safeguards

Closes Core-Foundry#808
@drips-wave

drips-wave Bot commented Sep 28, 2026

Copy link
Copy Markdown

@Rayhab2000 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Create Failure Recovery Runbook

1 participant